Method and system for improving data interaction efficiency of hyper-converged architecture

Abstract

The invention discloses a method and a system for improving data interaction efficiency of a hyper-converged architecture, relates to the technical field of hyper-converged architectures, and aims to solve the problems of network bottleneck and storage delay of data transmission among different virtual machines or containers in the conventional hyper-converged architecture. According to the scheme, a data transmission optimization module needs to be introduced into a hyper-converged architecture, so that the data access speed and the network efficiency are improved through data prefetching and an intelligent routing technology; the distributed cache module is used for relieving the pressure of a single node and ensuring accurate synchronization of data through data locality optimization and cache consistency management; the parallel data processing module is used for improving the data processing efficiency through task parallelization and data fragmentation technologies; and the self-adaptive load balancing module is used for dynamically adjusting resource allocation according to the system load and the resource use condition so as to realize the balance of calculation, storage and network load. The method reduces the delay, improves the throughput, and improves the overall performance and resource utilization rate of the system.

Description

Technical Field

[0001] The present invention relates to the technical field of hyper-converged architecture, and in particular to a method and system for improving the data interaction efficiency of a hyper-converged architecture. Background Art

[0002] With the development of cloud computing and AI industries, hyper-converged architecture has gradually become a new choice for data centers due to its low cost and easy scalability. Hyper-converged Infrastructure (HCI) is an emerging IT infrastructure architecture that integrates computing, storage and network resources to form a highly integrated system. This architecture aims to improve resource utilization, reduce energy consumption, simplify management and maintenance, and improve business flexibility and agility by eliminating the physical boundaries in traditional data centers.

[0003] In a hyper-converged architecture, data interaction efficiency directly affects system performance and resource utilization. In current hyper-converged architectures, data transmission between different virtual machines (VMs) or containers often faces network bottlenecks and storage latency issues, resulting in overall performance degradation. Summary of the invention

[0004] In view of the problems of network bottleneck and storage delay commonly faced by data transmission between different virtual machines (VMs) or containers in the current hyper-converged architecture, the present invention provides a method and system for improving the data interaction efficiency of the hyper-converged architecture, aiming to reduce delay, increase throughput, and improve the performance and resource utilization of the overall system.

[0005] In a first aspect, the present invention provides a method for improving the data interaction efficiency of a hyper-converged architecture, and the technical solution adopted to solve the above technical problems is as follows:

[0006] A method for improving data interaction efficiency of a hyper-converged architecture includes the following operations:

[0007] (1) Introducing a data transmission optimization module into the hyper-converged architecture, which improves data access speed and network efficiency through data pre-fetching and intelligent routing technology;

[0008] (2) Introducing a distributed cache module into the hyper-converged architecture, which reduces the pressure on a single node and ensures accurate data synchronization through data locality optimization and cache consistency management;

[0009] (3) Introducing a parallel data processing module into the hyper-converged architecture, which improves data processing efficiency through task parallelization and data sharding technologies;

[0010] (4) An adaptive load balancing module is introduced into the hyper-converged architecture. The adaptive load balancing module dynamically adjusts resource allocation according to system load and resource usage to achieve a balance between computing, storage, and network loads.

[0011] Optionally, the operation (1) specifically includes the following process:

[0012] The data prediction algorithm analyzes historical access patterns and current data access requests, predicts the data that may be needed in the future, and caches it in local storage in advance to achieve data pre-fetching;

[0013] During the data transmission process, the network load and the performance of the network path are monitored in real time, and the optimization algorithm is used to dynamically select the best path for data transmission to achieve intelligent routing.

[0014] Optionally, the operation (2) specifically includes the following process:

[0015] Adopting a data caching strategy based on geographic location, caching pre-specified data to storage media close to the computing nodes to optimize data locality;

[0016] Distributed consistency algorithms are used to ensure the accuracy and synchronization of cached data and implement cache consistency management.

[0017] Optionally, the operation (3) specifically includes the following process:

[0018] Task parallelization: Use multi-core processors and multi-threading technology to decompose data processing tasks into multiple subtasks and execute them simultaneously, and use the parallel computing framework to achieve efficient parallel processing;

[0019] Data sharding: Split a large data set into multiple small data pieces, process and store them separately, and dynamically adjust the processing and storage of small data pieces based on the load balancing algorithm.

[0020] Optionally, the operation (4) specifically includes the following process:

[0021] According to the real-time system load, dynamic resource scheduling technology is used to automatically adjust the allocation of computing resources and storage resources to achieve dynamic resource scheduling;

[0022] By monitoring the system load change trend, the prediction model is used to predict future load changes and make corresponding adjustments.

[0023] In a second aspect, the present invention provides a system for improving the data interaction efficiency of a hyper-converged architecture, and the technical solutions adopted to solve the above technical problems are as follows:

[0024] A system for improving the data interaction efficiency of a hyper-converged architecture includes:

[0025] Data transmission optimization module, used to improve data access speed and network efficiency through data pre-fetching and intelligent routing technology;

[0026] Distributed cache module, which is used to reduce the pressure on a single node and ensure accurate data synchronization through data locality optimization and cache consistency management;

[0027] Parallel data processing module, used to improve data processing efficiency through task parallelization and data sharding technology;

[0028] The adaptive load balancing module is used to dynamically adjust resource allocation according to system load and resource usage to achieve a balance in computing, storage, and network loads.

[0029] Optionally, the data transmission optimization modules involved include:

[0030] A data pre-fetch unit is used to analyze historical access patterns and current data access requests through a data prediction algorithm, predict the data that may be needed in the future, and cache it in local storage in advance;

[0031] The intelligent routing unit is used to monitor the network load and the performance of the network path in real time during data transmission, and dynamically select the best path for data transmission using an optimization algorithm.

[0032] Optionally, the distributed cache module introduced includes:

[0033] A local optimization unit is used to adopt a data caching strategy based on geographic location to cache pre-specified data in a storage medium that is closer to the computing node, thereby optimizing data locality.

[0034] The consistency management unit is used to ensure the accuracy and synchronization of cached data using a distributed consistency algorithm.

[0035] Optionally, the parallel data processing modules involved include:

[0036] Task parallelization unit, which is used to decompose data processing tasks into multiple subtasks and execute them simultaneously by using multi-core processors and multi-threading technology, and realize efficient parallel processing with the help of parallel computing framework;

[0037] The data sharding unit is used to split a large data set into multiple small data slices for separate processing and storage. At the same time, the processing and storage process of small data slices is dynamically adjusted based on the load balancing algorithm.

[0038] Optionally, the adaptive load balancing modules involved include:

[0039] Dynamic scheduling unit, which is used to automatically adjust the allocation of computing resources and storage resources according to the real-time load of the system using dynamic resource scheduling technology to ensure the best utilization of resources;

[0040] The prediction and adjustment unit is used to monitor the system load change trend, use the prediction model to predict future load changes, and make corresponding adjustments.

[0041] A method and system for improving the data interaction efficiency of a hyper-converged architecture of the present invention has the following beneficial effects compared with the prior art:

[0042] (1) The present invention can optimize the data transmission mechanism and processing flow, reduce delay, increase throughput, and enhance the performance and resource utilization of the overall system.

[0043] (2) The present invention can effectively improve the data interaction efficiency of the hyper-converged architecture; the data transmission optimization module and the distributed cache module significantly reduce network bottlenecks and storage delays, and improve the data transmission speed; the parallel data processing module and the adaptive load balancing module further improve the data processing speed and the overall performance of the system; these modules work together to enable the hyper-converged architecture to have better performance and stability when processing high-load data interactions, thereby improving resource utilization efficiency and system response speed. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Attached Figure 1 is a flowchart of the method implementation of the first embodiment of the present invention;

[0045] Attached Figure 2 This is a module connection block diagram of the second embodiment of the present invention. DETAILED DESCRIPTION

[0046] In order to make the technical solution, the technical problem solved and the technical effect of the present invention more clearly understood, the technical solution of the present invention is clearly and completely described below in conjunction with specific embodiments.

[0047] Embodiment 1:

[0048] Combined with Figure 1 This embodiment proposes a method for improving the data interaction efficiency of a hyper-converged architecture, including the following operations:

[0049] (1) Introducing a data transmission optimization module into the hyper-converged architecture, which improves data access speed and network efficiency through data pre-fetching and intelligent routing technologies. The specific process includes the following:

[0050] (1.1) Data pre-fetching is achieved by analyzing historical access patterns and current data access requests through data prediction algorithms, predicting the data that may be needed in the future, and caching it in local storage in advance. This operation improves the speed of data interaction by reducing the latency of real-time data requests. Data pre-fetching technology is usually based on statistical models, machine learning models, or deep learning models, and is designed to adapt to the needs of different application scenarios and improve the efficiency and response speed of data access.

[0051] (1.2) During the data transmission process, the network load and the performance of the network path are monitored in real time, and the optimization algorithm (such as Dijkstra algorithm, Bellman-Ford algorithm, etc.) is used to dynamically select the best path for data transmission to achieve intelligent routing. Intelligent routing technology needs to comprehensively consider factors such as network bandwidth, delay, and packet loss rate to optimize the utilization of network bandwidth. In this process, software-defined networking (SDN) technology can be used to flexibly adjust routing strategies.

[0052] (2) Introducing a distributed cache module into the hyper-converged architecture, the distributed cache module reduces the pressure on a single node and ensures accurate data synchronization through data locality optimization and cache consistency management. The specific process includes the following:

[0053] (2.1) Adopting a data caching strategy based on geographic location, caching pre-specified data to storage media that is closer to the computing node, improving access speed and optimizing data locality;

[0054] (2.2) Use distributed consistency algorithms (such as Paxos, Raft, etc.) to ensure the accuracy and synchronization of cached data and implement cache consistency management.

[0055] (3) Introducing a parallel data processing module into the hyper-converged architecture, the parallel data processing module improves data processing efficiency through task parallelization and data sharding technologies, specifically including the following processes:

[0056] (3.1) Task parallelization: Use multi-core processors and multi-threading technology to decompose data processing tasks into multiple subtasks and execute them simultaneously, and use parallel computing frameworks (such as Apache Spark, Hadoop, etc.) to achieve efficient parallel processing;

[0057] (3.2) Data Sharding: Split a large data set into multiple small data pieces, which are processed and stored separately. At the same time, the processing and storage of small data pieces are dynamically adjusted based on the load balancing algorithm to reduce the processing time of a single task and improve data processing efficiency.

[0058] (4) An adaptive load balancing module is introduced into the hyper-converged architecture. The adaptive load balancing module dynamically adjusts resource allocation according to system load and resource usage to achieve a balance between computing, storage, and network loads. The specific process includes the following:

[0059] (4.1) Based on the real-time load of the system, dynamic resource scheduling technology (such as Kubernetes Pod scheduling) is used to automatically adjust the allocation of computing resources and storage resources, realize dynamic resource scheduling, and ensure optimal resource utilization;

[0060] (4.2) By monitoring the system load change trend, use prediction models (such as time series analysis, regression models, etc.) to predict future load changes and make corresponding adjustments. Load prediction and adjustment strategies help avoid system overload and improve system stability.

[0061] Embodiment 2:

[0062] Combined with Figure 2 This embodiment proposes a system for improving the data interaction efficiency of a hyper-converged architecture, which includes:

[0063] Data transmission optimization module, used to improve data access speed and network efficiency through data pre-fetching and intelligent routing technology;

[0064] Distributed cache module, which is used to reduce the pressure on a single node and ensure accurate data synchronization through data locality optimization and cache consistency management;

[0065] Parallel data processing module, used to improve data processing efficiency through task parallelization and data sharding technology;

[0066] The adaptive load balancing module is used to dynamically adjust resource allocation according to system load and resource usage to achieve a balance in computing, storage, and network loads.

[0067] In this embodiment, the data transmission optimization module involved includes:

[0068] The data pre-fetch unit is used to analyze historical access patterns and current data access requests through data prediction algorithms, predict data that may be needed in the future, and cache it in local storage in advance; the data pre-fetch unit improves the data interaction speed by reducing the latency of real-time data requests; the data prediction algorithm is usually based on statistical models, machine learning models or deep learning models, and is designed to adapt to the needs of different application scenarios and improve the efficiency and response speed of data access;

[0069] The intelligent routing unit is used to monitor the network load and the performance of the network path in real time during the data transmission process, and dynamically select the best path for data transmission using optimization algorithms (such as the Dijkstra algorithm and the Bellman-Ford algorithm). The intelligent routing unit needs to comprehensively consider factors such as network bandwidth, delay, and packet loss rate to optimize the utilization of network bandwidth. In this process, the routing strategy can be flexibly adjusted using software-defined networking (SDN) technology.

[0070] In this embodiment, the distributed cache module involved in introduction includes:

[0071] The local optimization unit is used to adopt a data caching strategy based on geographic location to cache pre-specified data in a storage medium that is closer to the computing node, thereby improving access speed and achieving data locality optimization;

[0072] The consistency management unit is used to use distributed consistency algorithms (such as Paxos, Raft, etc.) to ensure the accuracy and synchronization of cached data and implement cache consistency management.

[0073] In this embodiment, the parallel data processing modules involved include:

[0074] Task parallelization unit, which is used to decompose data processing tasks into multiple subtasks and execute them simultaneously by using multi-core processors and multi-threading technology, and to achieve efficient parallel processing with the help of parallel computing frameworks (such as Apache Spark, Hadoop, etc.);

[0075] The data sharding unit is used to divide a large data set into multiple small data slices for separate processing and storage. At the same time, the processing and storage process of small data slices is dynamically adjusted based on the load balancing algorithm to reduce the processing time of a single task and improve data processing efficiency.

[0076] In this embodiment, the adaptive load balancing module includes:

[0077] Dynamic scheduling unit, which is used to automatically adjust the allocation of computing resources and storage resources according to the real-time load of the system using dynamic resource scheduling technology (such as Kubernetes Pod scheduling) to ensure optimal use of resources;

[0078] The prediction and adjustment unit is used to monitor the system load change trend, use prediction models (such as time series analysis, regression models, etc.) to predict future load changes, and make corresponding adjustments. Load prediction and adjustment strategies help avoid system overload and improve system stability.

[0079] In summary, the method and system for improving the data interaction efficiency of a hyper-converged architecture of the present invention can optimize the data transmission mechanism and processing flow, reduce latency, increase throughput, and improve the performance and resource utilization of the overall system.

[0080] The above specific examples are used to explain the principles and implementation methods of the present invention in detail. These examples are only used to help understand the core technical content of the present invention. Based on the above specific embodiments of the present invention, any improvements and modifications made by technicians in this technical field without departing from the principles of the present invention should fall within the scope of patent protection of the present invention.

Claims

1. A method for improving the data interaction efficiency of a hyper-converged architecture, characterized in that: It includes the following operations: (1) Introducing a data transmission optimization module into the hyper-converged architecture, which improves data access speed and network efficiency through data pre-fetching and intelligent routing technology; (2) Introducing a distributed cache module into the hyper-converged architecture, which reduces the pressure on a single node and ensures accurate data synchronization through data locality optimization and cache consistency management; (3) Introducing a parallel data processing module into the hyper-converged architecture, which improves data processing efficiency through task parallelization and data sharding technologies; (4) An adaptive load balancing module is introduced into the hyper-converged architecture. The adaptive load balancing module dynamically adjusts resource allocation according to system load and resource usage to achieve a balance between computing, storage, and network loads.

2. The method for improving the data interaction efficiency of a hyper-converged architecture according to claim 1, characterized in that: The operation (1) specifically includes the following process: The data prediction algorithm analyzes historical access patterns and current data access requests, predicts the data that may be needed in the future, and caches it in local storage in advance to achieve data pre-fetching; During the data transmission process, the network load and the performance of the network path are monitored in real time, and the optimization algorithm is used to dynamically select the best path for data transmission to achieve intelligent routing.

3. The method for improving the data interaction efficiency of a hyper-converged architecture according to claim 1, characterized in that: The operation (2) specifically includes the following process: Adopting a data caching strategy based on geographic location, caching pre-specified data to storage media close to the computing nodes to optimize data locality; Distributed consistency algorithms are used to ensure the accuracy and synchronization of cached data and implement cache consistency management.

4. The method for improving the data interaction efficiency of a hyper-converged architecture according to claim 1, characterized in that: The operation (3) specifically includes the following process: Task parallelization: Use multi-core processors and multi-threading technology to decompose data processing tasks into multiple subtasks and execute them simultaneously, and use the parallel computing framework to achieve efficient parallel processing; Data sharding: Split a large data set into multiple small data pieces, process and store them separately, and dynamically adjust the processing and storage of small data pieces based on the load balancing algorithm.

5. The method for improving the data interaction efficiency of a hyper-converged architecture according to claim 1, characterized in that: The operation (4) specifically includes the following process: According to the real-time system load, dynamic resource scheduling technology is used to automatically adjust the allocation of computing resources and storage resources to achieve dynamic resource scheduling; By monitoring the system load change trend, the prediction model is used to predict future load changes and make corresponding adjustments.

6. A system for improving the data interaction efficiency of a hyper-converged architecture, characterized in that: It includes embedded in the hyper-converged architecture: Data transmission optimization module, used to improve data access speed and network efficiency through data pre-fetching and intelligent routing technology; Distributed cache module, which is used to reduce the pressure on a single node and ensure accurate data synchronization through data locality optimization and cache consistency management; Parallel data processing module, used to improve data processing efficiency through task parallelization and data sharding technology; The adaptive load balancing module is used to dynamically adjust resource allocation according to system load and resource usage to achieve a balance in computing, storage, and network loads.

7. The system for improving data interaction efficiency of a hyper-converged architecture according to claim 6, characterized in that: The data transmission optimization module comprises: A data pre-fetch unit is used to analyze historical access patterns and current data access requests through a data prediction algorithm, predict the data that may be needed in the future, and cache it in local storage in advance; The intelligent routing unit is used to monitor the network load and the performance of the network path in real time during data transmission, and dynamically select the best path for data transmission using an optimization algorithm.

8. The system for improving data interaction efficiency of a hyper-converged architecture according to claim 6, characterized in that: The introduction of the distributed cache module includes: A local optimization unit is used to adopt a data caching strategy based on geographic location to cache pre-specified data in a storage medium that is closer to the computing node, thereby optimizing data locality. The consistency management unit is used to ensure the accuracy and synchronization of cached data using a distributed consistency algorithm.

9. The system for improving data interaction efficiency of a hyper-converged architecture according to claim 6, characterized in that: The parallel data processing module comprises: Task parallelization unit, which is used to decompose data processing tasks into multiple subtasks and execute them simultaneously by using multi-core processors and multi-threading technology, and realize efficient parallel processing with the help of parallel computing framework; The data sharding unit is used to split a large data set into multiple small data slices for separate processing and storage. At the same time, the processing and storage process of small data slices is dynamically adjusted based on the load balancing algorithm.

10. The system for improving data interaction efficiency of a hyper-converged architecture according to claim 6, characterized in that: The adaptive load balancing module includes: Dynamic scheduling unit, which is used to automatically adjust the allocation of computing resources and storage resources according to the real-time load of the system using dynamic resource scheduling technology to ensure the best utilization of resources; The prediction and adjustment unit is used to monitor the system load change trend, use the prediction model to predict future load changes, and make corresponding adjustments.

Images