Large-scale data processing acceleration system and method based on distributed cache

Abstract

The invention discloses a large-scale data processing acceleration system and method based on a distributed cache, and the system comprises a distributed cache module which is used for reading data from the cache; the intelligent prefetching module is used for loading the data blocks into the cache in advance; the intelligent cache module is used for marking the hotspot data and updating the hotspot data so as to ensure that the data in the cache is newest; the parallel processing module is used for task decomposition and task distribution, merging results of the subtasks into a final result, and returning the final result to a user; the dynamic adjustment cache module is used for automatically adjusting data distribution in the cache to maximize the utilization efficiency of the cache; and the dynamic load balancing module is used for monitoring the load and dynamically adjusting a task allocation strategy. According to the method, the overall performance of the system can be remarkably improved, and the data processing efficiency is greatly improved and the user experience is improved on the premise that the data accuracy is not influenced by adopting an intelligent caching mechanism and a parallel processing technology.

Description

Technical Field

[0001] The present invention belongs to the technical field of data processing, and specifically relates to a large-scale data processing acceleration system and method based on distributed cache. Background Art

[0002] With the development of information technology, the amount of data generated by enterprises and organizations is growing exponentially. In the field of business intelligence (BI), how to efficiently process and analyze these large-scale data has become the key to improving decision-making efficiency. However, existing BI systems face many challenges when processing large-scale data, such as slow data loading and long query response time, which seriously affect the user experience and the practicality of the system. Especially when facing PB-level data sets, the traditional single-machine processing method can no longer meet the needs. Therefore, it is particularly important to develop a method that can effectively improve the performance of large-scale data processing.

[0003] The disadvantages of the existing technical solutions are:

[0004] Slow data loading: When processing large-scale data, existing BI systems often face the problem of slow data loading. This is because data is usually stored on disks or remote databases, and each query needs to load data from these storage media, which not only increases I / O overhead, but also prolongs query response time.

[0005] Long query response time: As the amount of data increases, the complexity of the query also increases, resulting in longer query response time. Especially when executing complex queries, such as join operations or aggregation operations involving large amounts of data, the performance of existing BI systems often drops significantly.

[0006] Poor user experience: Due to slow data loading and long query response time, users’ waiting time increases, resulting in poor user experience. Long waiting times may cause users to lose patience and even give up using the BI system. Summary of the invention

[0007] In view of the above problems, the present invention is proposed to provide a large-scale data processing acceleration system and method based on distributed cache to overcome the above problems or at least partially solve the above problems.

[0008] To achieve the above object, the present invention adopts the following technical solutions:

[0009] A large-scale data processing acceleration system based on distributed cache, the system comprising:

[0010] Distributed cache module, used to read data from the cache to reduce data loading time and query response time;

[0011] Intelligent pre-fetch module, used to load data blocks into cache in advance to improve data query efficiency;

[0012] Intelligent cache module, used to mark and update hot data to ensure that the data in the cache is up to date;

[0013] Parallel processing module, used for task decomposition and task allocation, and merges the results of subtasks into the final result and returns it to the user;

[0014] Dynamically adjust the cache module, which is used to automatically adjust the data distribution in the cache to maximize the efficiency of cache utilization;

[0015] Dynamic load balancing module is used to monitor load and dynamically adjust task allocation strategies.

[0016] Optionally, the intelligent pre-fetching module includes:

[0017] A data query unit, used to predict the data blocks that the user is interested in based on historical query records and current query content;

[0018] A data loading unit, used to load the queried data blocks into the cache;

[0019] The data acquisition unit is used to obtain data from the cache to shorten the query response time.

[0020] Optionally, the intelligent cache module includes:

[0021] The data hotspot identification unit automatically identifies frequently accessed data areas by analyzing the user's historical query records and marks them as hotspot data;

[0022] Cache strategy unit, hot data is loaded into the memory cache first, and non-hot data is loaded on demand;

[0023] The cache update unit periodically checks the data in the cache and updates the hot data to ensure that the data in the cache is up to date.

[0024] Optionally, the parallel processing module includes:

[0025] Task decomposition unit, which decomposes complex query tasks into multiple subtasks, each of which can be executed independently;

[0026] The task allocation unit allocates subtasks to multiple processors or nodes for parallel execution, and each processor or node only processes its own subtask;

[0027] The result merging unit, after all subtasks are completed, merges the results of the subtasks into the final result and returns it to the user.

[0028] Optionally, the dynamic cache adjustment module includes:

[0029] An analysis unit, used to regularly analyze the user's query behavior;

[0030] The automatic adjustment unit is used to automatically adjust the data distribution in the cache to ensure that the most frequently accessed data is always kept in the cache.

[0031] Optionally, the dynamic load balancing module includes:

[0032] The load monitoring unit is used to monitor the load of each processor or node in real time and collect load data;

[0033] A dynamic adjustment unit dynamically adjusts the task allocation strategy according to load data to ensure load balancing of each processor or node;

[0034] The fault recovery unit automatically reallocates tasks to other available processors or nodes when the current processor or node fails to ensure the normal operation of the system.

[0035] The present invention also provides a large-scale data processing acceleration method based on distributed cache, the method is based on any of the above-mentioned systems, and the method includes:

[0036] Split the original data into multiple data blocks;

[0037] Distribute data blocks in caches on multiple servers;

[0038] When a user initiates a query request, the relevant data blocks are automatically located according to the query conditions and the data is read from the corresponding cache.

[0039] In summary, due to the adoption of the above technical solution, the beneficial effects of the present invention are:

[0040] 1. The present invention significantly improves the performance of processing large-scale data. Through the intelligent caching mechanism, hot data is loaded into the memory first, reducing the frequency of loading data from the disk or remote database, and significantly shortening the data loading time; parallel processing technology decomposes complex query tasks into multiple subtasks and assigns them to multiple processors or nodes for parallel execution, greatly reducing the query response time.

[0041] 2. The present invention improves the efficiency of data query. Based on historical query records and current query context, it predicts other data blocks that users may be interested in, and loads these data blocks into the cache in advance. In this way, when users really need this data, they can directly obtain it from the cache without having to load it again from the backend database, further shortening the query response time. By reducing data loading time and query response time, the user's waiting time is significantly reduced, improving the overall user experience.

[0042] 3. The design of the present invention dynamically adjusts the cache strategy, so that the system can adapt to changes in user query patterns, ensuring high efficiency in long-term operation. Through the dynamic load balancing mechanism, the task allocation strategy is dynamically adjusted according to the current load of each processor or node, avoiding the situation where some nodes are overloaded while other nodes are idle, ensuring the efficient operation of the system; regularly analyzing the user's query behavior, automatically adjusting the data distribution in the cache, ensuring that the most frequently accessed data is always retained in the cache. This dynamic adjustment mechanism enables the system to adapt to changes in user query patterns and ensures high efficiency in long-term operation. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 A flow chart of a large-scale data processing acceleration system based on distributed cache is provided in an embodiment of the present application. DETAILED DESCRIPTION

[0044] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0045] Embodiment 1:

[0046] See also Figure 1 This embodiment provides a large-scale data processing acceleration system based on distributed cache, the system comprising:

[0047] The distributed cache module is used to read data from the cache to reduce data loading time and query response time.

[0048] The intelligent pre-fetch module is used to load data blocks into the cache in advance to improve data query efficiency.

[0049] The intelligent pre-fetching module comprises:

[0050] The data query unit is used to predict the data blocks that the user may be interested in based on the historical query records and the current query content.

[0051] The data loading unit is used to load the queried data blocks into the cache.

[0052] The data acquisition unit is used to obtain data from the cache to shorten the query response time. When the user really needs this data, it can be immediately obtained from the cache without having to load it again from the backend database, further shortening the query response time.

[0053] The intelligent cache module is used to mark and update hotspot data to ensure that the data in the cache is up to date.

[0054] The intelligent cache module comprises:

[0055] The data hotspot identification unit automatically identifies frequently accessed data areas by analyzing the user's historical query records and marks them as hotspot data.

[0056] Cache strategy unit, hot data is loaded into the memory cache first, and non-hot data is loaded on demand;

[0057] The cache update unit periodically checks the data in the cache and updates the hot data to ensure that the data in the cache is up to date.

[0058] The parallel processing module is used for task decomposition and task allocation, and merges the results of subtasks into the final result and returns it to the user.

[0059] The parallel processing module comprises:

[0060] The task decomposition unit decomposes complex query tasks into multiple subtasks, each of which can be executed independently.

[0061] The task allocation unit allocates subtasks to multiple processors or nodes for parallel execution, and each processor or node only processes its own subtask.

[0062] The result merging unit, after all subtasks are completed, merges the results of the subtasks into the final result and returns it to the user.

[0063] The dynamic cache adjustment module is used to automatically adjust the data distribution in the cache to maximize the cache utilization efficiency.

[0064] The dynamic cache adjustment module includes:

[0065] The analysis unit is used to regularly analyze the user's query behavior.

[0066] The automatic adjustment unit is used to automatically adjust the data distribution in the cache to ensure that the most frequently accessed data is always kept in the cache, thereby maximizing the utilization efficiency of the cache.

[0067] Dynamic load balancing module is used to monitor load and dynamically adjust task allocation strategies.

[0068] The dynamic load balancing module includes:

[0069] The load monitoring unit is used to monitor the load of each processor or node in real time and collect load data.

[0070] The dynamic adjustment unit dynamically adjusts the task allocation strategy according to the load data to ensure the load balance of each processor or node.

[0071] The fault recovery unit automatically reallocates tasks to other available processors or nodes when the current processor or node fails to ensure the normal operation of the system.

[0072] This embodiment significantly improves the performance of the system when processing large-scale data. Through the intelligent caching mechanism, hot data is loaded into the memory first, reducing the frequency of loading data from the disk or remote database, and significantly shortening the data loading time; parallel processing technology breaks down complex query tasks into multiple subtasks and assigns them to multiple processors or nodes for parallel execution, greatly reducing the query response time.

[0073] This embodiment of the present invention improves the efficiency of data query. Based on historical query records and current query context, it predicts other data blocks that the user may be interested in, and loads these data blocks into the cache in advance. In this way, when the user really needs this data, it can be directly obtained from the cache without having to load it again from the backend database, further shortening the query response time. By reducing data loading time and query response time, the user's waiting time is significantly reduced, improving the overall user experience.

[0074] The present embodiment of the present invention dynamically adjusts the cache strategy design so that the system can adapt to changes in user query patterns, ensuring high efficiency in long-term operation. Through the dynamic load balancing mechanism, the task allocation strategy is dynamically adjusted according to the current load of each processor or node, avoiding the situation where some nodes are overloaded while other nodes are idle, ensuring the efficient operation of the system; regularly analyzing the user's query behavior, automatically adjusting the data distribution in the cache, ensuring that the most frequently accessed data is always retained in the cache. This dynamic adjustment mechanism enables the system to adapt to changes in user query patterns and ensures high efficiency in long-term operation.

[0075] Embodiment 2:

[0076] Embodiment 2 discloses a large-scale data processing acceleration method based on distributed cache. The method is based on the above-mentioned large-scale data processing acceleration system based on distributed cache, and the method includes the following steps:

[0077] S1. Divide the original data into multiple data blocks.

[0078] S2: Distribute and store data blocks in caches on multiple servers.

[0079] S3. When a user initiates a query request, the relevant data blocks are automatically located according to the query conditions, and the data is read from the corresponding cache.

[0080] The above contents are further detailed descriptions of the present invention in combination with specific preferred embodiments, and it cannot be determined that the specific implementation of the present invention is limited to these descriptions. For ordinary technicians in the technical field to which the present invention belongs, several simple deductions or substitutions can be made without departing from the concept of the present invention, which should be regarded as falling within the protection scope of the present invention.

Claims

1. A large-scale data processing acceleration system based on distributed cache, characterized in that: The system comprises: Distributed cache module, used to read data from the cache to reduce data loading time and query response time; Intelligent pre-fetch module, used to load data blocks into cache in advance to improve data query efficiency; Intelligent cache module, used to mark and update hot data to ensure that the data in the cache is up to date; Parallel processing module, used for task decomposition and task allocation, and merges the results of subtasks into the final result and returns it to the user; Dynamically adjust the cache module, which is used to automatically adjust the data distribution in the cache to maximize the efficiency of cache utilization; Dynamic load balancing module is used to monitor load and dynamically adjust task allocation strategies.

2. A large-scale data processing acceleration system based on distributed cache as claimed in claim 1, characterized in that: The intelligent pre-fetching module comprises: A data query unit, used to predict the data blocks that the user is interested in based on historical query records and current query content; A data loading unit, used to load the queried data blocks into the cache; The data acquisition unit is used to obtain data from the cache to shorten the query response time.

3. A large-scale data processing acceleration system based on distributed cache as claimed in claim 1, characterized in that: The intelligent cache module comprises: The data hotspot identification unit automatically identifies frequently accessed data areas by analyzing the user's historical query records and marks them as hotspot data; Cache strategy unit, hot data is loaded into the memory cache first, and non-hot data is loaded on demand; The cache update unit periodically checks the data in the cache and updates the hotspot data to ensure that the data in the cache is up to date.

4. A large-scale data processing acceleration system based on distributed cache as claimed in claim 1, characterized in that: The parallel processing module comprises: Task decomposition unit, which decomposes complex query tasks into multiple subtasks, each of which can be executed independently; The task allocation unit allocates subtasks to multiple processors or nodes for parallel execution, and each processor or node only processes its own subtask; The result merging unit, after all subtasks are completed, merges the results of the subtasks into the final result and returns it to the user.

5. A large-scale data processing acceleration system based on distributed cache as claimed in claim 1, characterized in that: The dynamic cache adjustment module includes: An analysis unit, used to regularly analyze the user's query behavior; The automatic adjustment unit is used to automatically adjust the data distribution in the cache to ensure that the most frequently accessed data is always kept in the cache.

6. A large-scale data processing acceleration system based on distributed cache as claimed in claim 1, characterized in that: The dynamic load balancing module includes: The load monitoring unit is used to monitor the load of each processor or node in real time and collect load data; A dynamic adjustment unit dynamically adjusts the task allocation strategy according to load data to ensure load balancing of each processor or node; The fault recovery unit automatically reallocates tasks to other available processors or nodes when the current processor or node fails to ensure the normal operation of the system.

7. A large-scale data processing acceleration method based on distributed cache, the method is based on a large-scale data processing acceleration system based on distributed cache as claimed in any one of claims 1 to 6, characterized in that: The method comprises: Split the original data into multiple data blocks; Distribute data blocks in caches on multiple servers; When a user initiates a query request, the relevant data blocks are automatically located according to the query conditions and the data is read from the corresponding cache.

Images