Method for refreshing exclusive cache in multi-cluster scene
By adopting process-level, application unit-level and data center-level exclusive cache refresh methods in multi-cluster scenarios, combined with identification files and ETL tools, the fine-grained and single-point problems of cache management in the existing technology are solved, and the problem of "single-grained" cache management in the existing technology is realized. Efficient and real-time cache updates and data consistency improve system performance.
Patent Information
- Application Number
- CN202411086148.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-08
- Publication Date
- 2024-11-22
AI Technical Summary
The existing multi-cluster cache refresh technology lacks fine-grained management, which is prone to single-point problems, and the complexity of distributed consistency protocols leads to high performance overhead, making it difficult to apply in large-scale and high-concurrency environments.
Adopt process-level, application unit-level and data center-level exclusive cache refresh method, monitor cache updates through identification files and timestamps, and use ETL tools to achieve cross-data center data synchronization to avoid data synchronization between multiple nodes and single point of failure. Batch synchronization and cache information batch loading strategy.
One-click batch update of cache is realized, which improves the accuracy and real-time updates, provides a fine-grained control strategy, reduces system risks, avoids single-point problems, and improves data consistency and system performance.
Smart Images

Figure CN119003549A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the technical field of cache management, and in particular relates to an exclusive cache refresh method in a multi-cluster scenario. Background Art
[0002] Distributed computing, as a technology to improve computing power and process large-scale data, has developed rapidly in the past few decades. With the rise of cloud computing and big data technology, distributed multi-cluster systems are widely used in online services, data analysis and other fields. In these systems, cache technology is crucial to improving data access speed and system performance. However, in a multi-cluster environment, due to the complexity of data distribution and access patterns, while caching improves the read performance of the system, it also introduces consistency issues between the database and the cache. In the actual business logic processing process, when the data in the database changes, the data in the cache needs to be modified synchronously to avoid data staleness or inconsistency.
[0003] Existing multi-cluster cache refresh technologies mainly rely on centralized cache managers or distributed algorithms to synchronize cache data on each node. These technologies include but are not limited to centralized cache management and cache management based on distributed consistency protocols. Although the existing technologies have made certain progress in cache management and consistency, they still have shortcomings: (1) Lack of fine-grained management of cache: Most of the existing technical solutions are aimed at cache update strategies at the same level. The technology lacks sufficient flexibility and is difficult to adapt to the specific needs of different application scenarios. There is a lack of flexible and customized management for cache synchronization between different processes, different application units and different data centers. (2) Single point problem of centralized cache management: As a single point, the centralized cache manager is prone to become a performance bottleneck, especially when there are high concurrent requests. The risk of single point failure is high. Once the manager has problems, the stability and availability of the entire cache system will be affected. As the cluster size grows, the existing technology may be difficult to scale linearly to adapt to more nodes and higher loads. (3) The complexity of using distributed consistency protocols to manage cache: Although distributed consistency protocols such as Raft or Paxos can ensure data consistency, the protocol itself is complex and difficult to implement. In large-scale clusters, these protocols may cause significant performance overhead because they require frequent communication and coordination between all nodes. In addition, in some scenarios, cache data synchronization is required in clusters across data centers and across regions. Due to network latency and the complexity of communication between nodes, this will lead to instability of the consistency protocol and is easily affected by network communication. These shortcomings limit the application of the technology in large-scale, high-concurrency environments. For this reason, an exclusive cache refresh method in multi-cluster scenarios is proposed. Summary of the invention
[0004] The purpose of the present invention is to provide an exclusive cache refresh method in a multi-cluster scenario, aiming to solve the problems raised in the above background technology.
[0005] To achieve the above object, the present invention provides the following technical solutions:
[0006] An exclusive cache refresh method in a multi-cluster scenario includes the following steps:
[0007] Step 1: Refresh the process-level cache;
[0008] Step 2: refresh the application unit level cache;
[0009] Step 3: Refresh the data center level cache.
[0010] Furthermore, the specific process of step 1 is: before each process executes a specific business logic process, it reads the latest modification timestamp of a specific identification file. If the modification timestamp of the file is greater than the timestamp previously recorded by the current process, the existing cache content is cleared, and the latest value is read from the database again during subsequent business logic processing to update the cache between processes.
[0011] Furthermore, the specific process of step 2 is: setting a background timer task to regularly read the refresh flag in the database; when a flag that needs to be refreshed is detected, batch modifying the modification timestamps of specific files; after reading the file timestamp update, the process within the entire application unit updates the original cache information to achieve unit-level cache refresh.
[0012] Furthermore, when an identification bit that needs to be refreshed is detected, in the step of batch modifying the modification time of specific files, if the data in the database is modified, the identification bit will be modified synchronously. The background scheduled task determines whether the current application unit needs to be refreshed by reading the identification bit in the database. When an identification bit that needs to be refreshed is detected, the background scheduled task batch modifies the modification time of specific files.
[0013] Furthermore, the specific process of step 3 is: when the identification bit of a single data center is modified, ETL is used to synchronize the latest modified database identification bit across centers to other data centers based on the database modification timestamp, and the cache refresh scheduled tasks of other data centers read the identification bit and perform batch cache cleanup.
[0014] Furthermore, in step 3, the database is provided with a plurality of refresh flags, and the refresh strategies of the plurality of application units are controlled by the plurality of refresh flags.
[0015] Furthermore, during the cache refresh process, the database identification bits may be synchronized in batches, and only some application units are refreshed, while the caches of other units remain unchanged.
[0016] Furthermore, during the process of loading the cache, the cache information is loaded in batches.
[0017] Compared with the prior art, the present invention has the following beneficial effects:
[0018] 1. The present invention proposes a hierarchical cache refresh strategy, which can achieve one-click batch update of cache at different levels, from process level, application unit level to data center level. In specific implementation, the present invention innovatively uses identification files and timestamps to monitor cache update requirements, improving the accuracy and real-time performance of cache updates. In addition, data synchronization across data centers is achieved through ETL tools, solving the problem of data consistency in distributed systems.
[0019] 2. The present invention proposes a fine-grained cache control strategy that achieves precise control of cache updates by refreshing specific file identification bits. This strategy allows the system to perform cache updates only for a portion of application units by modifying the identification bits of specific files, thereby achieving grayscale release and testing. By updating the cache in stages and batches and dynamically adjusting the refresh strategy, updates can be gradually applied to more application units, achieving a smooth transition and reducing system risks.
[0020] 3. The present invention transforms the traditional distributed multi-node common cache management into an exclusive cache management strategy, where each process, application unit, and data center manages its own cache content. This cache management method does not require data synchronization between multiple nodes in the cluster, and also avoids the single point problem in centralized cache management, which can improve data consistency, reduce resource conflicts, and enhance system performance. BRIEF DESCRIPTION OF THE DRAWINGS
[0021] Figure 1 It is the overall flow chart of the present invention.
[0022] Figure 2 Flow chart of step 1 in the present invention.
[0023] Figure 3 This is a flow chart of step 2 in the present invention.
[0024] Figure 4 Flow chart of step 3 in the present invention. DETAILED DESCRIPTION
[0025] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.
[0026] The specific implementation of the present invention is described in detail below in conjunction with specific embodiments.
[0027] In the scenario where multiple processes execute business logic tasks, each process has its own unique local cache. If the cached database is updated, each process needs to update the cached data. In a distributed scenario, when the configuration information of the application system is modified, the cache configuration information between multiple processes, multiple application units, and even different data centers may need to be updated synchronously. For different levels of cache information, the present invention sets a one-click refresh method for multi-level caches between different processes, different business processing units, and different data centers. The overall cache refresh strategy can be divided into process-level cache refresh, application unit-level cache refresh, and data center-level cache refresh from the bottom up. The overall process is as follows: Figure 1 shown.
[0028] An embodiment of the present invention provides an exclusive cache refresh method in a multi-cluster scenario, comprising the following steps:
[0029] Step 1: Process-level cache refresh ( Figure 2 ): The specific business in the present invention is scheduled and processed through multiple processes. Since operating system resources cannot be shared between processes, in order to further ensure the consistency of caches between different processes, the present invention introduces an identification file to determine whether the cache of the process needs to be refreshed. Before each process executes a specific business logic process, it reads the latest modification timestamp of a specific identification file. If the modification timestamp of the file is greater than the timestamp previously recorded by the current process, it means that the currently recorded cache is already an old value, and the existing cache content will be cleared. In the subsequent business logic processing process, the latest value will be read from the database again to update the cache between processes.
[0030] Step 2: Application unit level cache refresh ( Figure 3 ): For cache refresh at different unit levels, the present invention schedules the cache refresh timer task through the identification bit of the database by means of background scheduling timer tasks, and the background timer task periodically reads the refresh flag bit in the database. If the data in the database is modified, the identification bit will be modified synchronously. The background timer task determines whether the current application unit needs to be refreshed through the identification bit. When the identification bit that needs to be refreshed is detected, the background timer task batch modifies the modification time of specific files, so that the process in the entire application unit updates the original cache information after reading the file timestamp update, achieving the unit-level cache refresh.
[0031] Step 3: Data center cache refresh ( Figure 4): In some scenarios, business processing spans across multiple data centers, and each data center can carry business processing at the same time. When data is modified in one of the data centers, it needs to be synchronized to other data centers in a timely manner. In order to enable one-click cache refresh, the present invention synchronizes data between data centers for the refresh flag of the database to achieve one-click refresh of the cache between data centers. In specific implementation, the present invention uses the data synchronization tool ETL, and its underlying layer performs cross-center synchronization of the latest modified data based on the database modification timestamp. When the flag of a single data center is modified, it will be automatically synchronized to other data centers through ETL, and the cache refresh timed tasks of other data centers read the flag, and batch cache cleaning is completed by executing the timed tasks, so that the cache information can be re-read during the next business logic processing. The present invention designs the refresh flag of the database into multiple forms, and the refresh strategies of multiple application units can be controlled by multiple refresh flags. During the cache update process, you can choose to synchronize the database flags in batches, only refresh some application units, and keep the cache of other units unchanged, so as to achieve the purpose of grayscale release and avoid the risks that may be caused by global cache updates.
[0032] In the embodiment of the present invention, for the data synchronization between different data centers, the present invention adopts the database identification bit synchronization method based on ETL, and other data synchronization methods may also be adopted, such as data center-level data synchronization strategy based on message queues and CDC, etc. For different data identification bit methods, the present invention adopts a method based on identification file modification timestamp, and other file-specific identification methods may also be adopted, such as file name, file size, etc.
[0033] As a preferred embodiment of the present invention, in the specific process of cache loading, the cache information is loaded in batches. Different logical messages require different cache information. The process currently executing the business logic only loads the cache information required by the current business logic. By gradually loading the cache, the processing efficiency of the business logic is improved while reducing system overhead.
[0034] The above are only preferred embodiments of the present invention. It should be pointed out that, for those skilled in the art, several modifications and improvements can be made without departing from the concept of the present invention. These should also be regarded as the protection scope of the present invention. These will not affect the effect of the implementation of the present invention and the practicality of the patent.
Claims
1. An exclusive cache refresh method in a multi-cluster scenario, characterized in that: The following steps are involved: Step 1: Refresh the process-level cache; Step 2: refresh the application unit level cache; Step 3: Refresh the data center level cache.
2. The exclusive cache refresh method in a multi-cluster scenario according to claim 1, characterized in that: The specific process of step 1 is: before each process executes a specific business logic process, it reads the latest modification timestamp of a specific identification file. If the modification timestamp of the file is greater than the timestamp previously recorded by the current process, the existing cache content is cleared, and the latest value is read from the database again during the subsequent business logic processing to update the cache between processes.
3. The exclusive cache refresh method in a multi-cluster scenario according to claim 1, characterized in that: The specific process of step 2 is: setting a background timer task to regularly read the refresh flag in the database; when a flag that needs to be refreshed is detected, batch modifying the modification timestamps of specific files; after reading the file timestamp update, the process within the entire application unit updates the original cache information to achieve unit-level cache refresh.
4. The exclusive cache refresh method in a multi-cluster scenario according to claim 3, characterized in that: When an identification bit that needs to be refreshed is detected, in the step of batch modifying the modification time of specific files, if the data in the database is modified, the identification bit will be modified synchronously. The background scheduled task determines whether the current application unit needs to be refreshed by reading the identification bit in the database. When an identification bit that needs to be refreshed is detected, the background scheduled task batch modifies the modification time of specific files.
5. The exclusive cache refresh method in a multi-cluster scenario according to claim 1, characterized in that: The specific process of step 3 is as follows: when the identification bit of a single data center is modified, ETL is used to synchronize the latest modified database identification bit across centers to other data centers based on the database modification timestamp, and the cache refresh scheduled tasks of other data centers read the identification bit and perform batch cache cleanup.
6. The exclusive cache refresh method in a multi-cluster scenario according to claim 5, characterized in that: In the step 3, the database is provided with a plurality of refresh flags, and the refresh strategies of the plurality of application units are controlled by the plurality of refresh flags.
7. The exclusive cache refresh method in a multi-cluster scenario according to claim 5, characterized in that: During the cache refresh process, you can choose to synchronize the database flags in batches, refreshing only some application units while keeping the caches of other units unchanged.
8. The exclusive cache refresh method in a multi-cluster scenario according to claim 1, characterized in that: During the process of cache loading, cache information is loaded in batches.
