A method and system for realizing distributed multi-level cache consistency of industrial data

Abstract

The present invention relates to the field of computer engineering technology, and discloses a method and system for realizing distributed multi-level cache consistency of industrial data, including: receiving a data write request, designing a consistency protocol of a multi-level cache and a hierarchical cache elimination strategy; setting a multi-level cache classification and combination method, classifying and storing cache data, using different cache levels to process different cache write operations, dynamically adjusting the cache level and strategy according to the access mode and data characteristics monitored in real time, and updating the database; sending a cache update message to all application nodes, and the nodes receiving the cache update message update the local cache. The multi-level cache system of the present invention effectively improves the overall performance and response speed of the system through flexible cache combination, dynamic adjustment and intelligent preheating, and reduces the impact of first access delay and burst traffic. The multi-level cache consistency protocol ensures data consistency between different cache levels and enhances the reliability of the system.

Description

Technical Field

[0001] The present invention relates to the field of computer engineering technology, and in particular to a method and system for achieving distributed multi-level cache consistency of industrial data. Background Art

[0002] In group-level industrial software, when faced with high concurrency and large data volume processing requirements, a single cache architecture often fails to meet the requirements of system performance and scalability. In order to improve data access speed and reduce database burden, a distributed multi-level cache architecture has become an important solution. However, how to achieve data consistency and efficiency in this multi-level cache architecture is a complex and critical issue.

[0003] Distributed multi-level cache systems usually consist of local caches (such as Ehcache, Caffeine) and remote caches (such as Redis, Memcached). In this architecture, data consistency maintenance, coordination between cache levels, data synchronization mechanisms, and system performance are the main technical challenges. Although existing distributed cache systems have significantly improved performance and scalability, they still have some shortcomings in data consistency and cache management. First, traditional cache consistency protocols such as two-phase commit (2PC) and three-phase commit (3PC), although they can ensure data consistency, have high complexity and performance overhead, and are difficult to meet the needs of high-concurrency scenarios. Summary of the invention

[0004] In view of the above-mentioned problems, the present invention is proposed.

[0005] Therefore, the technical problems solved by the present invention are: how to solve the data consistency problem of distributed multi-level cache in industrial software and coordinate the work between cache levels; how to effectively improve data access speed and reduce the burden on the database.

[0006] In order to solve the above technical problems, the present invention provides the following technical solutions: a method for realizing distributed multi-level cache consistency of industrial data, comprising:

[0007] Receive data write requests, design multi-level cache consistency protocols and hierarchical cache elimination strategies;

[0008] Set up multi-level cache classification and combination methods, classify and store cache data, and use different cache levels to handle different cache write operations.

[0009] Dynamically adjust cache levels and strategies to update databases based on real-time monitored access patterns and data characteristics;

[0010] Send cache update messages to all application nodes. After receiving the cache update messages, the nodes update their local caches.

[0011] As a preferred solution of the method for realizing the consistency of distributed multi-level cache of industrial data described in the present invention, wherein: the method for setting the classification and combination of multi-level cache includes dividing the data cache mode into three types of L1, L2 and L3 according to the data access frequency and update frequency through the random forest classification algorithm;

[0012] Construct a random forest model and use regularization for data feature selection. The formula is expressed as:

[0013] ,

[0014] in, represents the loss function, and represent the actual value and the predicted value respectively. represents the regularization parameter, Indicates The weight of the feature, Indicates the total number of features; Indicates the total number of samples;

[0015] Use the Boosting algorithm to improve classification performance. The formula is:

[0016] ,

[0017] in, Indicates The weights of weak classifiers, Indicates The prediction results of the tree, Represents the total number of weak classifiers;

[0018] After classifying the data based on the data access frequency and update frequency, the data is cached. The request thread cache L1 is used to store the data accessed in this request, which is suitable for data with high frequency of access but short life cycle; the server local cache L2 stores frequently accessed data and is stored in the application server memory. It is used to store data with high access frequency and low update frequency; the remote distributed cache L3 is used to store data with low access frequency, large data volume and high update frequency.

[0019] As a preferred solution of the method for realizing distributed multi-level cache consistency of industrial data described in the present invention, wherein: the classified storage cache data includes strong consistency cache and final consistency cache based on data consistency and delay sensitivity; strong consistency cache ensures that the latest data can be obtained each time it is accessed, and is suitable for data with high consistency requirements, and L2 and L3 caches are used alone; final consistency cache allows a short-term inconsistent state, but eventually reaches a consistent state, and is suitable for data with low consistency requirements, and a combination of L1, L2 and L3 caches is used;

[0020] When the client initiates a data read request, the request first queries the L1 cache. If successful, the data is returned directly; if the L1 cache is unsuccessful, the L2 cache is queried. If successful, the data is returned and the L1 cache is updated; if the L2 cache is unsuccessful, the L3 cache is queried. If successful, the data is returned and the L1 and L2 caches are updated; if the L3 cache is also unsuccessful, the database is queried, the data is written to the L1, L2 and L3 caches, and the data is returned.

[0021] As a preferred solution of the method for realizing distributed multi-level cache consistency of industrial data described in the present invention, wherein: the hierarchical cache elimination strategy includes a cache invalidation strategy combined with TTL, LRU, and LFU elimination algorithms, dynamically managing data based on the cache to realize life cycle management of cache data;

[0022] TTL elimination algorithm for cache items , set the survival time The formula is:

[0023] ,

[0024] in, Represents a cache item The setup time; Indicates the survival time interval; use a scheduled task, every time interval Check the survival time of the cache item. The current time is ,like , then delete the cache item ;

[0025] The LRU elimination algorithm maintains an access timestamp for each cache; each time the cache is accessed, the access timestamp of the corresponding cache item is updated; for the cache item , maintain its access timestamp ; Each time a cache item is accessed When , update its access timestamp:

[0026] ,

[0027] The data structure uses a linked double-ended queue to protect cache items, sorted by access timestamp; each time a cache item is accessed When Move to the head of the queue; when the cache capacity reaches the upper limit When , the cache item at the tail of the queue is eliminated;

[0028] The LFU elimination algorithm maintains an access counter for each cache item. , maintain its access counter ;

[0029] ,

[0030] Use a priority queue to maintain cache items, sorted by access counter; each time a cache item is accessed When When the cache capacity reaches the upper limit When , the cache item with the lowest access count in the priority queue is eliminated;

[0031] The L1 cache uses the LRU strategy; the L2 cache uses the TTL strategy, assisted by the LRU strategy. When the cache capacity reaches the upper limit, the longest unused data is eliminated; the L3 cache uses the LFU strategy, assisted by the TTL strategy, to ensure that the cached data does not reside for a long time.

[0032] As a preferred solution of the method for realizing distributed multi-level cache consistency of industrial data described in the present invention, wherein: the different cache levels process different cache write operations including, in the data write operation, simultaneously updating the caches of all opened cache levels;

[0033] Based on data consistency and latency sensitivity, it is divided into strong consistency cache and final consistency cache; strong consistency cache uses L2 or L3 cache alone; final consistency cache uses L1, L2 or L3 cache combination;

[0034] Based on the data type, it is divided into static data cache and dynamic data cache. Static data uses a combination of L1 and L2 cache; dynamic data uses a combination of L1 and L3 cache;

[0035] For data with high access frequency and low update frequency, the data consistency is the final consistency cache, and the data type is the static data cache, using the L1+L2 combination.

[0036] As a preferred solution of the method for realizing distributed multi-level cache consistency of industrial data described in the present invention, wherein: the dynamic adjustment of cache levels and strategies includes creating a cache dynamic management class for managing dynamic management data of all caches in the system;

[0037] Calculate the weighted access count, calculate the weighted access count of each cache item, and use the exponential decay function to represent the impact of time on access frequency. The formula is expressed as:

[0038] ,

[0039] Calculate the weighted update times, calculate the weighted update times of each cache item, and also use the exponential decay function to represent the effect of time on the update frequency. The formula is:

[0040] ,

[0041] Calculate the cumulative number of visits from the creation time to the last access time. The formula for expressing the persistent heat of the data is:

[0042] ,

[0043] Calculate the product of the number of updates for each cache item, and the data stability formula is expressed as:

[0044] ,

[0045] Add weight and normalization function Adjust the scale of the results, and the final prediction hotspot data formula is expressed as:

[0046] ,

[0047] in, Indicates The number of times a cache item is accessed; Indicates The last access time of a cache item; Indicates The number of times a cache item is updated; Indicates The last update time of the cache item; Indicates The creation time of the cache item; Indicates The number of times a cache item is updated; Indicates The update time of each cache item; represents a time window; Indicates The weight of the cache item; represents the normalization function; represents the attenuation coefficient; An index variable representing the traversal access event; Indicates the total number of access events; Represents the index variable used to traverse all update events; Indicates the total number of update events;

[0048] Based on the predicted hotspot data, the predicted hotspot data is loaded into the L2 and L1 caches in advance before the system starts and traffic peaks.

[0049] Based on the frequency of visits Dynamically adjust the elimination strategy based on the data size. If the access frequency of data increases significantly, its cache level is upgraded from L3 to L2, and then to L1. If the access frequency of a certain data decreases significantly, its cache level is downgraded from L1 to L2, and then to L3.

[0050] Collect and calculate cache hit ratio , access frequency , average data size ;like , then increase the cache capacity, if , then the formula for reducing cache capacity is expressed as:

[0051] ,

[0052] in, Indicates the adjusted cache capacity; Indicates the current cache capacity; Indicates the target cache hit rate; Indicates the current cache hit rate; represents the adjustment factor;

[0053] A list of preheating data is generated during system operation; the formula for temporarily increasing cache capacity is expressed as:

[0054] ,

[0055] in: Indicates the peak cache capacity; Indicates the basic cache capacity; Indicates the current access frequency; represents the capacity adjustment factor;

[0056] At the same time, the data access frequency and pattern in the system are monitored, and the cache hierarchy and strategy are adjusted in real time according to the monitoring data; the number and time of access to each data item are recorded, the access pattern is analyzed, and the high-frequency access data is identified.

[0057] As a preferred solution of the method for realizing the consistency of distributed multi-level cache of industrial data described in the present invention, wherein: the updating of the local cache includes, data access preferentially obtains data from the nearest level, preferentially checks the L1 cache, and returns the data if successful; if the L2 cache is unsuccessful, checks the L3 cache, and updates the L1 and L2 caches and returns the data after success; when writing data, it is executed according to the level switch state, and selectively updates the caches of different levels synchronously or asynchronously;

[0058] When the client initiates a data write request, the request reaches the cache system, first updating the L1 cache. After the L1 cache is updated, a cache update notification is sent to the L2 cache and the L3 cache. After receiving the update notification, the L2 and L3 caches update the cache data and synchronize the version number. Finally, the data is written to the database to ensure that the data in the database is updated.

[0059] The L1 cache is updated synchronously to ensure the fastest response; the L2 and L3 caches use the thread pool to asynchronously update to create cache update messages and send them to all application nodes. After the nodes receive the cache update messages, they update the local L1 and L2 caches.

[0060] A system for realizing distributed multi-level cache consistency of industrial data using any of the methods described in the present invention, wherein:

[0061] The data writing module receives data writing requests, processes cache writing operations at different levels, and designs the consistency protocol and hierarchical cache elimination strategy for multi-level caches.

[0062] The cache management module sets the classification and combination methods of multi-level caches, stores cache data in categories, and uses different cache levels to process different cache write operations;

[0063] The real-time monitoring module dynamically adjusts the cache hierarchy and strategy and updates the database according to the access patterns and data characteristics of real-time monitoring;

[0064] The message passing module is responsible for sending cache update messages to all application nodes. After receiving the cache update messages, the nodes update the local cache and database.

[0065] A computer device comprises: a memory and a processor; the memory stores a computer program, comprising: the steps of implementing any one of the methods of the present invention when the processor executes the computer program.

[0066] A computer-readable storage medium stores a computer program, comprising: when the computer program is executed by a processor, the steps of implementing any one of the methods of the present invention are implemented.

[0067] Beneficial effects of the present invention: 1. The multi-level cache system effectively improves the overall performance and response speed of the system through flexible cache combination, dynamic adjustment and intelligent preheating, and reduces the impact of first access delay and burst traffic. 2. The multi-level cache consistency protocol ensures data consistency between different cache levels and enhances the reliability of the system. 3. The hierarchical cache elimination strategy and dynamic cache strategy optimize the utilization of cache resources and reduce the operating cost of the system. BRIEF DESCRIPTION OF THE DRAWINGS

[0068] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for describing the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work. Among them:

[0069] Figure 1 An overall flow chart of a method for implementing distributed multi-level cache consistency of industrial data provided by the first embodiment of the present invention. DETAILED DESCRIPTION

[0070] In order to make the above-mentioned purposes, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are described in detail below in conjunction with the drawings of the specification. Obviously, the described embodiments are part of the embodiments of the present invention, but not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary persons in the art without creative work should fall within the scope of protection of the present invention.

[0071] Example 1, reference Figure 1 , as an embodiment of the present invention, provides a method for realizing distributed multi-level cache consistency of industrial data, comprising:

[0072] S1: Receive data write requests and design multi-level cache consistency protocols and hierarchical cache elimination strategies.

[0073] Furthermore, the hierarchical cache elimination strategy includes a cache invalidation strategy combined with TTL, LRU, and LFU elimination algorithms, which dynamically manages data based on the cache to achieve lifecycle management of cache data.

[0074] Furthermore, the TTL elimination algorithm has a significant impact on cache items. , set the survival time The formula is:

[0075] ,

[0076] in, Represents a cache item The setup time; Indicates the survival time interval; use a scheduled task, every time interval Check the survival time of the cache item. The current time is ,like , then delete the cache item .

[0077] Furthermore, the LRU elimination algorithm maintains an access timestamp for each cache; each time the cache is accessed, the access timestamp of the corresponding cache item is updated; for the cache item , maintain its access timestamp ; Each time a cache item is accessed When , update its access timestamp:

[0078] ,

[0079] Furthermore, the data structure uses a linked double-ended queue to protect cache items, sorted by access timestamp; each time a cache item is accessed When Move to the head of the queue; when the cache capacity reaches the upper limit When , the cache item at the tail of the queue is eliminated.

[0080] Furthermore, the LFU elimination algorithm maintains an access counter for each cache item. , maintain its access counter , the formula is:

[0081] ,

[0082] Furthermore, a priority queue is used to maintain cache items, sorted by access counter; each time a cache item is accessed When When the cache capacity reaches the upper limit When , the cache item with the lowest access count in the priority queue is eliminated.

[0083] Furthermore, the L1 cache uses the LRU strategy; the L2 cache uses the TTL strategy, assisted by the LRU strategy. When the cache capacity reaches the upper limit, the longest unused data is eliminated; the L3 cache uses the LFU strategy, assisted by the TTL strategy, to ensure that the cached data does not reside for a long time.

[0084] It should be noted that the L1 cache is implemented using ConcurrentHashMap. It is mainly used to store data with extremely high access frequency and has the fastest response speed. The L2 cache is implemented using Ehcache and is used to store data with high access frequency. Compared with the L1 cache, it has a slightly slower response speed but a larger capacity. The L3 remote cache is implemented using Redis, has the largest capacity, and a relatively slow response speed. When the system starts, initialize cache instances at all levels, load necessary cache data, and make decisions and adjustments based on cache dynamic management data.

[0085] S2: Set up multi-level cache classification and combination methods, classify and store cache data, and use different cache levels to handle different cache write operations.

[0086] Furthermore, based on the data access frequency and update frequency, there are three types of caches: L1 is the request context cache, which is mainly used to store the data accessed in this request. It has a very short life span and is suitable for data with high frequency access but short life cycle. L2 is the local cache, which stores frequently accessed data and is stored in the application server memory. It is suitable for data with high access frequency and low update frequency. L3 is the remote distributed cache, such as Redis, which is used to store data with low access frequency but large data volume or high update frequency.

[0087] Furthermore, the Boosting algorithm is used to improve the classification performance. The formula is expressed as:

[0088] ,

[0089] in, Indicates The weights of weak classifiers, Indicates The prediction results of the tree, Represents the total number of weak classifiers.

[0090] Going further, feature selection and regularization are performed, using L1 or L2 regularization for feature selection:

[0091] ,

[0092] in, represents the loss function, and represent the actual value and the predicted value respectively. represents the regularization parameter, Indicates The weight of the feature, Indicates the total number of features; Represents the total number of samples.

[0093] Furthermore, the classified storage cache data includes strong consistency cache and eventual consistency cache based on data consistency and delay sensitivity; strong consistency cache ensures that the latest data can be obtained for each access, which is suitable for data with high consistency requirements, and uses L2 or L3 cache alone; eventual consistency cache allows short-term inconsistent states, but eventually reaches a consistent state, which is suitable for data with low consistency requirements, and uses a combination of L1, L2 or L3 cache.

[0094] Furthermore, when the client initiates a data read request, the request first queries the L1 cache. If successful, the data is returned directly; if the L1 cache is unsuccessful, the L2 cache is queried. If successful, the data is returned and the L1 cache is updated; if the L2 cache is unsuccessful, the L3 cache is queried. If successful, the data is returned and the L1 and L2 caches are updated; if the L3 cache is also unsuccessful, the database is queried, the data is written to the L1, L2 and L3 caches, and the data is returned.

[0095] Furthermore, different cache levels handle different cache write operations including, during a data write operation, simultaneously updating caches of all enabled cache levels.

[0096] Furthermore, based on data consistency and latency sensitivity, it is divided into strong consistency cache and final consistency cache. Strong consistency cache ensures that the latest data can be obtained every time it is accessed. It is suitable for data with high consistency requirements and can use L2 or L3 cache alone; final consistency cache allows short-term inconsistent state, but eventually reaches a consistent state. It is suitable for data with low consistency requirements and can use L1, L2 or L3 cache combination.

[0097] It should be noted that data cache is divided into static data cache and dynamic data cache based on data type. Static data cache stores data that does not change frequently, and can use a combination of L1 and L2 caches; dynamic data cache stores data that changes frequently, and can use a combination of L1 and L3 caches. Different classification methods need to be considered comprehensively for specific usage. For example, in industrial software, the user message list is data with high access frequency and low update frequency, the data consistency is the final consistency cache, and the data type is a static data cache. Comprehensive consideration can use the L1+L2 combination method to ensure fast cache acquisition while realizing cache data updates through message notification cache refresh strategies.

[0098] S3: Dynamically adjust the cache hierarchy and strategy and update the database based on real-time monitored access patterns and data characteristics.

[0099] Furthermore, dynamically adjusting the cache hierarchy and strategy includes creating a cache dynamic management class for managing dynamic management data of all caches in the system.

[0100] Furthermore, the weighted access count is calculated. The weighted access count of each cache item is calculated, and the exponential decay function is used to represent the effect of time on the access frequency. The formula is expressed as:

[0101] ,

[0102] Furthermore, the weighted update times are calculated. The weighted update times of each cache item are calculated. The exponential decay function is also used to represent the effect of time on the update frequency. The formula is expressed as:

[0103] ,

[0104] Furthermore, the cumulative number of visits from the creation time to the last access time is calculated, and the formula for expressing the persistent heat of the data is expressed as:

[0105] ,

[0106] Furthermore, by calculating the product of the number of updates of each cache item, the stability formula of the data is expressed as:

[0107] ,

[0108] Furthermore, adding weight and normalization function Adjust the scale of the results, and the final prediction hotspot data formula is expressed as:

[0109] ,

[0110] in, Indicates The number of times a cache item is accessed; Indicates The last access time of a cache item; Indicates The number of times a cache item is updated; Indicates The last update time of the cache item; Indicates The creation time of the cache item; Indicates The number of times a cache item is updated; Indicates The update time of each cache item; represents a time window; Indicates The weight of the cache item; represents the normalization function; represents the attenuation coefficient; An index variable representing the traversal access event; Indicates the total number of access events; Represents the index variable used to traverse all update events; Indicates the total number of update events;

[0111] Based on the predicted hotspot data, the predicted hotspot data is loaded into the L2 and L1 caches in advance before the system starts and before the traffic peak.

[0112] Furthermore, based on the access frequency The elimination strategy is adjusted dynamically according to the data size. If the access frequency of the data increases significantly, its cache level is promoted from L3 to L2, and then to L1; if the access frequency of a certain data decreases significantly, its cache level is downgraded from L1 to L2, and then to L3.

[0113] Furthermore, the cache hit rate is collected and calculated. , access frequency , average data size ;like , then increase the cache capacity, if , then the formula for reducing cache capacity is expressed as:

[0114] ,

[0115] in, Indicates the adjusted cache capacity; Indicates the current cache capacity; Indicates the target cache hit rate; Indicates the current cache hit rate; Represents the adjustment factor.

[0116] Furthermore, a list of preheating data is generated during system operation; the formula for temporarily increasing the cache capacity is expressed as:

[0117] ,

[0118] in: Indicates the peak cache capacity; Indicates the basic cache capacity; Indicates the current access frequency; Indicates the capacity adjustment factor.

[0119] Furthermore, the data access frequency and pattern in the system are monitored simultaneously, and the cache hierarchy and strategy are adjusted in real time based on the monitoring data; the number of accesses and time of each data item are recorded, the access pattern is analyzed, and the high-frequency access data is identified.

[0120] It should be noted that the UnifiedCacheManager class is implemented, which is responsible for implementing all cache-related operations. The cache is queried in sequence according to the cache level priority, and data read and write operations are performed. Enumerations or constants are used to define cache levels (L1, L2, L3), and the enabled state of each cache level is defined in the configuration file. The operations of different cache levels are determined based on the switch state of the corresponding level.

[0121] S4: Send cache update message to all application nodes. After receiving the cache update message, the node updates the local cache.

[0122] Furthermore, the updating of the local cache includes, when accessing data, first obtaining data from the nearest level, first checking the L1 cache, and returning the data if successful; if the L2 cache is unsuccessful, then checking the L3 cache, and after success, updating the L1 and L2 caches and returning the data; when writing data, executing according to the level switch state, selectively updating the caches of different levels synchronously or asynchronously.

[0123] Furthermore, when the client initiates a data write request, the request reaches the cache system, first updating the L1 cache. After the L1 cache is updated, a cache update notification is sent to the L2 cache and the L3 cache. After receiving the update notification, the L2 and L3 caches update the cache data and synchronize the version number. Finally, the data is written to the database to ensure that the data in the database is updated.

[0124] Furthermore, the L1 cache is updated synchronously to ensure the fastest response; the L2 and L3 caches use thread pools to asynchronously update to create cache update messages and send them to all application nodes. The nodes receive the cache update messages and update the local L1 and L2 caches.

[0125] Furthermore, there are three types of data consistency strategies, corresponding to cache usage scenarios with different consistency requirements: Write-through: Write operations are first written to the cache, and then synchronously written to the database to ensure strong data consistency. Write-back: Write operations are first written to the cache, and then asynchronously written to the database, which improves write performance but may have short-term inconsistencies. Invalidation notification: When data is updated, other cache nodes are notified through the message queue to invalidate the corresponding cache data to ensure eventual consistency.

[0126] Furthermore, the cache update notification mechanism is configured. When the data is updated in the database, the remote cache is updated synchronously and a message queue notification is initiated. The specific implementation is that L1 only updates the cache of the current request thread, and when the L2 cache is updated, the message queue (such as Kafka, RabbitMQ) is used to notify other nodes to update the cache. The L3 cache only calls the cache update method. At this point, the cache of all service nodes is updated.

[0127] It should be noted that in data write operations, all caches with enabled cache levels are updated simultaneously. The L1 cache is mainly for fast access to high-frequency data, so in write operations, it needs to be updated synchronously to ensure the fastest response time. The L2 cache usually stores high-frequency access data, and the consistency and persistence of the cache need to be considered. In order to reduce the blocking of the main thread by write operations, a thread pool is used for asynchronous updates. After updating the L2 cache, other application nodes are notified through the message queue to ensure that the L2 cache data of all nodes remains consistent. The L3 cache stores a large amount of data, and network latency and cache consistency issues need to be considered. A thread pool is also used for asynchronous updates. After updating the L3 cache, all application nodes are notified through the message queue to ensure the consistency of global cache data.

[0128] On the other hand, this embodiment also provides a system for implementing distributed multi-level cache consistency of industrial data, which includes:

[0129] The data write module receives data write requests, processes cache write operations at different levels, and designs multi-level cache consistency protocols and hierarchical cache elimination strategies.

[0130] The cache management module sets the classification and combination methods of multi-level cache, stores cache data in categories, and uses different cache levels to process different cache write operations.

[0131] The real-time monitoring module dynamically adjusts the cache level and strategy and updates the database according to the real-time monitored access patterns and data characteristics.

[0132] The message passing module is responsible for sending cache update messages to all application nodes. After receiving the cache update messages, the nodes update the local cache and database.

[0133] If the above functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc., which can store program code.

[0134] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute instructions), or in conjunction with such instruction execution systems, devices or apparatuses. For the purposes of this specification, "computer-readable medium" can be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in conjunction with such instruction execution systems, devices or apparatuses.

[0135] More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection with one or more wires (electronic device), a portable computer disk case (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), an optical fiber device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be a paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, deciphering or, if necessary, processing in another suitable manner, and then stored in a computer memory.

[0136] It should be understood that the various parts of the present invention can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, multiple steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0137] Example 2: The following is an embodiment of the present invention, which provides a method for achieving distributed multi-level cache consistency of industrial data. In order to verify the beneficial effects of the present invention, scientific demonstration is carried out through economic benefit calculation and simulation experiments.

[0138] In order to verify the effectiveness of the method of realizing the consistency of distributed multi-level cache of industrial data in the present invention, an experiment on an industrial data processing system was conducted. The purpose of this experiment is to receive data write requests, design a consistency protocol for multi-level cache and a hierarchical cache elimination strategy, set up multi-level cache classification and combination methods, classify and store cache data, use different cache levels to process different cache write operations, and evaluate the performance of the system under different access modes. A simulated industrial data processing system was used in the experiment. The system was deployed in a cluster containing multiple servers and adopted a three-layer cache architecture (L1, L2, L3).

[0139] In the experimental preparation stage, the three-layer cache system was initialized first. L1 cache is implemented using ConcurrentHashMap to store data with high frequency access but short life cycle; L2 cache is implemented using Ehcache to store data with high frequency access and low update frequency; L3 cache is implemented using Redis to store data with low access frequency but large data volume and high update frequency. The system receives data write requests, processes data according to the strategies of different cache levels, and ensures data consistency and optimal use of cache resources through the designed consistency protocol and layered cache elimination strategy.

[0140] During the experiment, the system processes data according to the following steps:

[0141] When the client initiates a data write request, the data is first written to the L1 cache, and then a cache update notification is sent to the L2 and L3 caches to ensure that the data is updated synchronously at all cache levels.

[0142] According to the access frequency and update frequency of data, the system classifies and stores data in different cache levels. High-frequency access data is stored in L1 cache, frequently accessed data is stored in L2 cache, and low-frequency access but large data volume data is stored in L3 cache.

[0143] The system monitors the access patterns and characteristics of data in real time and dynamically adjusts the cache hierarchy and strategy based on the monitoring data. For example, when the access frequency of a certain data suddenly increases, the system upgrades it from L3 to L2 or even L1 cache to cope with sudden high-concurrency requests.

[0144] Database update: After the cache data is updated, the system writes the data to the database to ensure data persistence. Cache update message: The system sends a cache update message to all application nodes. After receiving the cache update message, the node updates the local cache to ensure data consistency.

[0145] Table 1. Display of some experimental data

[0146] ,

[0147] As can be seen from the data in Table 1, the table lists the access frequency and update frequency of different data types. For data with high access frequency and low update frequency (such as data types 1, 2, and 3), the system can dynamically adjust the cache hierarchy to store it in L1 or L2 cache first, thereby improving the cache hit rate and reducing the response time. For example, the access frequency of data type 1 is 50 times / second and the update frequency is 1 time / second. The system stores it in the L1 cache, and the cache hit rate reaches 90%, and the response time is only 20 milliseconds, showing a significant performance improvement.

[0148] For data with high access frequency, the present invention effectively improves the cache hit rate by designing a multi-level cache consistency protocol and a hierarchical cache elimination strategy. The access frequency of data type 3 in the table is 200 times / second. Although the update frequency is 10 times / second, the system can still maintain its hit rate at 80% through real-time monitoring and dynamic adjustment of strategies, and can still maintain good performance under high concurrency.

[0149] Data types 4 and 5 have low access frequencies, but the data size is small and sensitive to latency. The system stores them in the L1 cache and ensures cache hit rates of 95% and 98% respectively through real-time monitoring and adjustment of strategies. The response time is only 15 milliseconds and 10 milliseconds, significantly reducing access latency.

[0150] For data with strong consistency requirements (such as data types 2, 3, and 7), the system uses L2 or L3 cache alone and combines it with the cache update message mechanism to ensure that the latest data can be obtained every time it is accessed. For example, the access frequency of data type 2 is 100 times / second and the update frequency is 5 times / second. The system stores it in the L2 cache and uses the TTL and LRU strategies to achieve a cache hit rate of 85% and a response time of 25 milliseconds, ensuring data consistency and efficient access.

[0151] Table 2 Comparative experimental table

[0152] ,

[0153] From the data in Table 2, it can be seen that the cache hit rate of our invention is significantly higher than that of the prior art. The cache hit rate of the prior art is 75%, while the cache hit rate of our invention reaches 90%, an increase of 15%. Compared with the prior art, our invention has shown significant innovation and novelty in many aspects, and has high practical value and promotion prospects.

[0154] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present invention may be modified or replaced by equivalents without departing from the spirit and scope of the technical solutions of the present invention, which should all be included in the scope of the claims of the present invention.

Claims

1. A method for realizing distributed multi-level cache consistency of industrial data, characterized in that: include: Receive data write requests, design multi-level cache consistency protocols and hierarchical cache elimination strategies; Set up multi-level cache classification and combination methods, classify and store cache data, and use different cache levels to handle different cache write operations; Dynamically adjust cache levels and strategies to update databases based on real-time monitored access patterns and data characteristics; Send cache update messages to all application nodes. After receiving the cache update messages, the nodes update their local caches. The method for setting the multi-level cache classification and combination includes classifying the data cache modes into three types: L1, L2, and L3 according to the data access frequency and update frequency through a random forest classification algorithm; Construct a random forest model and use regularization for data feature selection. The formula is expressed as: Among them, L(w) represents the loss function, y i and represent the actual value and the predicted value respectively, λ represents the regularization parameter, and w j represents the weight of the jth feature, P represents the total number of features; N represents the total number of samples; Use the Boosting algorithm to improve classification performance. The formula is: in, represents the weight of the m0th weak classifier, represents the prediction result of the m0th tree, and M represents the total number of weak classifiers; After classifying data based on data access frequency and update frequency, data caching is performed. The request thread cache L1 is used to store data accessed in this request, which is suitable for data with high frequency of access but short life cycle; the server local cache L2 stores frequently accessed data and is stored in the application server memory. It is used to store data with high access frequency and low update frequency; the remote distributed cache L3 is used to store data with low access frequency, large data volume and high update frequency; The hierarchical cache elimination strategy includes a cache invalidation strategy combined with TTL, LRU, and LFU elimination algorithms to dynamically manage data based on the cache to achieve lifecycle management of cache data; TTL elimination algorithm for cache item C i , set the survival time TTL (C i )The formula is expressed as: TTL(C i )=T s +Δ t Among them, T s Represents cache item C i The setup time of Δ t Indicates the survival time interval; use a scheduled task, every time interval Δ t Check the survival time of the cache item, the current time is T c , if T c ≥TTL(C i ), then delete cache item C i ; The LRU elimination algorithm maintains an access timestamp for each cache; each time the cache is accessed, the access timestamp of the corresponding cache item is updated; for cache item C i , maintain its access timestamp T a (C i ); Each time cache item C is accessed i When , update its access timestamp: T a (C i )=T c The data structure uses a linked double-ended queue to protect cache items, sorted by access timestamp; each time a cache item C is accessed i When C i Move to the head of the queue; when the cache capacity reaches the upper limit S, eliminate the cache items at the tail of the queue; The LFU elimination algorithm maintains an access counter for each cache. i , maintain its access counter F(C i ); F(C i )=F(C i )+1 Use a priority queue to maintain cache items, sorted by access counter; each time you access cache item C i When C in the priority queue is updated i When the cache capacity reaches the upper limit S, the cache item with the lowest access count in the priority queue is eliminated; The L1 cache uses the LRU strategy; the L2 cache uses the TTL strategy, assisted by the LRU strategy. When the cache capacity reaches the upper limit, the longest unused data is eliminated; the L3 cache uses the LFU strategy, assisted by the TTL strategy, to ensure that the cached data does not reside for a long time; The dynamic adjustment of cache levels and strategies includes creating a cache dynamic management class for managing dynamic management data of all caches in the system; Calculate the weighted access count, calculate the weighted access count of each cache item, and use the exponential decay function to represent the impact of time on access frequency. The formula is expressed as: Calculate the weighted update times, calculate the weighted update times of each cache item, and also use the exponential decay function to represent the effect of time on the update frequency. The formula is: Calculate the cumulative number of visits from the creation time to the last access time. The formula for expressing the persistent heat of the data is: Calculate the product of the number of updates for each cache item, and the data stability formula is expressed as: Add weight w i The scale of the result is adjusted by the normalization function φ, and the final prediction hotspot data formula is expressed as: Among them, A i represents the number of accesses to the i-th cache item; T a,i represents the last access time of the i-th cache item; W U,i represents the update times of the i-th cache item; T u,i Indicates the last update time of the i-th cache item; T c,i Indicates the creation time of the i-th cache item; U k represents the update times of the kth cache item; T u,k represents the update time of the kth cache item; Δt represents the time window; w i represents the weight of the i-th cache item; φ represents the normalization function; λ0 represents the decay coefficient; j represents the index variable for traversing access events; n represents the total number of access events; k represents the index variable for traversing all update events; m represents the total number of update events; Based on the predicted hotspot data, the predicted hotspot data is loaded into the L2 and L1 caches in advance before the system starts and traffic peaks. Dynamically adjust the elimination strategy based on the access frequency F and data size. If the access frequency of data increases significantly, its cache level is upgraded from L3 to L2, and then to L1. If the access frequency of a certain data decreases significantly, its cache level is downgraded from L1 to L2, and then to L3. Collect and calculate cache hit rate H, access frequency F, and average data size If H <H min , then increase the cache capacity, if H>H max , then the formula for reducing cache capacity is expressed as: C new =C current ×(1+k0×(H target -H)) Among them, C new Indicates the adjusted cache capacity; C current Indicates the current cache capacity; H target represents the target cache hit rate; H represents the current cache hit rate; k0 represents the adjustment coefficient; A list of preheating data is generated during system operation; the formula for temporarily increasing cache capacity is expressed as: C peak =C base +β×F Where: C peak represents the peak cache capacity; C base represents the basic cache capacity; F represents the current access frequency; β represents the capacity adjustment coefficient; At the same time, the data access frequency and pattern in the system are monitored, and the cache hierarchy and strategy are adjusted in real time according to the monitoring data; the number and time of access to each data item are recorded, the access pattern is analyzed, and the high-frequency access data is identified.

2. The method for realizing distributed multi-level cache consistency of industrial data according to claim 1, characterized in that: The classified storage cache data includes being divided into strong consistency cache and final consistency cache based on data consistency and delay sensitivity; Strong consistency cache ensures that the latest data can be obtained every time it is accessed. It is suitable for data with high consistency requirements and uses L2 and L3 caches alone. Eventual consistency cache allows short-term inconsistent states but eventually reaches a consistent state. It is suitable for data with low consistency requirements and uses a combination of L1, L2, and L3 caches. When the client initiates a data read request, the request first queries the L1 cache. If successful, the data is returned directly. If the L1 cache is unsuccessful, the L2 cache is queried. If successful, the data is returned and the L1 cache is updated. If the L2 cache is unsuccessful, the L3 cache is queried. If successful, the data is returned and the L1 and L2 caches are updated. If the L3 cache is also unsuccessful, the database is queried, the data is written to the L1, L2 and L3 caches, and the data is returned.

3. The method for realizing distributed multi-level cache consistency of industrial data according to claim 2, characterized in that: The different cache levels processing different cache write operations include, in a data write operation, simultaneously updating caches of all enabled cache levels; Based on data consistency and latency sensitivity, it is divided into strong consistency cache and eventual consistency cache; Strongly consistent cache uses L2 or L3 cache alone; eventually consistent cache uses a combination of L1, L2 or L3 cache; Based on the data type, it is divided into static data cache and dynamic data cache. Static data uses a combination of L1 and L2 cache; dynamic data uses a combination of L1 and L3 cache; For data with high access frequency and low update frequency, the data consistency is the final consistency cache, and the data type is the static data cache, using the L1+L2 combination.

4. The method for realizing distributed multi-level cache consistency of industrial data according to claim 3, characterized in that: The updating of the local cache includes, data access preferentially obtains data from the nearest level, preferentially checks the L1 cache, and returns the data if successful; if the L2 cache is unsuccessful, checks the L3 cache, and if successful, updates the L1 and L2 caches and returns the data; When writing data, it is executed according to the level switch status, and caches at different levels are updated synchronously or asynchronously selectively; When the client initiates a data write request, the request reaches the cache system, first updating the L1 cache. After the L1 cache is updated, a cache update notification is sent to the L2 cache and the L3 cache. After receiving the update notification, the L2 and L3 caches update the cache data and synchronize the version number. Finally, the data is written to the database to ensure that the data in the database is updated. The L1 cache is updated synchronously to ensure the fastest response; the L2 and L3 caches use the thread pool to asynchronously update to create cache update messages and send them to all application nodes. The nodes receive the cache update messages and update the local L1 and L2 caches.

5. A system for realizing distributed multi-level cache consistency of industrial data using the method according to any one of claims 1 to 4, characterized in that: The data writing module receives data writing requests, processes cache writing operations at different levels, and designs the consistency protocol and hierarchical cache elimination strategy for multi-level caches. The cache management module sets the classification and combination methods of multi-level caches, stores cache data in categories, and uses different cache levels to process different cache write operations; The real-time monitoring module dynamically adjusts the cache hierarchy and strategy and updates the database according to the access patterns and data characteristics of real-time monitoring; The message passing module is responsible for sending cache update messages to all application nodes. After receiving the cache update messages, the nodes update the local cache and database.

6. A computer device comprising: A memory and a processor; the memory stores a computer program, characterized in that: when the processor executes the computer program, the steps of the method for realizing distributed multi-level cache consistency of industrial data as described in any one of claims 1-4 are implemented.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method for distributed multi-level cache consistency of industrial data as described in any one of claims 1 to 4 are implemented.

Images