Data caching method, device and storage medium

Abstract

The present invention discloses a data caching method, device, and storage medium. When a first node obtains data to be cached based on a data acquisition request, the first node determines a caching result based on the data to be cached. When the caching result is to cache the data to be cached in a corresponding second-level cache, the first node caches identification information of the data to be cached in a third-level cache. Multiple nodes that receive the data acquisition request subscribe to the identification information. When a second node clears the data to be cached, the nodes that subscribe to the identification information clear the corresponding second-level cache. This technical solution improves the consistency of data caching.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a data caching method, device, and storage medium. Background Art

[0002] Distributed caching is currently widely used in various network application systems, especially in situations with large business volumes. Distributed caching effectively supports highly concurrent data access. Each node in a distributed service cluster has a local cache. Because local caches are distributed across nodes, time differences between nodes, or anomalies such as update failures in the local cache, can lead to inconsistent local cache data across nodes. Summary of the Invention

[0003] In order to solve the above technical problems, the embodiments of the present invention hope to provide a data caching method, device and storage medium, and set up a distributed service cluster, in which the second-level cache includes nodes with the same cached data subscribing to the same identification information in the third-level cache, so that when any node clears the cached data, other nodes that subscribe to the identification information can promptly clear the same data in the corresponding second-level cache, thereby improving the consistency of the data cache.

[0004] The technical solution of the present invention is achieved as follows:

[0005] The present invention provides a data caching method, which is applied to a distributed service cluster, wherein the distributed service cluster includes multiple nodes, each of the multiple nodes includes a corresponding first-level cache and second-level cache, and the multiple nodes also share the same third-level cache and database. The method includes:

[0006] In a case where the data to be cached is obtained based on the data acquisition request using the first node, determining a cache result based on the data to be cached using the first node; the first node is any one of the multiple nodes, and the data to be cached is obtained from the corresponding first-level cache, second-level cache, third-level cache, or database;

[0007] When the cache result is to cache the data to be cached in the corresponding secondary cache, using the first node to cache the identification information of the data to be cached in the tertiary cache;

[0008] using nodes among the multiple nodes that receive the data acquisition request to subscribe to the identification information respectively;

[0009] When the second node is used to clear the data to be cached, the node among the multiple nodes that subscribes to the identification information is used to clear the data to be cached in the corresponding secondary cache respectively; the second node is any one of the multiple nodes that subscribes to the identification information.

[0010] In the above method, the step of obtaining the data to be cached based on the data acquisition request by the first node includes:

[0011] In a case where the data acquisition request is acquired by using the first node, searching the corresponding first-level cache for cache data matching the data acquisition request by using the first node;

[0012] In a case where the cache data matching the data acquisition request is not found from the corresponding first-level cache using the first node, searching for the cache data matching the data acquisition request from the corresponding second-level cache using the first node;

[0013] In a case where the cache data matching the data acquisition request is not found from the corresponding secondary cache using the first node, searching the cache data matching the data acquisition request from the third level cache using the first node;

[0014] If the first node is used to find no cache data matching the data acquisition request from the third-level cache, the first node is used to find data matching the data acquisition request from the database, and the found data is determined as the data to be cached.

[0015] In the above method, searching the database for data matching the data acquisition request and determining the found data as the data to be cached includes:

[0016] Generating a target data acquisition task based on the data acquisition request, and splitting the target data acquisition task into multiple data acquisition subtasks;

[0017] Determine, from the plurality of nodes, a target node for executing each of the plurality of data acquisition subtasks, and obtain a plurality of target nodes corresponding one-to-one to the plurality of data acquisition subtasks;

[0018] Utilizing the multiple target nodes, respectively executing the multiple data acquisition subtasks to obtain multiple task results corresponding to the multiple data acquisition subtasks;

[0019] Data aggregation is performed on the multiple task results, and the aggregated data is determined as the data to be cached.

[0020] In the above method, determining a cache result based on the data to be cached includes:

[0021] Acquire data characteristic information of the data to be cached; the data characteristic information includes at least one or more of the data size of the data to be cached and the number of data requests;

[0022] If the importance of the data represented by the data feature information is greater than or equal to a first preset threshold, determining the caching result is to cache the to-be-cached data in the secondary cache corresponding to the first node;

[0023] When the data importance represented by the data feature information is less than the first preset threshold, the cache result is determined to be caching the to-be-cached data into the third-level cache.

[0024] In the above method, after determining that the cache result is to cache the data to be cached in the secondary cache corresponding to the first node, the method further includes:

[0025] If the importance of the data represented by the data characteristic information is greater than a second preset threshold, writing the data to be cached into the local file system of the first node;

[0026] When the first node is restarted, reading the data to be cached from the local file system;

[0027] When the first node clears the data to be cached in the corresponding secondary cache, the data to be cached is cleared from the local file system.

[0028] In the above method, after determining the cache result based on the data to be cached, the method further includes:

[0029] In the case of caching the data to be cached based on the cache result, a data expiration time of the data to be cached is set based on data feature information of the data to be cached.

[0030] In the above method, after using the nodes among the multiple nodes that have received the data acquisition request to subscribe to the identification information respectively, the method further includes:

[0031] In a case where an exception occurs in a communication service subscribing to the identification information in the third-level cache and the second node clears the data to be cached, notifying a node different from the second node among the multiple nodes by using a message bus of the multiple nodes;

[0032] The to-be-cached data in the corresponding secondary cache are cleared respectively by using nodes different from the second node among the multiple nodes.

[0033] The present invention provides a data caching device, which is applied to a distributed service cluster, wherein the distributed service cluster includes multiple nodes, each of the multiple nodes includes a corresponding first-level cache and second-level cache, and the multiple nodes also share the same third-level cache and database, including:

[0034] a determination module configured to, when the data to be cached is obtained based on the data acquisition request using the first node, determine a cache result based on the data to be cached using the first node; the first node is any one of the multiple nodes, and the data to be cached is obtained from the corresponding first-level cache, second-level cache, third-level cache, or database;

[0035] a cache module configured to cache identification information of the data to be cached in the third-level cache using the first node when the cache result is to cache the data to be cached in the corresponding second-level cache;

[0036] a subscription module, configured to subscribe to the identification information respectively using the nodes among the multiple nodes that receive the data acquisition request;

[0037] A clearing module is used to clear the data to be cached in the corresponding secondary cache respectively using the nodes among the multiple nodes that subscribe to the identification information when the data to be cached is cleared using the second node; the second node is any one of the multiple nodes that subscribes to the identification information.

[0038] The present invention provides a data cache device, comprising: a processor, a memory and a communication bus;

[0039] The communication bus is used to realize the communication connection between the processor and the memory;

[0040] The processor is used to execute the computer program stored in the memory to implement the above data caching method.

[0041] The present invention provides a computer-readable storage medium, which stores one or more computer programs. The one or more computer programs can be executed by one or more processors to implement the above-mentioned data caching method.

[0042] The present invention provides a data caching method, device and storage medium, the method comprising: when a first node is used to obtain data to be cached based on a data acquisition request, the first node is used to determine a caching result based on the data to be cached; when the caching result is to cache the data to be cached into the corresponding secondary cache, the first node is used to cache identification information of the data to be cached in the tertiary cache; the nodes that receive the data acquisition request from multiple nodes are used to subscribe to the identification information respectively; when a second node is used to clear the data to be cached, the nodes that subscribe to the identification information from multiple nodes are used to clear the data to be cached in the corresponding secondary cache respectively. The technical solution provided by the present invention sets a distributed service cluster, in which the nodes of the secondary cache including the same cached data subscribe to the same identification information in the tertiary cache, so that when any node clears the cached data, the other nodes that subscribe to the identification information can promptly clear the same data in the corresponding secondary cache, thereby improving the consistency of the data cache. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 A schematic diagram of a data caching method according to an embodiment of the present invention;

[0044] Figure 2 A schematic diagram of an exemplary node data cache structure provided by an embodiment of the present invention;

[0045] Figure 3 A schematic diagram of an exemplary data acquisition process provided by an embodiment of the present invention;

[0046] Figure 4 A schematic diagram of an exemplary task processing flow provided by an embodiment of the present invention;

[0047] Figure 5 A schematic diagram of an exemplary local cache persistence process provided by an embodiment of the present invention;

[0048] Figure 6 A schematic diagram of exemplary subscription identification information provided by an embodiment of the present invention;

[0049] Figure 7 An exemplary process diagram of clearing cache data provided by an embodiment of the present invention Figure 1 ;

[0050] Figure 8 An exemplary process diagram of clearing cache data provided by an embodiment of the present invention Figure 2 ;

[0051] Figure 9 A schematic diagram of the structure of a data cache device provided by an embodiment of the present invention Figure 1 ;

[0052] Figure 10 A schematic diagram of the structure of a data cache device provided by an embodiment of the present invention Figure 2 . DETAILED DESCRIPTION

[0053] The following will be combined with the accompanying drawings of the embodiments of the present invention to clearly and completely describe the technical solutions of the present invention. It should be understood that the specific embodiments described herein are merely for the purpose of explaining the related applications and are not intended to limit such applications. It should also be noted that, for ease of description, only the portions relevant to the related applications are shown in the drawings.

[0054] The present invention provides a data caching method, which is applied to a distributed service cluster. The distributed service cluster includes multiple nodes, each of which includes a corresponding first-level cache and second-level cache, and the multiple nodes also share the same third-level cache and database. The data caching method is implemented by a data caching device. Figure 1 A flow chart of a data caching method provided by an embodiment of the present invention. Figure 1 As shown, it mainly includes the following steps:

[0055] S101. When the data to be cached is obtained based on a data acquisition request using the first node, a cache result is determined based on the data to be cached using the first node; the first node is any one of the multiple nodes, and the data to be cached is obtained from the corresponding first-level cache, second-level cache, third-level cache or database.

[0056] In an embodiment of the present invention, when the data caching device obtains the data to be cached based on the data acquisition request using the first node, the data caching device determines the caching result based on the data to be cached using the first node; the first node is any one of the multiple nodes, and the data to be cached is obtained from the corresponding first-level cache, second-level cache, third-level cache or database.

[0057] It should be noted that, in an embodiment of the present invention, the distributed service cluster includes multiple nodes, and each of the multiple nodes includes corresponding first-level cache and second-level cache. In the prior art, the implementation of the first-level cache and second-level cache corresponding to each node can be implemented based on Mybatis as the object-relational mapping (ORM) framework of the operating database. The present invention takes into account that in Mybatis, the first-level cache is a local cache (localCache) of the permanent cache (PerpetualCache) type of the base executor (BaseExecutor), which is essentially a simple lookup table (map), relatively primitive, not highly configurable, and has no memory protection. However, it is a session-level cache, and since the session can be closed in time, the first-level cache will not be occupied by a long-term saturation type, and therefore it is not expanded; in the second-level cache, the cache is cross-session, and therefore needs to perform better in terms of cache hit rate, and as a local cache, better performance is required, therefore, the present invention adds a local third-party cache library Caffeine to implement it on the basis of the Mybatis framework; in addition, since the original second-level cache in the Mybatis framework can be used selectively, and the original first-level cache and second-level cache differ only in data range, the present invention sets the original first-level cache and second-level cache in the Mybatis framework as the first-level cache, and sets the added local third-party cache library Caffeine as the second-level cache.

[0058] Caffeine is a high-performance cache library based on the object-oriented programming language Java 1.8. The in-memory cache provided by Caffeine uses an API modeled after Google Guava. Caffeine is an improvement based on the design experience of Google Guava Cache. Caffeine offers the following features: data can be loaded into the cache automatically or asynchronously; when Caffeine reaches its maximum capacity based on frequency and recent access, it switches to size-based mode; entries have a time-based expiration, measured since the last access or write; Caffeine performs asynchronous refreshes when the first request for an entry expires; keys are automatically enclosed in weak references; values ​​are automatically enclosed in weak or soft references; notifications are received when data is evicted (or otherwise deleted); written data is propagated to external resources; and the accumulated cache records the number of requests.

[0059] It should be noted that in embodiments of the present invention, multiple nodes also share the same third-level cache and database. Exemplarily, the third-level cache can be a Remote Dictionary Server (Redis). Redis is a non-relational database (NoSQL) and is an open-source, memory-based, and optionally persistent key-value store written in ANSI C. It includes multiple data structures and supports networking. Redis uses memory as its data storage medium, resulting in extremely high data read and write efficiency, far exceeding that of a database. For example, when setting and retrieving a 256-byte string, its read speed can reach up to 110,000 times per second and its write speed can reach up to 81,000 times per second. Unlike memcache, data stored in Redis is persistent and survives power outages or reboots. Because Redis's storage is divided into memory, disk storage, and log files, Redis can reload data from disk into memory after a reboot. This can be configured through configuration files, which is why Redis achieves persistence. Redis also uses a time-based caching strategy, adding an expiration time to keys. This cache expiration time reduces the spread of faults when cache inconsistencies occur.

[0060] Figure 2 The following is a schematic diagram of an exemplary node data cache structure provided by an embodiment of the present invention. Figure 2 As shown in the figure, the distributed service cluster includes multiple nodes, each of which includes a corresponding first-level cache, second-level cache, third-level cache, and database. The first-level cache includes a local cache and a global cache. The local cache is equipped with a cache executor (CachingExecutor), an executor (Executor), and a local cache (Local cache), while the global cache is equipped with a mapper (Mapper). The second-level cache can be a local third-party cache library Caffeine / EHCache. The third-level cache can be a distributed cache Redis.

[0061] It should be noted that, in an embodiment of the present invention, the data processing device can use each of a plurality of nodes to receive a data acquisition request. When the data processing device uses the first node of the plurality of nodes to receive a data acquisition request, the data processing device will search for data in the corresponding first-level cache, second-level cache, third-level cache, and database in sequence based on the data acquisition request. If the data to be cached is not obtained in the database, it means that the first-level cache, the second-level cache, and the third-level cache have not cached the data. The data to be cached is obtained in the database only after executing a database query. After obtaining the data to be cached, the data caching device can use the first node to determine the caching result based on the data to be cached.

[0062] It should be noted that, in an embodiment of the present invention, the first-level cache, the second-level cache, and the third-level cache can be understood as a database storing key-value pairs; wherein the key is the identification information of the data to be cached carried by the data acquisition request, and the value is the data to be cached, that is, the corresponding cached data is obtained based on the identification information carried by the data acquisition request.

[0063] Specifically, in an embodiment of the present invention, the data caching device uses the first node to obtain the data to be cached based on the data acquisition request, including: when the data acquisition request is obtained by using the first node, using the first node to search for cached data that matches the data acquisition request from the corresponding first-level cache; when the cached data that matches the data acquisition request is not found from the corresponding first-level cache by using the first node, using the first node to search for cached data that matches the data acquisition request from the corresponding second-level cache; when the cached data that matches the data acquisition request is not found from the corresponding second-level cache by using the first node, using the first node to search for cached data that matches the data acquisition request from the third-level cache; when the cached data that matches the data acquisition request is not found from the third-level cache by using the first node, using the first node to search for data that matches the data acquisition request from the database, and determining the found data as the data to be cached.

[0064] It should be noted that, in an embodiment of the present invention, when the data caching device obtains a data acquisition request using the first node, it will use the first node to first search for cached data that matches the data acquisition request from the corresponding first-level cache. If the identification information (key) carried in the data acquisition request can find the corresponding cached data (value) in the corresponding first-level cache, the cached data can be directly returned, that is, a hit; if the cached data is not found in the corresponding first-level cache, that is, a miss, the cached data that matches the data acquisition request will be further searched from the corresponding second-level cache. If the key carried in the data acquisition request can find the corresponding value in the corresponding second-level cache, the value can be directly returned; if the cached data is not found in the corresponding second-level cache, the cached data that matches the data acquisition request will be further searched from the third-level cache redis. If the key carried in the data acquisition request can find the corresponding value in the corresponding third-level cache, the data to be cached can be directly returned; if the cached data is not found in the corresponding third-level cache, data search will be further performed from the database based on the data acquisition request, and the found data will be determined as the data to be cached.

[0065] Figure 3 An exemplary data acquisition process provided by an embodiment of the present invention is shown in FIG. Figure 1 .like Figure 3 As shown, the data caching device uses the first node to obtain the identification information of the data to be cached, and then generates a data acquisition request based on the obtained identification information, and then searches the first-level node corresponding to the first node for a value that matches the identification information key carried in the data acquisition request. If it hits in the first-level cache, the hit value is used as the data to be cached; if it does not hit in the first-level cache, continue to search for a value that matches the identification information key from the second cache; if it hits in the second-level cache, the hit value is used as the data to be cached, and the request time is updated; if it does not hit in the second-level cache, continue to search for a value that matches the identification information key from the third-level cache redis; if it hits in the third-level cache redis, the hit value is used as the data to be cached; if it does not hit in the third-level cache redis, continue to search for a value that matches the identification information key from the database, that is, execute a Structured Query Language (SQL) query, and then store the matching value as the data to be cached in the second-level cache or the third-level cache redis.

[0066] Specifically, in an embodiment of the present invention, the data caching device searches for data that matches the data acquisition request from a database, and determines the found data as data to be cached, including: generating a target data acquisition task based on the data acquisition request, and splitting the target data acquisition task into multiple data acquisition subtasks; determining a target node for executing each subtask in the multiple data acquisition subtasks from multiple nodes, and obtaining multiple target nodes corresponding one-to-one to the multiple data acquisition subtasks; using the multiple target nodes, respectively executing the multiple data acquisition subtasks, and obtaining multiple task results corresponding one-to-one to the multiple data acquisition subtasks; performing data aggregation on the multiple task results, and determining the aggregated data as data to be cached.

[0067] It should be noted that, in an embodiment of the present invention, the process of the data processing device searching for data matching the data acquisition request from the database can be a process of executing an SQL query. In traditional programming, concurrent programming is performed in the code of the business layer to improve the execution efficiency of the code. The processing of the Data Access Object (DAO) layer is handed over to the ORM framework to complete, and the interior of the ORM framework is a black box. In the traditional Mybatis framework, it is single-threaded execution. The main reason is that the steps inside it have a relatively strong front-and-back dependency, so it is not suitable for concurrency, or the input-output brought by concurrency is relatively low. After carefully analyzing the internal execution process of Mybatis, it can be found that the bottleneck of executing SQL operations internally is the process of interacting with the database server. Therefore, the data processing device uses a sharding mechanism to process data acquisition requests.

[0068] It should be noted that, in an embodiment of the present invention, for a data acquisition request that does not hit in the three-level cache, the data caching device will perform a real query operation, that is, search the database for data that matches the data acquisition request; at this time, the data processing device can generate a target data acquisition task based on the data acquisition request, and then, according to the configuration of the business configuration file, split the target data acquisition task into multiple data acquisition subtasks, wherein each subtask is a shard, and the task items should be completely independent from the business level, such as querying the two SQL statements with id 1 and id 2. Then, from multiple nodes, determine the target node for executing each subtask in the multiple data acquisition subtasks, and obtain multiple target nodes corresponding to the multiple data acquisition subtasks one by one, so as to use the multiple target nodes to execute the multiple data acquisition subtasks respectively, and obtain multiple task results corresponding to the multiple data acquisition subtasks one by one. Finally, perform data aggregation on the multiple task results, and determine the aggregated data as the data to be cached.

[0069] Figure 4The following is a flow chart of an exemplary task execution provided by an embodiment of the present invention. Figure 4 As shown, the process of task execution is discussed using three nodes as an example: For example, when querying data in the range of id in (1, 3, 8), the node that receives the data acquisition request (assuming that node A receives the request) automatically becomes the master node and acts as the registration center, and the other nodes in the distributed service cluster become slave nodes; at this time, the master node divides the query id in (1, 3, 8) into three shards, id = 1, id = 3, and id = 8 each as a shard. These three shards can be executed by three nodes (node ​​A, node B, and node C), which is parallel execution. After the three nodes complete the execution and obtain the results, they will return the actual results to the master node (node ​​A) for data encapsulation (aggregation). At the same time, based on the amount of data, they decide whether to store it in the corresponding second-level cache or the third-level cache.

[0070] Specifically, in an embodiment of the present invention, the data caching device determines a caching result based on the data to be cached, including: obtaining data feature information of the data to be cached; the data feature information includes at least one or more of the data size of the data to be cached and the number of data requests; when the data importance represented by the data feature information is greater than or equal to a first preset degree threshold, determining that the caching result is to cache the data to be cached to the second-level cache corresponding to the first node; when the data importance represented by the data feature information is less than the first preset degree threshold, determining that the caching result is to cache the data to be cached to the third-level cache.

[0071] It should be noted that in an embodiment of the present invention, after the data cache device obtains the data to be cached from the database, it will obtain the data feature information of the data to be cached. Since both the first-level cache and the second-level cache are local caches with limited memory space, the first-level cache can clear the local cache when the session is closed, so there will be no situation where the cache occupies the memory for a long time. However, in the second-level cache, it spans sessions. At this time, when the amount of data is large, storing it in the local cache puts a lot of pressure on the memory. Therefore, a first preset degree threshold can be set according to the importance of the data represented by the data feature information. If it exceeds the set first preset degree threshold, it will directly enter the second-level cache. If the importance of the data represented by the data feature information does not reach the first preset degree threshold, it will enter the local second-level cache. The specific preset degree threshold can be set according to actual needs and application scenarios. The first preset degree threshold can be dynamically adjusted according to the business characteristics of the data acquisition request and hot update can be achieved. For example: the data cache device can be set to have a greater importance for data represented by data to be cached if the data volume is smaller, and a smaller importance for data represented by data to be cached if the data volume is larger; the greater the number of data requests, the greater the importance for data represented by data to be cached, and the smaller the number of data requests, the smaller the importance for data represented by data to be cached. In this way, data with greater importance is cached in the secondary cache of the local cache. Then, when the same data acquisition request is received next time, the data to be cached can be directly obtained from the local secondary cache, avoiding network input and output (IO) delays, and caching large amounts of data into the remote third-level cache, reducing the space pressure of the secondary cache. This data caching method that is adaptive according to the data feature information of the data to be cached can better reduce the pressure on the local memory and improve data acquisition efficiency.

[0072] Specifically, in an embodiment of the present invention, after the data caching device determines that the caching result is to cache the data to be cached in the secondary cache corresponding to the first node, it can also perform the following steps: when the data importance represented by the data feature information is greater than a second preset degree threshold, the data to be cached is written to the local file system of the first node; when the first node is restarted, the data to be cached is read from the local file system; when the first node clears the data to be cached in the corresponding secondary cache, the data to be cached is cleared from the local file system.

[0073] It should be noted that in the embodiments of the present invention, since the first-level cache and the second-level cache are memory cache blocks, their biggest feature is speed. However, because they remain in the memory, the cached data will be lost after the node is restarted. At this time, all data acquisition requests will be searched from the database, which will suddenly increase the pressure on the database. Not only will the data interface respond slowly, but it may also drag down the entire database. Therefore, the data processing device considers at this time whether the data importance represented by the data feature information of the data to be cached is greater than a second preset threshold, and the second preset threshold can be set according to actual needs and application scenarios; illustratively, it can be determined whether the identification information (key) of the data to be cached needs to be locally persisted based on the configuration in the node configuration file. If persistence is required, a higher importance can be set; or, the data caching device can also set an importance value for the data to be cached based on the number of requests for the data to be cached; the more times the data is requested, the greater the data importance represented by the data to be cached, and the fewer times the data is requested, the smaller the data importance represented by the data to be cached; therefore, for relatively important data, that is, the data importance represented by the data feature information of the data to be cached is greater than the second preset threshold, the data caching device can use the first node to write the data to be cached to the local file system of the first node; wherein the local file system is a local disk, and illustratively, the implementation steps of writing the data to be cached to the local file system can be: by submitting the cache task to the queue, and asynchronously processing by the loop coroutine, a file with the same name as the node can be created when the node is first started to cache the locally persisted data, and then after the node is restarted, the data to be cached will be read from the local file system.

[0074] Figure 5 The following is a flow chart of an exemplary local cache persistence provided by an embodiment of the present invention. Figure 5 As shown, when the data processing device caches data locally, it will store a copy of important data in the local file system according to the business configuration and execute it through a loop coroutine. When the node restarts, it will read the data in the local file system for data initialization.

[0075] It should be noted that, in an embodiment of the present invention, when the data processing device uses the first node to clear the data to be cached in the corresponding secondary cache, the data processing device will use the first node to clear the data to be cached from the local file system to avoid the existence of dirty data in the local file system. The data processing device can use the first node to add the data to be cleared to the task queue, and there is a special loop coroutine to delete the data from the local file system to ensure that the expired data in the persistent file can be deleted in time. The exemplary data clearing implementation process is as follows: Figure 5As shown: taking tasks from the task queue, deleting data in the local file system, and using coroutines for processing. First, the coroutine will not affect the process of the main thread, so it will not slow down the interface. The backup and clearing operations are not coupled with business relationships, so it is completely possible to do them in the background. Second, the coroutine will not affect the project status. If it is a regular thread, after the main thread crashes (down), the project will not be down because the regular thread is still there. At this time, the project has already had problems, but it is impossible to discover that the project is down in time, which will cause problems. Coroutines do not have this problem.

[0076] Specifically, in an embodiment of the present invention, after the data caching device determines the caching result based on the data to be cached, it can also perform the following steps: when caching the data to be cached based on the caching result, setting the data expiration time of the data to be cached based on the data feature information of the data to be cached.

[0077] It should be noted that, in an embodiment of the present invention, the data cache device selects a time-based caching method in the eviction cache strategy; exemplarily, expire After Access (long, TimeUnit) is adopted, that is, the timing starts after the last access or write, and expires after the specified time.

[0078] It should be noted that, in an embodiment of the present invention, the data caching device will set the data expiration time of the data to be cached based on the data feature information when caching the data to be cached based on the cache result. Moreover, for most scenarios, it conforms to the characteristic that the most frequently used data has a long retention time and the least used data has a short retention time, that is, the expiration time is determined based on the number of data requests. When the expiration time arrives, the data processing device will clear the data cached in the secondary cache. At this time, if this part of the data is also stored in the local file system, it will also be deleted accordingly.

[0079] S102: When the cache result is to cache the data to be cached in the corresponding L2 cache, use the first node to cache identification information of the data to be cached in the L3 cache.

[0080] In an embodiment of the present invention, when the caching result is to cache the data to be cached in the corresponding L2 cache, the data caching device uses the first node to cache identification information of the data to be cached in the L3 cache.

[0081] It should be noted that, in an embodiment of the present invention, when the data caching device uses the first node to determine that the cache result is to cache the data to be cached in the corresponding second-level cache, it will use the first node to cache an identification information value of the data to be cached in the third-level cache; then, when the data caching device uses multiple nodes to receive the same data acquisition request, it will also cache an identification information value of the data to be cached in the third-level cache. Because the identification information of the data to be cached is the same, when the second node caches the identification information value of the data to be cached in the third-level cache, it will overwrite the identification information cached by the previous node. In this way, only one identification information is stored in redis.

[0082] S103: Utilize the nodes that receive the data acquisition request among the multiple nodes to subscribe to the identification information respectively.

[0083] In an embodiment of the present invention, the data caching device uses nodes that receive data acquisition requests among multiple nodes to subscribe to identification information respectively.

[0084] It should be noted that in an embodiment of the present invention, after the data processing device stores a key value as an identifier of the second-level cache in the third-level cache, it will use the node that receives the data acquisition request among multiple nodes to subscribe to the identification information key, that is, monitor the key item.

[0085] Figure 6 Schematic diagram of an exemplary subscription identification information provided by an embodiment of the present invention. Figure 6 As shown, the data processing device uses multiple nodes that receive data acquisition requests to subscribe to the identification information in the third-level cache redis.

[0086] Specifically, in an embodiment of the present invention, the data caching device uses the nodes among multiple nodes that receive the data acquisition request to subscribe to the identification information respectively, and can also perform the following steps: when an exception occurs in the communication service for subscribing to the identification information in the third-level cache and the second node clears the data to be cached, the message bus of multiple nodes is used to notify the nodes among the multiple nodes that are different from the second node; the nodes among the multiple nodes that are different from the second node are used to clear the corresponding data to be cached in the second-level cache respectively.

[0087] It should be noted that in an embodiment of the present invention, in actual situations, the communication service that subscribes to the identification information in the third-level cache may be abnormal. In this way, there will be inconsistencies between the second-level caches corresponding to multiple nodes. At this time, the data caching device will start the bus mechanism, that is, use the message bus of multiple nodes to notify other nodes in the multiple nodes to clear the corresponding cached data in the second-level cache to ensure that the data cached in the second-level cache corresponding to each node remains consistent.

[0088] It should be noted that in an embodiment of the present invention, when a data caching device uses a node to clear its L2 cache, it uses that node to publish a message to notify other nodes. Upon receiving the message from the message bus, the other nodes clear their local L2 caches. For example, a lightweight message broker connects the distributed nodes and is used to broadcast state changes, such as configuration changes, or other message instructions. The core concept is to extend Spring Boot applications through distributed launchers, which can also be used to establish a communication channel between multiple applications.

[0089] Figure 7 An exemplary process diagram of clearing cache data provided by an embodiment of the present invention Figure 1 .like Figure 7 As shown, the data processing device will trigger the interface (curl trigger) of any node to clear the cache. For example, for node A (clientA), clientA clears the local cache and sends a command to the message bus. Node B (clientB) and node C (clientC) receive the broadcast in the message bus and start clearing the local secondary cache in their respective nodes.

[0090] S104. When the second node is used to clear the data to be cached, the nodes subscribed to the identification information among the multiple nodes are used to clear the data to be cached in the corresponding secondary cache respectively; the second node is any node subscribed to the identification information among the multiple nodes.

[0091] In an embodiment of the present invention, when the data caching device uses the second node to clear the data to be cached, it uses the nodes that subscribe to the identification information among multiple nodes to clear the data to be cached in the corresponding secondary cache respectively; the second node is any one of the nodes that subscribe to the identification information among the multiple nodes.

[0092] It should be noted that, in an embodiment of the present invention, after the data caching device uses a node that receives a data acquisition request from multiple nodes to subscribe to the identification information key, if any node that subscribes to the identification information clears the data to be cached and clears the identification information stored in the third-level cache at the same time, because other nodes that subscribe to the identification information are listening to this identification information, then other nodes that subscribe to the identification information can receive a command to clear the data to be cached, so as to clear the corresponding data to be cached in the second-level cache respectively.

[0093] Figure 8 An exemplary process diagram of clearing cache data provided by an embodiment of the present invention Figure 2 .like Figure 8As shown, node A (clientA), node B (clientB), and node C (clientC) all subscribe to the identification information key in the third-level cache redis. If the cached data of clientA becomes invalid at this time and the local cache is cleared, the third-level cache redis will be notified to clear the identification information key. In this way, other nodes that subscribe to the identification information: clientB, clientC can receive the command to clear the cached data, so as to clear the cached data in the corresponding second-level cache respectively.

[0094] The present invention provides a data caching method, which is applied to a distributed service cluster. The distributed service cluster includes multiple nodes, each of the multiple nodes includes a corresponding first-level cache and a second-level cache, and the multiple nodes also share the same third-level cache and database. The method includes: when a first node obtains data to be cached based on a data acquisition request, the first node is used to determine a caching result based on the data to be cached; when the caching result is to cache the data to be cached in the corresponding second-level cache, the first node is used to cache identification information of the data to be cached in the third-level cache; the nodes that receive the data acquisition request among the multiple nodes are used to subscribe to the identification information; when a second node is used to clear the data to be cached, the nodes that subscribe to the identification information among the multiple nodes are used to clear the data to be cached in the corresponding second-level cache. The data caching method provided by the present invention sets the nodes in the distributed service cluster whose second-level cache includes the same cached data to subscribe to the same identification information in the third-level cache, so that when any node clears the cached data, the other nodes that subscribe to the identification information can promptly clear the same data in the corresponding second-level cache, thereby improving the consistency of the data cache.

[0095] The present invention provides a data caching device, which is applied to a distributed service cluster. The distributed service cluster includes multiple nodes, each of which includes a corresponding first-level cache and second-level cache, and the multiple nodes also share the same third-level cache and database. Figure 9 A schematic diagram of the structure of a data cache device provided by an embodiment of the present invention Figure 1 .like Figure 9 Shown, including:

[0096] Determining module 901 is configured to, when the data to be cached is obtained based on the data acquisition request using the first node, determine a cache result based on the data to be cached using the first node; the first node is any one of the multiple nodes, and the data to be cached is obtained from the corresponding first-level cache, second-level cache, third-level cache, or database;

[0097] A cache module 902 is configured to cache the identification information of the data to be cached in the third-level cache using the first node when the cache result is to cache the data to be cached in the corresponding second-level cache;

[0098] A subscription module 903 is configured to use the nodes among the multiple nodes that receive the data acquisition request to subscribe to the identification information respectively;

[0099] The clearing module 904 is used to clear the data to be cached in the corresponding secondary cache respectively using the node that subscribes to the identification information among the multiple nodes when using the second node to clear the data to be cached; the second node is any one of the multiple nodes that subscribes to the identification information.

[0100] In one embodiment of the present invention, the data caching device also includes a search module (not shown in the figure), which is used to use the first node to search for cache data matching the data acquisition request from the corresponding first-level cache when the data acquisition request is obtained using the first node; use the first node to search for cache data matching the data acquisition request from the corresponding second-level cache when the first node is used to obtain cache data matching the data acquisition request; use the first node to search for cache data matching the data acquisition request from the third-level cache when the first node is used to obtain cache data matching the data acquisition request; use the first node to search for data matching the data acquisition request from the database when the first node is used to obtain cache data matching the data acquisition request, and determine the found data as the data to be cached.

[0101] In one embodiment of the present invention, the search module (not shown in the figure) is also used to generate a target data acquisition task based on the data acquisition request, and split the target data acquisition task into multiple data acquisition subtasks; determine the target node for executing each subtask in the multiple data acquisition subtasks from the multiple nodes, and obtain multiple target nodes corresponding one-to-one to the multiple data acquisition subtasks; use the multiple target nodes to execute the multiple data acquisition subtasks respectively, and obtain multiple task results corresponding one-to-one to the multiple data acquisition subtasks; perform data aggregation on the multiple task results, and determine the aggregated data as the data to be cached.

[0102] In one embodiment of the present invention, the determination module 901 is further used to obtain data feature information of the data to be cached; the data feature information includes at least one or more of the data size of the data to be cached and the number of data requests; when the data importance represented by the data feature information is greater than or equal to a first preset degree threshold, the caching result is determined to be caching the data to be cached to the second-level cache corresponding to the first node; when the data importance represented by the data feature information is less than the first preset degree threshold, the caching result is determined to be caching the data to be cached to the third-level cache.

[0103] In one embodiment of the present invention, the data caching device also includes a persistence module (not shown in the figure), which is used to write the data to be cached into the local file system of the first node when the importance of the data represented by the data feature information is greater than a second preset threshold; read the data to be cached from the local file system when the first node is restarted; and clear the data to be cached from the local file system when the first node clears the data to be cached in the corresponding secondary cache.

[0104] In one embodiment of the present invention, the data caching device also includes a setting module (not shown in the figure) for setting the data expiration time of the data to be cached based on the data feature information of the data to be cached when caching the data to be cached based on the cache result.

[0105] In one embodiment of the present invention, the data caching device also includes a notification module (not shown in the figure), which is used to use the message bus of the multiple nodes to notify the nodes among the multiple nodes that are different from the second node when an abnormality occurs in the communication service subscribing to the identification information in the third-level cache and the second node clears the data to be cached; and use the nodes among the multiple nodes that are different from the second node to clear the data to be cached in the corresponding second-level cache respectively.

[0106] The present invention provides a data caching device. Figure 10 A schematic diagram of the structure of a data cache device provided by an embodiment of the present invention Figure 2 .like Figure 10 As shown, the data cache device includes: a processor 1001, a memory 1002 and a communication bus 1003;

[0107] The communication bus 1003 is used to realize the communication connection between the processor 1001 and the memory 1002;

[0108] The processor 1001 is configured to execute the computer program stored in the memory 1002 to implement the above-mentioned data caching method.

[0109] The present invention provides a data caching device. When a first node among multiple nodes searches for data in a corresponding first-level cache, second-level cache, third-level cache, and database in sequence based on a data acquisition request until the data to be cached is acquired in the database, the first node is used to determine a caching result based on the data to be cached; when the caching result determined by the first node is to cache the data to be cached in the corresponding second-level cache, the first node is used to cache identification information of the data to be cached in the third-level cache; when a node among the multiple nodes that receives the data acquisition request subscribes to the identification information respectively; when a second node among the multiple nodes that subscribes to the identification information clears the data to be cached, the node among the multiple nodes that subscribes to the identification information clears the data to be cached in the corresponding second-level cache respectively. The data caching device provided by the present invention is configured in a distributed service cluster, where nodes in the second-level cache that include the same cached data subscribe to the same identification information in the third-level cache, so that when any node clears the cached data, other nodes that subscribe to the identification information can promptly clear the same data in the corresponding second-level cache, thereby improving the consistency of the data cache.

[0110] The present invention provides a computer-readable storage medium storing one or more computer programs, wherein the one or more computer programs can be executed by one or more processors to implement the above-mentioned data caching method. The computer-readable storage medium can be a volatile memory (volatile memory), such as a random-access memory (RAM); or a non-volatile memory (non-volatile memory), such as a read-only memory (ROM), a flash memory, a hard disk drive (HDD), or a solid-state drive (SSD); or can be a device including one or any combination of the above-mentioned memories, such as a mobile phone, a computer, a tablet device, a personal digital assistant, etc.

[0111] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Thus, the present invention may take the form of hardware embodiments, software embodiments, or embodiments combining software and hardware. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage and optical storage, etc.) containing computer-usable program code.

[0112] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowcharts and / or block diagrams, as well as combinations of processes and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowcharts and / or block diagrams. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0113] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0114] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 The steps for the function specified in one or more boxes.

[0115] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in the present invention should be included in the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be based on the scope of protection of the claims.

Claims

1. A data caching method, characterized in that: Applied to a distributed service cluster, the distributed service cluster includes multiple nodes, each of the multiple nodes includes a corresponding first-level cache and second-level cache, and the multiple nodes also share the same third-level cache and database, the method includes: In a case where the data to be cached is obtained based on the data acquisition request using the first node, determining a cache result based on the data to be cached using the first node; the first node is any one of the multiple nodes, and the data to be cached is obtained from the corresponding first-level cache, second-level cache, third-level cache, or database; When the cache result is to cache the data to be cached in the corresponding secondary cache, using the first node to cache the identification information of the data to be cached in the tertiary cache; using nodes among the multiple nodes that receive the data acquisition request to subscribe to the identification information respectively; When the second node is used to clear the data to be cached, the node among the multiple nodes that subscribes to the identification information is used to clear the data to be cached in the corresponding secondary cache respectively; the second node is any one of the multiple nodes that subscribes to the identification information.

2. The method according to claim 1, characterized in that The obtaining of the data to be cached by the first node based on the data obtaining request includes: In a case where the data acquisition request is acquired by using the first node, searching the corresponding first-level cache for cache data matching the data acquisition request by using the first node; In a case where the cache data matching the data acquisition request is not found from the corresponding first-level cache using the first node, searching for the cache data matching the data acquisition request from the corresponding second-level cache using the first node; In a case where the cache data matching the data acquisition request is not found from the corresponding secondary cache using the first node, searching the cache data matching the data acquisition request from the third level cache using the first node; If the first node is used to find no cache data matching the data acquisition request from the third-level cache, the first node is used to find data matching the data acquisition request from the database, and the found data is determined as the data to be cached.

3. The method according to claim 2, characterized in that The step of searching the database for data matching the data acquisition request and determining the found data as the data to be cached includes: Generating a target data acquisition task based on the data acquisition request, and splitting the target data acquisition task into multiple data acquisition subtasks; Determine, from the plurality of nodes, a target node for executing each of the plurality of data acquisition subtasks, and obtain a plurality of target nodes corresponding one-to-one to the plurality of data acquisition subtasks; Utilizing the multiple target nodes, respectively executing the multiple data acquisition subtasks to obtain multiple task results corresponding to the multiple data acquisition subtasks; Data aggregation is performed on the multiple task results, and the aggregated data is determined as the data to be cached.

4. The method according to claim 1, wherein The determining of a cache result based on the data to be cached includes: Acquire data characteristic information of the data to be cached; the data characteristic information includes at least one or more of the data size of the data to be cached and the number of data requests; If the importance of the data represented by the data feature information is greater than or equal to a first preset threshold, determining the caching result is to cache the to-be-cached data in the secondary cache corresponding to the first node; When the data importance represented by the data feature information is less than the first preset threshold, the cache result is determined to be caching the to-be-cached data into the third-level cache.

5. The method according to claim 4, characterized in that After determining that the cache result is to cache the data to be cached in the secondary cache corresponding to the first node, the method further includes: If the importance of the data represented by the data characteristic information is greater than a second preset threshold, writing the data to be cached into the local file system of the first node; When the first node is restarted, reading the data to be cached from the local file system; When the first node clears the data to be cached in the corresponding secondary cache, the data to be cached is cleared from the local file system.

6. The method according to claim 1, characterized in that After determining the cache result based on the data to be cached, the method further includes: In the case of caching the data to be cached based on the cache result, a data expiration time of the data to be cached is set based on data feature information of the data to be cached.

7. The method according to claim 1, characterized in that After using the nodes among the multiple nodes that have received the data acquisition request to subscribe to the identification information respectively, the method further includes: In a case where an exception occurs in a communication service subscribing to the identification information in the third-level cache and the second node clears the data to be cached, notifying a node different from the second node among the multiple nodes by using a message bus of the multiple nodes; The to-be-cached data in the corresponding secondary cache are cleared respectively by using nodes different from the second node among the multiple nodes.

8. A data cache device, characterized in that: Applied to a distributed service cluster, the distributed service cluster includes multiple nodes, each of the multiple nodes includes a corresponding first-level cache and second-level cache, and the multiple nodes also share the same third-level cache and database, including: a determination module configured to, when the data to be cached is obtained based on the data acquisition request using the first node, determine a cache result based on the data to be cached using the first node; the first node is any one of the multiple nodes, and the data to be cached is obtained from the corresponding first-level cache, second-level cache, third-level cache, or database; a cache module configured to cache identification information of the data to be cached in the third-level cache using the first node when the cache result is to cache the data to be cached in the corresponding second-level cache; a subscription module, configured to subscribe to the identification information respectively using the nodes among the multiple nodes that receive the data acquisition request; A clearing module is used to clear the data to be cached in the corresponding secondary cache respectively using the nodes among the multiple nodes that subscribe to the identification information when the data to be cached is cleared using the second node; the second node is any one of the multiple nodes that subscribes to the identification information.

9. A data cache device, characterized in that: include: processor, memory, and communication bus; The communication bus is used to realize the communication connection between the processor and the memory; The processor is configured to execute the computer program stored in the memory to implement the data caching method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores one or more computer programs, and the one or more computer programs can be executed by one or more processors to implement the data caching method according to any one of claims 1 to 7.

Images