Data caching management method, device, terminal device and storage medium

Abstract

The present application discloses a data cache management method, apparatus, terminal device, and storage medium. The data cache management method includes: when a query request is received, determining a corresponding target interface; detecting whether the target interface exists through a preset access filter to obtain a detection result; and querying data in a preset multi-level cache database based on the detection result to obtain target data. Based on the solution of the present application, by caching data in high-traffic business scenarios from the dimension of interfaces, it is possible to preferentially cache high-traffic API interfaces and set up a multi-level cache database, such that the performance and data capacity of each level of cache database for data caching are different, solving the technical problems of poor data cache performance and low flexibility in high-traffic business scenarios, and improving the overall performance and flexibility of the data cache system.

Description

Technical Field

[0001] This application relates to the technical field of data management, and particularly to a data cache management method, apparatus, terminal device, and storage medium. Background Art

[0002] Currently, there are many instantaneous high-traffic business scenarios. For the existing strategies for high-traffic business scenarios, generally all services use the same single cache structure, which has the following problems:

[0003] On the one hand, the request volume of high-traffic services may exceed the processing capacity of the single cache structure, resulting in a decrease in cache hit rate and affecting the performance and response time of the system; on the other hand, different services may have different requirements for cache capacity, data consistency requirements, etc., but it is difficult for a single cache structure to flexibly allocate cache resources according to the characteristics of the services. Summary of the Invention

[0004] The main purpose of this application is to provide a data cache management method, apparatus, terminal device, and storage medium, aiming to solve the technical problems of poor data cache performance and low flexibility in high-traffic business scenarios.

[0005] To achieve the above object, this application provides a data cache management method, and the data cache management method includes:

[0006] When a query request is received, determine the corresponding target interface;

[0007] Detect whether the target interface exists through a preset access filter to obtain a detection result;

[0008] Based on the detection result, query the data in a preset multi-level cache database to obtain target data.

[0009] Optionally, the multi-level cache database includes a primary cache database, a secondary cache database, and a tertiary cache database. The step of querying the data in the preset multi-level cache database based on the detection result includes:

[0010] If the target interface does not exist, query the data in the secondary cache database;

[0011] If the data in the secondary cache database does not exist, query the data in the tertiary cache database;

[0012] If the data in the tertiary cache database exists, pull the data in the tertiary cache database through the secondary cache database;

[0013] If the target interface exists, query the data in the primary cache database;

[0014] If the data in the primary cache database does not exist, execute the step of querying the data in the secondary cache database.

[0015] Optionally, after the step of querying the data in the secondary cache database, it further includes:

[0016] Detect whether the target data exists through the admission filter;

[0017] If the target data exists, push the target data to the primary cache database through the secondary cache database.

[0018] Optionally, the data cache management method further includes the following steps:

[0019] When detecting changed data, delete the data in the primary cache database;

[0020] Asynchronously write the changed data to the secondary cache database and the tertiary cache database;

[0021] Push the changed data to the primary cache database through the secondary cache database.

[0022] Optionally, the data cache management method further includes the following steps:

[0023] Respectively obtain the key quantity of the secondary cache database and the key quantity of the tertiary cache database;

[0024] When detecting that the key quantity of the secondary cache database is not equal to the key quantity of the tertiary cache database, pull the data of the tertiary cache database through the secondary cache database and delete the data in the primary cache database.

[0025] Optionally, the data cache management method further includes the following steps:

[0026] Based on a preset timer, synchronize the data in the preset data source to the tertiary cache database and the secondary cache database; and / or

[0027] Based on a preset listening policy, synchronize the data to the tertiary cache database and the secondary cache database through the data source.

[0028] Optionally, before the step of detecting whether the target interface exists through a preset admission filter to obtain a detection result, it further includes:

[0029] Based on a preset sliding window, count the request times of several service interfaces;

[0030] Sort the request times to obtain an interface sorted list of the several service interfaces;

[0031] Determine the access ratio according to the preset local resources and data volume;

[0032] Based on the access ratio and the interface sorting list, determine the access filter.

[0033] An embodiment of the present application also proposes a data cache management device, which includes:

[0034] An acquisition module, configured to determine a corresponding target interface when receiving a query request;

[0035] An access module, configured to detect whether the target interface exists through a preset access filter to obtain a detection result;

[0036] A query module, configured to query the data of a preset multi-level cache database based on the detection result to obtain target data.

[0037] An embodiment of the present application also proposes a terminal device, which includes a memory, a processor, and a data cache management program stored on the memory and executable on the processor. When the data cache management program is executed by the processor, it implements the steps of the data cache management method described above.

[0038] An embodiment of the present application also proposes a computer-readable storage medium, on which a data cache management program is stored. When the data cache management program is executed by a processor, it implements the steps of the data cache management method described above.

[0039] The data cache management method, device, terminal device, and storage medium proposed by the embodiments of the present application determine a corresponding target interface when receiving a query request; detect whether the target interface exists through a preset access filter to obtain a detection result; and query the data of a preset multi-level cache database based on the detection result to obtain target data. Based on the solution of the present application, by caching the data in high-traffic business scenarios from the interface dimension, it is possible to preferentially cache high-traffic API interfaces and set up a multi-level cache database, so that the performance and data capacity of each level of cache database for data caching are different, solving the technical problems of poor data caching performance and low flexibility in high-traffic business scenarios, improving the overall performance and flexibility of the data cache system, and thereby reducing the load of the system. Description of the Drawings

[0040] Figure 1 It is a schematic diagram of the functional modules of the terminal device to which the data cache management device of the present application belongs;

[0041] Figure 2Schematic flowchart of the first exemplary embodiment of the data cache management method of the present application;

[0042] Figure 3 Multi-level cache architecture of the multi-level cache database for the data cache management method of the present application;

[0043] Figure 4 Schematic flowchart of the second exemplary embodiment of the data cache management method of the present application;

[0044] Figure 5 Schematic diagram of data query for the data cache management method of the present application;

[0045] Figure 6 Schematic flowchart of the third exemplary embodiment of the data cache management method of the present application;

[0046] Figure 7 Schematic flowchart of the fourth exemplary embodiment of the data cache management method of the present application;

[0047] Figure 8 Schematic diagram of self-healing of cached data for the data cache management method of the present application;

[0048] Figure 9 Schematic flowchart of the fifth exemplary embodiment of the data cache management method of the present application;

[0049] Figure 10 Schematic diagram of synchronization of cached data for the data cache management method of the present application;

[0050] Figure 11 Schematic flowchart of the sixth exemplary embodiment of the data cache management method of the present application;

[0051] Figure 12 Admission judgment flowchart for the data cache management method of the present application.

[0052] The realization, functional features and advantages of the present application will be further described with reference to the embodiments and the accompanying drawings. Detailed implementation manners

[0053] It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0054] The main solution of the embodiment of this application is as follows: when a query request is received, determine the corresponding target interface; detect whether the target interface exists through a preset access filter to obtain a detection result; based on the detection result, query the data in the preset multi-level cache database to obtain target data. Based on the solution of this application, by caching data in high-traffic business scenarios in terms of interfaces, it is possible to preferentially cache high-traffic API interfaces and set up a multi-level cache database, such that the performance and data capacity of each level of cache database for data caching are different, solving the technical problems of poor data caching performance and low flexibility in high-traffic business scenarios, improving the overall performance and flexibility of the data caching system, and thereby reducing the system load.

[0055] The embodiment of this application takes into account that in the existing data caching strategy, there are some limitations in using the cache for all services. Especially when facing high-traffic business scenarios, the following problems exist:

[0056] I. The cache architecture is single and the performance is limited: Different services may have different requirements for cache capacity, expiration time, data consistency requirements, etc. However, in the case of a single cache structure, it is difficult to perform personalized cache settings for different business scenarios, resulting in unbalanced resource allocation and unable to optimally utilize cache resources.

[0057] II. The existing strategy is to use the cache for all services and cannot be dynamically adjusted: When all services use the same cache structure, high-traffic services may cause huge pressure on the cache system. When all services use the same cache structure, the data of low-traffic services may be overwhelmed by the data of high-traffic services, resulting in a decrease in cache hit rate.

[0058] Therefore, the solution of the embodiment of this application starts from the actual problem of improving the performance and flexibility of data caching in high-traffic business scenarios, combines the adaptive allocation ability of the multi-level cache architecture for data caching and the analysis ability of the access filter for traffic priority, and designs a method for multi-level cache adaptation based on traffic priority to solve the technical problems of poor data caching performance and low flexibility in high-traffic business scenarios, and improve the overall performance and flexibility of the data caching system.

[0059] Specifically, referring to Figure 1 , Figure 1 is a schematic diagram of the function modules of the terminal device to which the data caching management device of this application belongs. This data caching management device can be a device independent of the terminal device and capable of performing data caching management, and it can be carried on the terminal device in the form of hardware or software. This terminal device can be an intelligent mobile terminal with data processing functions such as a mobile phone or a tablet computer, or a fixed terminal device or a server with data processing functions, etc.

[0060] In this embodiment, the terminal device to which the data cache management device belongs at least includes an output module 110, a processor 120, a memory 130, and a communication module 140.

[0061] The memory 130 stores an operating system and a data cache management program. When the data cache management device receives a query request, it can determine the corresponding target interface; the detection result obtained by detecting whether the target interface exists through a preset access filter; based on the detection result, query the data of a preset multi-level cache database to obtain the target data, and information such as a multi-level cache database created based on a primary cache database, a secondary cache database, and a tertiary cache database is stored in the memory 130; the output module 110 can be a display screen, etc. The communication module 140 can include a WIFI module, a mobile communication module, a Bluetooth module, etc., and communicates with external devices or servers through the communication module 140.

[0062] Among them, when the data cache management program in the memory 130 is executed by the processor, the following steps are implemented:

[0063] When receiving a query request, determine the corresponding target interface;

[0064] Detect whether the target interface exists through a preset access filter to obtain a detection result;

[0065] Based on the detection result, query the data of a preset multi-level cache database to obtain the target data.

[0066] Further, when the data cache management program in the memory 130 is executed by the processor, the following steps are also implemented:

[0067] If the target interface does not exist, query the data of the secondary cache database;

[0068] If the data of the secondary cache database does not exist, query the data of the tertiary cache database;

[0069] If the data of the tertiary cache database exists, pull the data of the tertiary cache database through the secondary cache database;

[0070] If the target interface exists, query the data of the primary cache database;

[0071] If the data of the primary cache database does not exist, execute the step of querying the data of the secondary cache database.

[0072] Further, when the data cache management program in the memory 130 is executed by the processor, the following steps are also implemented:

[0073] Detect whether the target data exists through the access filter;

[0074] If the target data exists, push the target data to the first-level cache database through the second-level cache database.

[0075] Furthermore, when the data cache management program in the memory 130 is executed by the processor, the following steps are further implemented:

[0076] When detecting changed data, delete the data in the first-level cache database;

[0077] Asynchronously write the changed data to the second-level cache database and the third-level cache database;

[0078] Push the changed data to the first-level cache database through the second-level cache database.

[0079] Furthermore, when the data cache management program in the memory 130 is executed by the processor, the following steps are further implemented:

[0080] Obtain the key quantity of the second-level cache database and the key quantity of the third-level cache database respectively;

[0081] When detecting that the key quantity of the second-level cache database is not equal to the key quantity of the third-level cache database, pull the data of the third-level cache database through the second-level cache database, and delete the data in the first-level cache database.

[0082] Furthermore, when the data cache management program in the memory 130 is executed by the processor, the following steps are further implemented:

[0083] Based on a preset timer, synchronize the data in the preset data source to the third-level cache database and the second-level cache database; and / or

[0084] Based on a preset listening strategy, synchronize data to the third-level cache database and the second-level cache database through the data source.

[0085] Furthermore, when the data cache management program in the memory 130 is executed by the processor, the following steps are further implemented:

[0086] Based on a preset sliding window, count the request times of several service interfaces;

[0087] Sort the request times to obtain the interface sort list of the several service interfaces;

[0088] Determine the access ratio according to the preset local resources and data volume;

[0089] Determine the access filter based on the access ratio and the interface sort list.

[0090] In this embodiment, through the above solution, specifically, when a query request is received, the corresponding target interface is determined; the existence of the target interface is detected by a preset access filter to obtain a detection result; based on the detection result, the data in the preset multi-level cache database is queried to obtain the target data. Based on the solution of the present application, by caching the data in high-traffic business scenarios in terms of interfaces, the API interfaces with high traffic can be preferentially cached, and a multi-level cache database is set up, so that the performance and data capacity of each level of cache database for data caching are different, solving the technical problems of poor data caching performance and low flexibility in high-traffic business scenarios, improving the overall performance and flexibility of the data caching system, and thus reducing the system load.

[0091] Based on the above terminal device architecture but not limited to the above architecture, the method embodiments of the present application are proposed. It should be noted that the embodiments in the method of the present application can be combined with each other into a complete solution.

[0092] Referring to Figure 2 , Figure 2 is a schematic flowchart of the first exemplary embodiment of the data caching management method of the present application. The data caching management method includes:

[0093] Step S210, when a query request is received, determine the corresponding target interface;

[0094] The execution subject of the method in this embodiment can be a data caching management device, or a data caching management terminal device or server. In this embodiment, a data caching management device is taken as an example, and this data caching management device can be integrated on terminal devices such as smart phones and tablet computers with data processing functions.

[0095] Starting from the actual problem of improving the performance and flexibility of data caching in high-traffic business scenarios, combining the adaptive allocation ability of the multi-level cache architecture for data caching and the analysis ability of the access filter for traffic priority, this embodiment designs a method for multi-level cache adaptation based on traffic priority, mainly realizing the management of data caching, especially data caching, solving the technical problems of poor data caching performance and low flexibility in high-traffic business scenarios, and improving the overall performance and flexibility of the data caching system.

[0096] Step S220, detect whether the target interface exists through a preset access filter to obtain a detection result;

[0097] Specifically, the access filter is used to quickly determine whether an interface exists in the interface sorted list (API list). The detection result indicates whether the target interface can be accessed, that is, high-traffic requests can be preferentially processed according to the priority of the traffic. Among them, the access filter can divide the traffic into different priorities according to business requirements and traffic characteristics. For example, the priority of the traffic can be determined according to indicators such as access frequency, business importance, and data popularity.

[0098] Exemplarily, when a user makes a data query, a corresponding query request is generated. The query request will call the corresponding interface. Through a certain period of time, several access requests of several users, the corresponding number of access requests in this time period can be obtained. Then, the local cache access filter sorts according to the number of access requests of the API, and an API list sorted from high to low according to the number of requests can be obtained. Then, take the top N APIs with the access ratio in this API list and store them in the Bloom filter as the local cache access filter.

[0099] Step S230, based on the detection result, query the data in the preset multi-level cache database to obtain the target data.

[0100] Specifically, the target data is the data corresponding to the query request; the local cache is used to provide low-latency and high-throughput data access, the secondary cache is used to provide distributed cache storage and management, and the tertiary cache is used to provide data backup and persistence. In the multi-level cache architecture, according to the priority division of the traffic, different levels of caches can be configured for different levels of traffic. High-priority traffic can directly access the faster local cache, medium-priority traffic can access the relatively fast intermediate cache (such as Redis), and low-priority traffic can access the larger-capacity but relatively slower-backup cache (such as ES).

[0101] Exemplarily, this embodiment uses a multi-level cache database to manage the cache data. Figure 3 The multi-level cache architecture of the multi-level cache database for the data cache management method of this application includes: MongoDB as the data source, Caffeine is used as the first-level cache (local cache), Redis as the second-level cache, and Elasticsearch (ES) is used as the third-level cache to act as the data fallback solution for Redis to ensure the high availability of the data query service.

[0102] Among them, the first-level cache (local cache): Caffeine is a high-performance Java-based caching library used as the local cache for the client. It stores frequently used data in the application memory, enabling very fast data reading. The advantage of the local cache is that data can be directly read from the memory without going through network transmission or disk access, thus having low latency and high throughput.

[0103] The second-level cache (Redis): Redis is a memory-based data storage system widely used as a distributed cache. In a multi-level cache architecture, Redis serves as the second-level cache, meeting the requirements of high-concurrency read and write and data sharing that the local cache cannot handle. Redis has fast read and write speeds and rich data structure support, capable of storing and managing a large amount of business data, thereby reducing the load pressure on the database.

[0104] The third-level cache (Elasticsearch, ES): Elasticsearch is a distributed search and analysis engine with high scalability and powerful query capabilities. In a multi-level cache architecture, ES is used as the third-level cache as a fallback solution for Redis data. When Redis fails to hit the cache, the system retrieves data from ES and caches it in Redis for subsequent fast access. This design of the three-level cache ensures the high availability and persistence of data. Even if Redis fails or data is lost, data can still be recovered from ES.

[0105] Through the multi-level cache architecture, the system can make full use of the characteristics and advantages of different-level caches to achieve higher performance and scalability. This hierarchical cache architecture enables the system to maintain an efficient data query service under different scenarios and loads.

[0106] In this embodiment, through the above solution, specifically, when a query request is received, the corresponding target interface is determined; the target interface is detected by a preset access filter to obtain a detection result; based on the detection result, data in a preset multi-level cache database is queried to obtain target data. Based on the solution of this application, by caching data in high-traffic business scenarios from the dimension of the interface, it is possible to preferentially cache high-traffic API interfaces and set up a multi-level cache database, such that the performance and data capacity of data caching for each level of cache database are different, solving the technical problems of poor data caching performance and low flexibility in high-traffic business scenarios, improving the overall performance and flexibility of the data caching system, and thereby reducing the system load.

[0107] Refer to Figure 4 , Figure 4 is the flowchart of the second exemplary embodiment of the data caching management method of this application. Based on the aboveFigure 2 , 3 In the embodiment shown, the multi-level cache database includes a first-level cache database, a second-level cache database, and a third-level cache database. Step S230, based on the detection result, querying the data of the preset multi-level cache database includes:

[0108] Step S410, if the target interface does not exist, query the data of the second-level cache database;

[0109] Specifically, the detection result is used to indicate whether the interface exists in the access filter (Bloom filter).

[0110] Further, in step S410, if the target interface does not exist, after querying the data of the second-level cache database, it further includes:

[0111] Detect whether the target data exists through the access filter;

[0112] If the target data exists, push the target data to the first-level cache database through the second-level cache database.

[0113] Specifically, the result queried from Redis is judged by the Bloom filter. If the result already exists in the Bloom filter, it means the data has been cached and the target data can be directly returned. If the target data does not exist in the Bloom filter, the target data is stored in the local cache and the result is returned to the client.

[0114] Step S420, if the data of the second-level cache database does not exist, query the data of the third-level cache database;

[0115] Specifically, when querying, first judge whether the API to be queried exists in the Bloom filter. If it does not exist, directly query the Redis database. If it exists in the Bloom filter, then first query the local cache. If the data exists in the local cache, return the target data. If the data does not exist in the local cache, continue to query the Redis database and judge the target data through the Bloom filter and then store it in the local cache.

[0116] Step S430, if the data of the third-level cache database exists, pull the data of the third-level cache database through the second-level cache database;

[0117] Step S440, if the target interface exists, query the data of the first-level cache database;

[0118] Step S450, if the data of the first-level cache database does not exist, execute the step of querying the data of the second-level cache database.

[0119] Exemplarily, referring to Figure 5 , Figure 5 which is a schematic diagram of data query for the data cache management method of this application. First, through the sliding window algorithm, the number of requests of the current service's API within a certain time window is statistically counted and sorted according to the number of requests; then, the API admission ratio N allowed by the local cache is determined according to the local resources and the amount of data; then, a list is sorted according to the access times of the APIs, and the top N APIs with the admission ratio are taken and stored in the Bloom filter as the local cache admission filter; then, when querying, it is judged whether the current interface exists in the Bloom filter. If the current interface does not exist, redis is directly queried; if the data cannot be queried from redis, ES is queried; if the current interface exists, the local cache is queried. If the local cache data does not exist, redis is queried again; finally, the target data in redis is judged by the Bloom filter and then stored in the local cache, and the target data is returned.

[0120] Through the above solution in this embodiment, compared with the single-level cache, through the multi-level cache strategy with adaptive traffic characteristics, the multi-level cache can be more flexible and have higher performance, ensuring data consistency; through the data cache management with traffic priority in terms of interfaces, the cache hit rate and the concurrency ability of the system can be improved, thereby solving the redis io bottleneck problem of instantaneous high traffic.

[0121] Referring to Figure 6 , Figure 6 which is a schematic flowchart of the third exemplary embodiment of the data cache management method of this application. Based on the above Figure 2 , 4 shown embodiments, the data cache management method further includes the following steps:

[0122] Step S610, when detecting changed data, delete the data in the primary cache database;

[0123] Specifically, first, when the local cache data changes, a write cache operation is performed to delete the local cache, and then it is written to redis and asynchronously written to ES. The data modification in redis pushes the changed data to the primary cache (local cache) through redis's pub / sub for data update; then, for data update, the changed data is pushed to the local cache through redis's pub / sub mode; then, for the data pushed to the local cache, first judge whether the current interface is in the Bloom filter. If the current interface does not exist in the Bloom filter, this change is ignored; otherwise, the changed data is saved to the local cache and an expiration time is set.

[0124] Step S620, asynchronously write the changed data to the secondary cache database and the tertiary cache database;

[0125] Exemplarily, when data changes occur, a write cache operation is first performed. This operation first deletes the corresponding data in the local cache to ensure that the latest data can be loaded. Then, the updated data is written to Redis and asynchronously written to ES. This can ensure that the data in Redis is the latest, and the asynchronous writing to ES can achieve data persistence and backup. At the same time, through the pub / sub mechanism of Redis, the message of data change is pushed to the first-level cache (local cache) for data update. Real-time data update can be achieved to ensure that the data in the local cache is synchronized with the data source.

[0126] Step S630, push the changed data to the first-level cache database through the second-level cache database.

[0127] Specifically, before pushing the data to the local cache, it is first necessary to determine whether the current interface is in the Bloom filter. The role of the Bloom filter is to quickly determine whether an element exists in a set. If the current interface is not in the Bloom filter, it means that the interface is not within the cache access range, and this change operation will be ignored. If the current interface exists in the Bloom filter, the changed data is saved to the local cache and an appropriate expiration time is set. By using the Bloom filter to judge and filter the pushed data, it is ensured that the qualified data is stored in the local cache, improving the efficiency and real-time performance of data update.

[0128] Among them, the usage mode of the cache can be dynamically adjusted according to the real-time change of traffic and the adjustment of priority. For example, when the priority of a certain traffic changes, the usage strategy of the cache can be re-adjusted according to the new priority. Or, according to the cache hit rate and performance metrics, monitor the traffic situation in real time and dynamically adjust parameters such as the cache capacity.

[0129] It should be noted that the configuration details will be adjusted according to the cache system and data source used. At the same time, the parameter settings of the Bloom filter also need to be optimized according to the actual situation to ensure accuracy and performance.

[0130] Through the above solution in this embodiment, specifically, through the write cache operation, it is ensured that the latest data can be loaded into the cache, and the data change and update of the local cache can be realized; through the pub / sub mechanism of Redis, the changed data is pushed to the local cache, ensuring the real-time performance of the cache data and the data consistency of the multi-level cache.

[0131] Refer to Figure 7 , Figure 7 is the flowchart of the fourth exemplary embodiment of the data cache management method of this application. Based on the above Figure 2 , 4In the illustrated embodiment, the data cache management method further includes the following steps:

[0132] Step S710: Obtain the key quantity of the secondary cache database and the key quantity of the tertiary cache database respectively;

[0133] Specifically, referring to Figure 8 , Figure 8 is a schematic diagram of cache data self-healing for the data cache management method of this application. If it is a local cache change, the corresponding data in the local cache is directly deleted, ensuring the consistency of the data in the local cache with the data source.

[0134] If it is a data change in Redis, the watchdog task dynamically monitors the data change, compares whether the number of keys in Redis and ES is equal, and if they are equal, continues to monitor.

[0135] Step S720: When it is detected that the key quantity of the secondary cache database is not equal to the key quantity of the tertiary cache database, pull the data of the tertiary cache database through the secondary cache database and delete the data of the primary cache database.

[0136] Specifically, if the number of keys in Redis and ES is not equal, the data synchronization mechanism is triggered, actively pull the data in ES and synchronize it to Redis, and clear the data in the local cache. If a change triggers data synchronization, configure the maximum number of synchronizations allowed per day, and update the synchronization count every morning. Every morning, the data is synchronized from the data source to ES and then to Redis. Monitor the data of the data source, and when a change occurs, incrementally synchronize the changed data to ES and Redis. Redis synchronizes the update information to the local cache through the push mode in the push / pull mode.

[0137] Among them, a watchdog task dynamically monitors the data change in Redis. This task compares whether the number of keys (key) in Redis and ES is equal. If the number of keys is equal, it means that the data in Redis and ES is consistent, and the task continues to monitor the change. If the number of keys is not equal, it means that the data is inconsistent, and a data synchronization operation will be triggered.

[0138] Redis synchronizes the update information to the local cache through the push mode (pub / sub). When the data in Redis changes, the changed message is pushed to the local cache that subscribes to the message, so that the data in the local cache remains real-time.

[0139] Through the above solution, in this embodiment, the self-healing ability of the cache system can be realized specifically through the self-healing repair task. By means of regular full synchronization and incremental synchronization operations, the consistency between the cache data and the data source is maintained. By listening for data changes and triggering synchronization operations, real-time data updates are achieved. At the same time, the data consistency of the multi-level cache and the repair of abnormal situations are ensured through the push mode and the pull mode.

[0140] Referring to Figure 9 , Figure 9 is a schematic flowchart of the fifth exemplary embodiment of the data cache management method of this application. Based on the above Figure 2 , 4 illustrated embodiment, the data cache management method further includes the following steps:

[0141] Step S910, based on a preset timer, synchronize the data in the preset data source to the tertiary cache database and the secondary cache database;

[0142] Specifically, referring to Figure 10 , Figure 10 is a schematic diagram of cache data synchronization of the data cache management method of this application. In the embodiment of this application, the synchronization includes full synchronization and incremental synchronization. Among them, the full synchronization strategy can be that the data is fully synchronized from the data source MongoDB database to ES and then to Redis at 3 o'clock every morning.

[0143] Exemplarily, at a fixed time point (such as 3 o'clock every morning), all the data is synchronously and completely written from the data source (MongoDB database) to the target cache (ES and Redis) once. That is, all the data will be re-extracted and written into the cache to ensure that the data in the cache is consistent with the data in the data source. This synchronization method is applicable to scenarios where the cache needs to be updated regularly, such as when the data changes greatly every day.

[0144] And / or step S920, based on a preset listening strategy, synchronize the data to the tertiary cache database and the secondary cache database through the data source.

[0145] Specifically, the incremental update strategy can be that the data changes in MongoDB are used to dynamically update the data in ES and Redis through the listening strategy.

[0146] Exemplarily, incremental update means that through a listening policy, when the data in the data source (MongoDB) changes, the changed data is updated to the target cache (ES and Redis) in real time. This method avoids the overhead of full synchronization and only updates the changed data to the cache to maintain the real-time nature of the cache. Incremental update is usually implemented by subscribing to database change events or using mechanisms such as message queues. This synchronization method is suitable for scenarios with high requirements for data changes and the need for real-time response.

[0147] Among them, when it is found that the number of keys in Redis and ES is not equal, a data synchronization operation will be triggered. Actively pull data from ES and synchronize the data to Redis to ensure that the data in Redis is consistent with ES. At the same time, the operation of synchronizing data to the local cache will not be triggered, and the data in the local cache will be cleared, which can avoid using expired or incorrect local cache data in the case of data inconsistency.

[0148] Through the above solution in this embodiment, specifically, by using a combination of full synchronization and incremental update, the performance and real-time requirements of data synchronization can be balanced. Full synchronization ensures that all data is synchronized to the cache at a fixed time point, providing a consistent data view and the initial state of data loading. Incremental update maintains real-time synchronization between the cache and the data source, providing timely response and lower synchronization latency.

[0149] Refer to Figure 11 , Figure 11 is the flowchart of the sixth exemplary embodiment of the data cache management method of this application. Based on the above Figure 2 shown embodiment, before step S220, detecting whether the target interface exists through a preset admission filter and obtaining a detection result, further includes:

[0150] Step S1110, based on a preset sliding window, count the number of requests of several service interfaces;

[0151] Specifically, the sliding window algorithm is an algorithm for counting the number of requests within a certain time window. It sets a time window with a fixed length, and records the number of requests for each API within this window. As time goes by, the window slides, new request counts are recorded, and old request counts are discarded. In this way, the request frequency of the API can be statistically analyzed in real time and sorted according to the number of requests.

[0152] Refer to Figure 12 , Figure 12This is the access judgment flowchart of the data cache management method of this application. The API access times are counted through the time window algorithm, and a certain percentage of APIs are added to the Bloom filter through the hash algorithm as the access condition for local cache usage. The data in the Bloom filter will be reset in the next time window.

[0153] Step S1120: Sort the request times to obtain the interface sorted list of the several service interfaces.

[0154] Specifically, when a user queries data, the corresponding interface will be called. Through the several access requests of several users within a period of time, the corresponding access request times within this time period can be obtained. Then, the local cache access filter sorts according to the API access request times, and an API list sorted from high to low according to the request times can be obtained.

[0155] Step S1130: Determine the access ratio according to the preset local resources and data volume.

[0156] Specifically, the local cache access ratio (N) is determined according to the local resources and the data volume of the request access times. It represents the ratio of the APIs allowed to be cached in the overall API list. For example, if the access ratio is set to 30%, only the APIs with high request times among the top 30% of the API list will be put into the local cache.

[0157] Step S1140: Determine the access filter based on the access ratio and the interface sorted list.

[0158] Specifically, add the API to the Bloom filter as the local cache access condition: Add the API to the Bloom filter through the hash algorithm according to a certain percentage. Adding a certain percentage of APIs to the Bloom filter means that these APIs can be cached and meet the access conditions.

[0159] It should be noted that after the end of each time window, the data in the Bloom filter is reset. That is, each time window has its own Bloom filter for recording the APIs allowed to be cached within the current time window. Resetting the Bloom filter can clear the data of the previous time window and prepare for the next time window. Among them, the length of the time window, the access percentage, and the configuration parameters of the Bloom filter should be adjusted according to actual needs and system performance.

[0160] In this embodiment, through the above solution, specifically, within each time window, the access times of the APIs are counted; according to the set percentage, a part of the APIs are selected and added to the Bloom filter; the APIs in the Bloom filter represent the APIs that can be cached and meet the admission conditions; after the time window ends, the data in the Bloom filter is reset to prepare for the statistics and admission judgment of the next time window, and the APIs can be sorted according to the request times, and it can be dynamically determined which APIs should be cached. Using the Bloom filter can reduce the access times to Redis and improve the query efficiency. At the same time, by restricting the admission ratio of the local cache, the waste of resources caused by excessive caching can be avoided.

[0161] In addition, an embodiment of the present application also proposes a data cache management device, which includes:

[0162] An acquisition module, configured to determine a corresponding target interface when receiving a query request;

[0163] An admission module, configured to detect whether the target interface exists through a preset admission filter to obtain a detection result;

[0164] A query module, configured to query the data of a preset multi-level cache database based on the detection result to obtain target data.

[0165] For the principle and implementation process of data cache management in this embodiment, please refer to the above embodiments and will not be elaborated here.

[0166] In addition, an embodiment of the present application also proposes a terminal device, which includes a memory, a processor, and a data cache management program stored on the memory and executable on the processor. When the data cache management program is executed by the processor, it implements the steps of the data cache management method as described above.

[0167] Since when the data cache management program is executed by the processor, all the technical solutions of the foregoing embodiments are adopted, it has at least all the beneficial effects brought by all the technical solutions of the foregoing embodiments, which will not be elaborated here one by one.

[0168] In addition, an embodiment of the present application also proposes a computer-readable storage medium, on which a data cache management program is stored. When the data cache management program is executed by the processor, it implements the steps of the data cache management method as described above.

[0169] Since when the data cache management program is executed by the processor, all the technical solutions of the foregoing embodiments are adopted, it has at least all the beneficial effects brought by all the technical solutions of the foregoing embodiments, which will not be elaborated here one by one.

[0170] Compared with the prior art, the data cache management method, device, terminal device and storage medium proposed in the embodiments of the present application determine the corresponding target interface when receiving a query request; detect whether the target interface exists through a preset access filter to obtain a detection result; and query the data in the preset multi-level cache database based on the detection result to obtain the target data. Based on the solution of the present application, by caching the data in high-traffic business scenarios in terms of interfaces, it is possible to preferentially cache high-traffic API interfaces and set up a multi-level cache database, such that the performance and data capacity of each level of cache database for data caching are different, solving the technical problems of poor data cache performance and low flexibility in high-traffic business scenarios, improving the overall performance and flexibility of the data cache system, and thereby reducing the load of the system.

[0171] It should be noted that in this article, the terms "including", "comprising" or any other variant thereof are intended to cover non-exclusive inclusion, such that a process, method, article or system including a series of elements not only includes those elements but also includes other elements not expressly listed, or further includes elements inherent to such process, method, article or system. Without further limitation, an element defined by the phrase "including a..." does not exclude the existence of additional identical elements in the process, method, article or system including such element.

[0172] The serial numbers of the above embodiments of the present application are only for description and do not represent the advantages or disadvantages of the embodiments.

[0173] Through the description of the above embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus a necessary general hardware platform. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, in essence, or the part that contributes to the prior art can be embodied in the form of a software product. The computer software product is stored in a storage medium as described above (such as ROM / RAM, magnetic disk, optical disc) and includes several instructions for causing a terminal device (which can be a mobile phone, computer, server, controlled terminal, or network device, etc.) to execute the methods of each embodiment of the present application.

[0174] The above are only the preferred embodiments of the present application and do not limit the patent scope of the present application. Any equivalent structural or equivalent process transformation made by using the specification and drawings of the present application, or directly or indirectly applied in other related technical fields, shall be equally included in the patent protection scope of the present application.

Claims

1. A data cache management method, characterized in that, The described data cache management method includes the following steps: When a query request is received, determine the corresponding target interface; Detect whether the target interface exists through a preset access filter to obtain a detection result; Based on the detection result, query the data in the preset multi-level cache database to obtain target data; The multi-level cache database includes a first-level cache database, a second-level cache database, and a third-level cache database; The step of querying the data in the preset multi-level cache database based on the detection result includes: If the target interface does not exist, query the data in the second-level cache database; If the target interface exists, query the data in the first-level cache database; Before the step of detecting whether the target interface exists through a preset access filter to obtain a detection result, it further includes: Based on a preset sliding window, count the request times of several service interfaces; Sort the request times to obtain an interface sorted list of the several service interfaces; Determine an access ratio according to preset local resources and data volume; Based on the access ratio and the interface sorted list, determine the access filter.

2. The data cache management method according to claim 1, characterized in that After the step of querying the data in the second-level cache database, it further includes: If the data in the second-level cache database does not exist, query the data in the third-level cache database; If the data in the third-level cache database exists, pull the data in the third-level cache database through the second-level cache database; After the step of querying the data in the first-level cache database, it further includes: If the data in the first-level cache database does not exist, execute the step of querying the data in the second-level cache database.

3. The data cache management method according to claim 1, characterized in that After the step of querying the data in the second-level cache database, it further includes: Detect whether the target data exists through the access filter; If the target data exists, push the target data to the first-level cache database through the second-level cache database.

4. The data cache management method according to claim 2, wherein The data cache management method further includes the following steps: When detecting changed data, delete the data in the first-level cache database; Asynchronously write the changed data to the second-level cache database and the third-level cache database; Push the changed data to the first-level cache database through the second-level cache database.

5. The data cache management method according to claim 2, wherein The data cache management method further includes the following steps: Obtain the key quantity of the second-level cache database and the key quantity of the third-level cache database respectively; When detecting that the key quantity of the second-level cache database is not equal to the key quantity of the third-level cache database, pull the data in the third-level cache database through the second-level cache database and delete the data in the first-level cache database.

6. The data cache management method according to claim 2, characterized in that The data cache management method further includes the following steps: Based on a preset timer, synchronize the data in the preset data source to the third-level cache database and the second-level cache database; and / or Based on a preset listening strategy, synchronize the data to the third-level cache database and the second-level cache database through the data source.

7. A data cache management device, characterized in that, The data cache management device includes: An acquisition module, configured to determine a corresponding target interface when a query request is received; An access module, configured to detect whether the target interface exists through a preset access filter, and obtain a detection result; A query module, configured to query data in a preset multi-level cache database based on the detection result, and obtain target data; The multi-level cache database includes a first-level cache database, a second-level cache database, and a third-level cache database; The querying data in the preset multi-level cache database based on the detection result includes: If the target interface does not exist, query the data in the second-level cache database; If the target interface exists, query the data in the first-level cache database; Before detecting whether the target interface exists through the preset access filter and obtaining the detection result, it further includes: Based on a preset sliding window, count the request times of several service interfaces; Sort the request times to obtain an interface sorting list of the several service interfaces; Determine an access ratio according to preset local resources and data volume; Determine the access filter based on the access ratio and the interface sorting list.

8. A terminal device, characterized in that, The terminal device includes a memory, a processor, and a data cache management program stored on the memory and executable on the processor. When the data cache management program is executed by the processor, it implements the steps of the data cache management method according to any one of claims 1-6.

9. A computer-readable storage medium, characterized in that, A data cache management program is stored on the computer-readable storage medium. When the data cache management program is executed by the processor, it implements the steps of the data cache management method according to any one of claims 1-6.

Images