Responsive cache preheating method based on data and authority change notification

Through the responsive cache warm-up method based on data and permission change notifications, the problem of untimely cache updates and insufficient retry mechanisms is solved, the cache hit rate and system performance are improved, and the cache efficiency and reliability are ensured.

CN118964404AActive Publication Date: 2024-11-15BEIJING BAILONG MAYUN TECH CO LTD
View PDF 2 Cites 0 Cited by

Patent Information

Application Number
CN202411016273.5
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-27
Publication Date
2024-11-15
Estimated Expiration
2044-07-27

AI Technical Summary

Technical Problem

The existing cache system cannot be updated in time when data and permissions change, and lacks a failed retry mechanism, resulting in low cache hit rate and unsuccessful updates.

Method used

Responsive cache warm-up method based on data and permission change notifications is adopted to ensure that cache updates are timely and successful by building a query engine, listening to data set events, building event processing logic, timed consumption task queues and thread pools are processed.

Benefits of technology

It improves the timeliness and success rate of cache warm-up, enhances the effectiveness of cache, prevents excessive server pressure or downtime, and achieves high scalability.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118964404A_ABST
    Figure CN118964404A_ABST
Patent Text Reader

Abstract

The invention provides a responsive cache preheating method based on data and permission change notification. The responsive cache preheating method comprises the following steps: S1, constructing a query engine, and preferentially reading a result from a redis cache by utilizing the query engine; s2, inserting a query record in the back-end request, wherein the query record is used for selecting preheating target data; s3, updating the data of the data set: if the updating is successful, writing a new event into a data set event table; s4, monitoring a data set and a permission change event; s5, constructing data set event processing logic; the accuracy and timeliness of the cache preheating time are improved by introducing a notification mechanism, each cache preheating request is executed by using an independent task, a perfect failure retry strategy is provided, the high success rate of the preheating request is ensured, the data needing to be preheated are selected by using query statistics and a priority strategy, and the efficiency is improved. A user query path is simulated to construct query parameters to preheat the cache, and the cache effectiveness is high.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to a responsive cache preheating method, in particular to a responsive cache preheating method based on data and permission change notification, belonging to the technical field of data cache preheating. Background Art

[0002] Data cache preheating is a technical means to improve data access efficiency, especially in computer storage and database management. Cache preheating usually refers to loading data into the cache before the data is actually needed so that it can be quickly accessed when needed; cache preheating can significantly improve data retrieval performance and reduce access latency, which is especially important for applications that need to process large amounts of data. During the preheating process, various algorithms and strategies can be used to decide which data should be loaded into the cache, which usually involves characteristics such as the access frequency and modification frequency of the data.

[0003] When querying business data, the current cache system directly queries the database if it is the first query. After the data is found, it is saved to the cache system and the result is returned. When the same parameters are used to request data next time, the cache will be hit, and the data will be queried directly from the cache system. When an operation that modifies business data is performed, the cache data will be cleared synchronously. The cache is preheated regularly, and the cache is preheated by inferring parameter changes based on yesterday's query records instead of user queries. However, the following problems still exist:

[0004] 1. Unable to perceive data and permission changes in time, and cache updates are not timely;

[0005] 2. The cache update logic lacks a complete failure retry mechanism, and cannot guarantee the success rate of cache updates;

[0006] 3. After business data and permissions are changed, the cache is cleared and not updated, resulting in a low cache hit rate;

[0007] 4. In the preheating logic, the query parameters are constructed to preheat the cache by inferring parameter changes based on yesterday's query records. The parameter inference is inaccurate and the cache availability is low.

[0008] To this end, a responsive cache warming method based on data and permission change notification is proposed. Summary of the invention

[0009] In view of this, the present invention provides a responsive cache preheating method based on data and permission change notifications to solve or alleviate the technical problems existing in the prior art and at least provide a beneficial choice.

[0010] The technical solution of the embodiment of the present invention is implemented as follows: a responsive cache preheating method based on data and permission change notification, comprising the following steps:

[0011] S1. Build a query engine - use the query engine to read results from the redis cache first;

[0012] S2. Insert query records into the backend request to select preheat target data for use;

[0013] S3, Data set data update - if the update is successful, the updated event will be written into the data set event table;

[0014] S4, monitor data set and permission change events;

[0015] S5, build data set event processing logic;

[0016] S6. Regularly consume and preheat the task queue;

[0017] S7. Before submitting the task to the thread pool, update the task status to "in process" - refresh the cached data;

[0018] S8. After receiving the request, the query engine service checks the request type. If it is a cache refresh request, it executes the cache refresh logic and directly queries the data from the database. After the query is successful, it updates the value of the corresponding key in redis.

[0019] Further preferably, in S1, when there is no result corresponding to the request in the redis cache, the database query task is automatically submitted through the query engine, and after the query is successful, the engine automatically saves the result to the redis cache.

[0020] Further preferably, in S2, when a user queries data in the system, the backend request initiated contains logic for counting the user's data query records. Whenever a user queries a piece of data, the system automatically inserts a query record.

[0021] Among them, the query record is regenerated based on the user ID and data ID as the logical library primary key.

[0022] Further preferably, in S3, the data set data is updated every 12-36 hours.

[0023] Further preferably, in S4, a scheduled task is used to query unprocessed time records, and then the time processing logic is cyclically called to process events.

[0024] Further preferably, in S5, when the data set event processing logic is executed, firstly, the data to be preheated is selected from the data query record according to the data set ID; then, the data cache preheating task information is first put into the priority task queue according to the priority strategy; then, the distributed lock is acquired, and after the acquisition is successful, all the data in the priority queue are found out in reverse order according to the score;

[0025] The query range is data within 6 to 8 days, that is, date>=T-(6 to 8), and the data set ID = the data set ID currently being processed;

[0026] The priority strategy is: the more recent the query time, the higher the priority. The more recent the query time, the more active it is recently and needs to be refreshed first.

[0027] Among them, if the total number of cache warm-up task information total Count<(3 to 5), all of them will be processed by the fast processing queue. Otherwise, the data will be divided into 3 to 5 parts, 1 part for the fast processing queue, and 2-4 parts for the slow processing queue, that is: index=total Count / (3 to 5), [0, index] for the fast processing task queue, and (index, total Count-1] for the slow processing task queue.

[0028] Further preferably, in S6, before consumption, failed tasks and tasks in a long-time processing state are added back to the fast and slow task queues, and then elements are popped from the queues in a loop, and cache preheating logic and cache generation are executed in the loop. The execution of tasks is accelerated in parallel using a thread pool, and the task status is recorded and updated during task execution, and the task is stopped when the queue is empty.

[0029] The number of threads in the thread pool of the fast queue is greater than that of the slow queue. The number of threads is configurable. The queue uses a synchronous blocking queue. When the thread pool is full, the thread that submits the task will be automatically suspended and will be called up again when the thread pool is idle.

[0030] The task status includes initial state, processing, success and failure.

[0031] Further preferably, in S7, the update status before the task is executed is in process, and the cache preheating logic constructs the global filter parameters according to the default parameters in the analysis dashboard where the user data is located, and then calls the data query and cache refresh methods, which are responsible for constructing the downstream RPC interface parameters of the data query and judging the status of the task according to the parameters;

[0032] The judgment method is: if the RPC interface call is successful and the service request status code is 200, it means that the cache refresh is successful, and then the status of the cache preheating task is updated to success, otherwise the status of the cache preheating task is updated to failure.

[0033] Further preferably, in S7, the cache preheating logic captures an overall exception, and if the exception is captured, the task status is updated to failure, waiting for the next cycle of processing.

[0034] Further preferably, in S8, key is formed by concatenating the md5 values ​​of the sql generated by data compilation, ensuring one sql for one cache.

[0035] The embodiment of the present invention has the following advantages due to the adoption of the above technical solution:

[0036] 1. The present invention improves the accuracy and timeliness of cache preheating timing by introducing a notification mechanism, and each cache preheating request is executed using an independent task, with a complete failure retry strategy, ensuring a high success rate of preheating requests. By using query statistics and priority strategies to select data that needs to be preheated, and simulating user query paths to construct query parameters to preheat the cache, the cache has high effectiveness.

[0037] 2. The present invention extracts the cache update logic into an independent scheduling task, cooperates with the retry mechanism to ensure the success rate of the task, and cooperates with the data set event processing logic to use the peak shaving strategy, which can effectively prevent the server from responding slowly or crashing due to high pressure.

[0038] 3. The present invention is based on an event notification mechanism and has strong scalability.

[0039] The above summary is for illustrative purposes only and is not intended to be limiting in any way. In addition to the illustrative aspects, embodiments and features described above, further aspects, embodiments and features of the present invention will be readily apparent by reference to the accompanying drawings and the following detailed description. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0041] Figure 1 This is a schematic diagram of the first part of the cache refresh logic of the present invention;

[0042] Figure 2 This is a schematic diagram of the second part of the cache refresh logic of the present invention;

[0043] Figure 3 It is a schematic diagram of the first part of the data scheduling logic of the present invention;

[0044] Figure 4 This is a schematic diagram of the second part of the data scheduling logic of the present invention. DETAILED DESCRIPTION

[0045] In the following, only some exemplary embodiments are briefly described. As those skilled in the art will appreciate, the described embodiments may be modified in various ways without departing from the spirit or scope of the present invention. Therefore, the drawings and descriptions are considered to be exemplary and non-restrictive in nature.

[0046] It should be noted that the terms "first", "second", "symmetrical", "array", etc. are only used to distinguish between description and position description purposes, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the definition of "first", "symmetrical", etc. can explicitly or implicitly include one or more of these features; similarly, when the number of certain features is not limited in the form of words such as "two" or "three", it should be noted that this feature also explicitly or implicitly includes one or more feature quantities.

[0047] The embodiments of the present invention are described in detail below with reference to the accompanying drawings.

[0048] Embodiment 1

[0049] like Figure 1-4 As shown, an embodiment of the present invention provides a responsive cache preheating method based on data and permission change notification, comprising the following steps:

[0050] S1. Build a query engine - use the query engine to read results from the redis cache first;

[0051] S2. Insert query records into the backend request to select preheat target data for use;

[0052] S3, Data set data update - if the update is successful, the updated event will be written into the data set event table;

[0053] S4, monitor data set and permission change events;

[0054] S5, build data set event processing logic;

[0055] S6. Regularly consume and preheat the task queue;

[0056] S7. Before submitting the task to the thread pool, update the task status to "in process" - refresh the cached data;

[0057] S8. After receiving the request, the query engine service checks the request type. If it is a cache refresh request, it executes the cache refresh logic and directly queries the data from the database. After the query is successful, it updates the value of the corresponding key in redis.

[0058] In one embodiment, in S1, when there is no result corresponding to the request in the redis cache, the database query task is automatically submitted through the query engine. After the query is successful, the engine automatically saves the result to the redis cache;

[0059] By introducing an event notification mechanism, we can quickly respond to user requests and instantly preheat the cache, which improves the timeliness of cache updates while ensuring data accuracy and system performance.

[0060] In one embodiment, in S2, when a user queries data in the system, the backend request initiated contains logic for counting the user's data query records. Every time a user queries a piece of data, the system automatically inserts a query record.

[0061] By collecting statistics on query records, we can analyze user behavior, optimize system performance and data caching strategies based on statistical data.

[0062] Among them, the query record is regenerated based on the user ID and data ID as the logical library primary key.

[0063] In one embodiment, in S3, the data set data is updated every 12 hours;

[0064] Set the data update time according to actual needs.

[0065] In one embodiment, in S4, a timer task is used to query unprocessed time records, and then the time processing logic is cyclically called to process events;

[0066] By cyclically calling the time processing logic to process events, task scheduling can be performed efficiently and reliably, and unprocessed time records can be automatically processed to ensure data accuracy and timely response of the system.

[0067] In one embodiment, in S5, when the data set event processing logic is executed, firstly, the data to be preheated is selected from the data query record according to the data set ID; then, the data cache preheating task information is first put into the priority task queue according to the priority strategy; then, the distributed lock is acquired, and after the acquisition is successful, all the data in the priority queue are found out according to the score in reverse order;

[0068] By utilizing dataset event processing logic to effectively manage dataset cache preheating tasks, you can ensure that datasets can be quickly and accurately loaded into the cache when needed, which helps improve system responsiveness and user experience.

[0069] The query range is data within 6 to 8 days, that is, date>=T-6, and the data set ID = the data set ID currently being processed;

[0070] By setting the query range, you can obtain data within the corresponding range based on the set data.

[0071] The priority strategy is: the more recent the query time, the higher the priority. The more recent the query time, the more active it is recently and needs to be refreshed first.

[0072] Since the closer the query time is, the more active the data set is, the cache needs to be refreshed first to ensure that subsequent requests can quickly obtain the latest data and ensure that users can obtain the latest data in a timely manner.

[0073] Among them, if the total number of cache warm-up task information total Count<3, all of them are processed by the fast processing queue. Otherwise, the data is divided into 3 parts, 1 part for the fast processing queue and 2 parts for the slow processing queue, that is: index=total Count / 3, [0, index] for the fast processing task queue and (index, total Count-1] for the slow processing task queue.

[0074] In one embodiment, in S6, failed tasks and tasks in a long-time processing state are added to the fast and slow task queues before consumption, and then elements are popped from the queues in a loop, and cache preheating logic and cache generation are executed in the loop. The execution of tasks is accelerated in parallel using a thread pool, and the task status is recorded and updated during the task execution process. The task is stopped when the queue is empty.

[0075] By re-adding failed and long-processing tasks to the fast and slow task queues, cache warm-up tasks can be effectively managed, the efficiency of task processing can be improved, and the stability of task execution and data accuracy can be ensured.

[0076] The number of threads in the thread pool of the fast queue is greater than that of the slow queue. The number of threads is configurable. The queue uses a synchronous blocking queue. When the thread pool is full, the thread that submits the task will be automatically suspended and will be called up again when the thread pool is idle.

[0077] By configuring the thread pool parameters of the fast queue and the slow queue, you can effectively manage the execution of cache warm-up tasks, improve the system's processing power and response speed, help balance the system's resource usage, and avoid performance bottlenecks caused by uneven resource allocation.

[0078] The task status includes initial state, processing, success and failure.

[0079] In one embodiment, in S7, the status is updated to "processing" before the task is executed. In the cache warm-up logic, the global filter parameters are constructed according to the default parameters in the analysis dashboard where the user data is located, and then the data query and cache refresh methods are called. The data query and cache refresh methods are responsible for constructing the downstream RPC interface parameters of the data query and judging the status of the task according to the parameters.

[0080] By constructing global filter parameters and downstream RPC interface parameters for data queries, it can be ensured that when users perform data analysis and queries, the data in the cache has been prepared according to the user's specific filter conditions, thereby improving query efficiency and user experience, reducing the time users wait for data preparation, and also reducing the query pressure on the backend data source.

[0081] The judgment method is as follows: if the RPC interface call is successful and the service request status code is 200, it means that the cache refresh is successful, and then the status of the cache preheating task is updated to success, otherwise the status of the cache preheating task is updated to failure;

[0082] By using the data of whether the interface call is successful and the business request status code to determine the status of the task, the correctness of the cache warm-up task is ensured, and a quick response can be made when problems arise.

[0083] In one embodiment, in S7, the cache warm-up logic captures the overall exception, and if the exception is captured, the task status is updated to failure, waiting for the next cycle to be processed;

[0084] During the cache preheating process, if an exception occurs, such as database connection failure, data read error, cache write failure, etc., these exceptions are captured. Once an exception is captured, the current preheating task status is updated to "failed"; this notifies the system that the task has not been successfully completed, and the failed preheating task will be retried in the next cycle. This means that the system will retry the cache preheating operation at a new time point instead of retrying immediately, avoiding repeated attempts immediately when an exception occurs, resulting in resource waste and increased system load.

[0085] In one embodiment, in S8, key is the md5 value of the SQL generated by data compilation, ensuring one SQL is cached;

[0086] A key is a unique identifier used to identify a specific entry in the cache. By using a unique key, each SQL query corresponds to a unique cache entry.

[0087] Embodiment 2

[0088] like Figure 1-4 As shown, an embodiment of the present invention also provides a responsive cache preheating method based on data and permission change notification, comprising the following steps:

[0089] S1. Build a query engine - use the query engine to read results from the redis cache first;

[0090] S2. Insert query records into the backend request to select preheat target data for use;

[0091] S3, Data set data update - if the update is successful, the updated event will be written into the data set event table;

[0092] S4, monitor data set and permission change events;

[0093] S5, build data set event processing logic;

[0094] S6. Regularly consume and preheat the task queue;

[0095] S7. Before submitting the task to the thread pool, update the task status to "in process" - refresh the cached data;

[0096] S8. After receiving the request, the query engine service checks the request type. If it is a cache refresh request, it executes the cache refresh logic and directly queries the data from the database. After the query is successful, it updates the value of the corresponding key in redis.

[0097] In one embodiment, in S1, when there is no result corresponding to the request in the redis cache, the database query task is automatically submitted through the query engine. After the query is successful, the engine automatically saves the result to the redis cache;

[0098] By introducing an event notification mechanism, we can quickly respond to user requests and instantly preheat the cache, which improves the timeliness of cache updates while ensuring data accuracy and system performance.

[0099] In one embodiment, in S2, when a user queries data in the system, the backend request initiated contains logic for counting the user's data query records. Every time a user queries a piece of data, the system automatically inserts a query record.

[0100] By collecting statistics on query records, we can analyze user behavior, optimize system performance and data caching strategies based on statistical data.

[0101] Among them, the query record is regenerated based on the user ID and data ID as the logical library primary key.

[0102] In one embodiment, in S3, the data set data is updated every 36 hours;

[0103] Set the data update time according to actual needs.

[0104] In one embodiment, in S4, a timer task is used to query unprocessed time records, and then the time processing logic is cyclically called to process events;

[0105] By cyclically calling the time processing logic to process events, task scheduling can be performed efficiently and reliably, and unprocessed time records can be automatically processed to ensure data accuracy and timely response of the system.

[0106] In one embodiment, in S5, when the data set event processing logic is executed, firstly, the data to be preheated is selected from the data query record according to the data set ID; then, the data cache preheating task information is first put into the priority task queue according to the priority strategy; then, the distributed lock is acquired, and after the acquisition is successful, all the data in the priority queue are found out according to the score in reverse order;

[0107] By utilizing dataset event processing logic to effectively manage dataset cache preheating tasks, you can ensure that datasets can be quickly and accurately loaded into the cache when needed, which helps improve system responsiveness and user experience.

[0108] The query range is data within 8 days, that is, date>=T-8, and the data set ID = the data set ID currently being processed;

[0109] By setting the query range, you can obtain data within the corresponding range based on the set data.

[0110] The priority strategy is: the more recent the query time, the higher the priority. The more recent the query time, the more active it is recently and needs to be refreshed first.

[0111] Since the closer the query time is, the more active the data set is, the cache needs to be refreshed first to ensure that subsequent requests can quickly obtain the latest data and ensure that users can obtain the latest data in a timely manner.

[0112] Among them, if the total number of cache warm-up task information total Count<5, all of them are processed by the fast processing queue. Otherwise, the data is divided into 5 parts, 1 part for the fast processing queue, and 4 parts for the slow processing queue, that is: index=total Count / 5, [0, index] for the fast processing task queue, and (index, total Count-1] for the slow processing task queue.

[0113] In one embodiment, in S6, failed tasks and tasks in a long-time processing state are added to the fast and slow task queues before consumption, and then elements are popped from the queues in a loop, and cache preheating logic and cache generation are executed in the loop. The execution of tasks is accelerated in parallel using a thread pool, and the task status is recorded and updated during the task execution process. The task is stopped when the queue is empty.

[0114] By re-adding failed and long-processing tasks to the fast and slow task queues, cache warm-up tasks can be effectively managed, the efficiency of task processing can be improved, and the stability of task execution and data accuracy can be ensured.

[0115] The number of threads in the thread pool of the fast queue is greater than that of the slow queue. The number of threads is configurable. The queue uses a synchronous blocking queue. When the thread pool is full, the thread that submits the task will be automatically suspended and will be called up again when the thread pool is idle.

[0116] By configuring the thread pool parameters of the fast queue and the slow queue, you can effectively manage the execution of cache warm-up tasks, improve the system's processing power and response speed, help balance the system's resource usage, and avoid performance bottlenecks caused by uneven resource allocation.

[0117] The task status includes initial state, processing, success and failure.

[0118] In one embodiment, in S7, the status is updated to "processing" before the task is executed. In the cache warm-up logic, the global filter parameters are constructed according to the default parameters in the analysis dashboard where the user data is located, and then the data query and cache refresh methods are called. The data query and cache refresh methods are responsible for constructing the downstream RPC interface parameters of the data query and judging the status of the task according to the parameters.

[0119] By constructing global filter parameters and downstream RPC interface parameters for data queries, it can be ensured that when users perform data analysis and queries, the data in the cache has been prepared according to the user's specific filter conditions, thereby improving query efficiency and user experience, reducing the time users wait for data preparation, and also reducing the query pressure on the backend data source.

[0120] The judgment method is as follows: if the RPC interface call is successful and the service request status code is 200, it means that the cache refresh is successful, and then the status of the cache preheating task is updated to success, otherwise the status of the cache preheating task is updated to failure;

[0121] By using the data of whether the interface call is successful and the business request status code to determine the status of the task, the correctness of the cache warm-up task is ensured, and a quick response can be made when problems arise.

[0122] In one embodiment, in S7, the cache preheating logic captures the overall exception, and if the exception is captured, the task status is updated to failure, waiting for the next cycle to be processed;

[0123] During the cache preheating process, if an exception occurs, such as database connection failure, data read error, cache write failure, etc., these exceptions are captured. Once an exception is captured, the current preheating task status is updated to "failed"; this notifies the system that the task has not been successfully completed, and the failed preheating task will be retried in the next cycle. This means that the system will retry the cache preheating operation at a new time point instead of retrying immediately, avoiding repeated attempts immediately when an exception occurs, resulting in resource waste and increased system load.

[0124] In one embodiment, in S8, key is the md5 value of the SQL generated by data compilation, ensuring one SQL is cached;

[0125] A key is a unique identifier used to identify a specific entry in the cache. By using a unique key, each SQL query corresponds to a unique cache entry.

[0126] Embodiment 3

[0127] For example, if a user queries a chart within a cache cycle, Figure 1-4 As shown, an embodiment of the present invention also provides a responsive cache preheating method based on data and permission change notification, comprising the following steps:

[0128] S1. Build a query engine - the query engine will first read the results from the redis cache. If there is no corresponding result in the cache, the engine will automatically submit the database query task. After the query is successful, the engine will automatically save the result to the redis cache.

[0129] S2. When a user queries a chart in the system, the backend request initiated contains the logic of counting the user's chart query records. Every time a user queries a chart, the system automatically inserts a query record (using the user ID and chart ID as logical primary key to deduplicate). The query record is used to select the preheat target chart.

[0130] S3, the data set data will be updated every 24 hours, and after the update is successful, the update event will be written to the data set event table;

[0131] S4, monitor data sets, permission update events - scheduled tasks to query unprocessed event records, and then cyclically call event processing logic to process events;

[0132] S5. Constructing data set event processing logic - When executing the data set event processing logic, the chart that needs to be preheated will be selected from the chart query record according to the data set ID (data with query time within 7 days, date>=T-7, data set ID=currently processed data set ID), and the chart cache preheating task information will be put into the priority task queue (redis sortSet data structure, score is the millisecond value of the chart's most recent query timestamp) according to the priority strategy (the closer the latest query time is, the higher the priority is. The closer the latest query time is, the more active it is recently and needs to be refreshed first). Then obtain the distributed lock. After successful acquisition, all data in the priority queue are found in reverse order according to score (query time). If the total number of cache preheating task information total Count<4, all will be processed by the fast processing queue; otherwise, the chart will be divided into four parts, one for the fast processing queue and three for the slow processing queue, that is: index=total Count / 4, [0, index] for the fast processing task queue, (index, total Count-1] for the slow processing task queue;

[0133] S6. Regularly consume and preheat the task queue - before consumption, failed tasks and tasks in the processing state for a long time will be added to the fast and slow task queues (before starting, a distributed lock will be obtained to solve concurrency problems, and tasks will be added to the corresponding queues according to their speed types), and then pop elements from the queue in a loop, and execute the cache preheating logic and generate cache in the loop. The thread pool is used to speed up the execution of tasks in parallel (the number of threads in the thread pool of the fast queue is greater than that of the slow queue, and the number of threads is configurable. The queue uses a synchronous blocking queue. When the thread pool is full, the thread that submits the task will be automatically suspended and called up when the thread pool is idle). During the task execution process, the task status is recorded and updated: initial state, processing, success, failure, and the task is stopped when the queue is empty;

[0134] S7. Before submitting the task to the thread pool, update the task status to "Processing" - Before submitting the task to the thread pool, update the task status to "Processing". In the cache preheating logic, build the global filter parameters according to the default parameters in the analysis dashboard where the user chart is located, and then call the chart query and cache refresh methods. The chart query and cache refresh methods are responsible for building the downstream rpc interface parameters of the chart query. If the rpc interface call is successful and the business request status code is 200, it means that the cache refresh is successful. Then update the cache preheating task status to success, otherwise update the cache preheating task status to failure. The cache preheating logic captures the overall exception. If an exception is captured, the task status is updated to failure and waits for the next cycle to be processed.

[0135] S8. After receiving the request, the query engine service checks the request type. If it is a cache refresh request, it executes the cache refresh logic and directly queries the data from the database. After the query is successful, it updates the value of the corresponding key in redis.

[0136] Explanation of relevant terms:

[0137] Dataset: A collection of data. This collection can be a data table or the result of a SQL query.

[0138] Diagram: The carrier of the analysis logic in the Bi system, describing the various information required by the analysis logic.

[0139] Cache: In order to speed up query, some data in the business system is stored in a faster medium. The data read from this medium is cache.

[0140] Cache hit: indicates that the requested data can be obtained from the cache without going back to the source.

[0141] The above are only specific embodiments of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art can easily think of various changes or substitutions within the technical scope disclosed by the present invention, which should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

Claims

1. A responsive cache preheating method based on data and permission change notification, characterized in that: The following steps are involved: S1. Build a query engine - use the query engine to read results from the redis cache first; S2. Insert query records into the backend request to select preheat target data for use; S3, Data set data update - if the update is successful, the updated event will be written into the data set event table; S4, monitor data set and permission change events; S5, build data set event processing logic; S6. Regularly consume and preheat the task queue; S7. Before submitting the task to the thread pool, update the task status to "in process" - refresh the cached data; S8. After receiving the request, the query engine service checks the request type. If it is a cache refresh request, it executes the cache refresh logic and directly queries the data from the database. After the query is successful, it updates the value of the corresponding key in redis.

2. The responsive cache preheating method based on data and permission change notification according to claim 1 is characterized in that: In S1, when there is no result corresponding to the request in the redis cache, the database query task is automatically submitted through the query engine. After the query is successful, the engine automatically saves the result to the redis cache.

3. The responsive cache preheating method based on data and permission change notification according to claim 1 is characterized in that: In S2, when a user queries data in the system, the backend request initiated contains logic for counting the user's data query records. Every time a user queries a piece of data, the system automatically inserts a query record. Among them, the query record is regenerated based on the user ID and data ID as the logical library primary key.

4. The responsive cache preheating method based on data and permission change notification according to claim 1 is characterized in that: In the S3, the data set data is updated every 12-36 hours.

5. The responsive cache preheating method based on data and permission change notification according to claim 1 is characterized in that: In S4, a scheduled task is used to query unprocessed time records, and then the time processing logic is cyclically called to process events.

6. The responsive cache preheating method based on data and permission change notification according to claim 4 is characterized in that: In S5, when the data set event processing logic is executed, firstly, the data to be preheated is selected from the data query record according to the data set ID; then, the data cache preheating task information is put into the priority task queue according to the priority strategy; then, the distributed lock is acquired, and after successful acquisition, all the data in the priority queue are found out in reverse order according to the score; The query range is data within 6 to 8 days, that is, date>=T-(6 to 8), and the data set ID = the data set ID currently being processed; The priority strategy is: the more recent the query time, the higher the priority. The more recent the query time, the more active it is recently and needs to be refreshed first. Among them, if the total number of cache warm-up task information totalCount<(3 to 5), all of them will be processed by the fast processing queue. Otherwise, the data will be divided into 3 to 5 parts, 1 part for the fast processing queue, and 2-4 parts for the slow processing queue, that is: index = totalCount / (3 to 5), [0, index] for the fast processing task queue, and (index, tatalCount-1] for the slow processing task queue.

7. The responsive cache preheating method based on data and permission change notification according to claim 6 is characterized in that: In S6, before consumption, failed tasks and tasks in a long-time processing state are added back to the fast and slow task queues, and then elements are popped from the queues in a loop, and cache preheating logic and cache generation are executed in the loop. Task execution is accelerated in parallel using a thread pool, and task status is recorded and updated during task execution. The task is stopped when the queue is empty. The number of threads in the thread pool of the fast queue is greater than that of the slow queue. The number of threads is configurable. The queue uses a synchronous blocking queue. When the thread pool is full, the thread that submits the task will be automatically suspended and will be called up again when the thread pool is idle. The task status includes initial state, processing, success and failure.

8. The responsive cache preheating method based on data and permission change notification according to claim 1, characterized in that: In S7, the update status before the task is executed is in process, and the cache preheating logic constructs the global filter parameters according to the default parameters in the analysis dashboard where the user data is located, and then calls the data query and cache refresh methods, which are responsible for constructing the downstream RPC interface parameters of the data query and judging the status of the task according to the parameters; The judgment method is: if the RPC interface call is successful and the service request status code is 200, it means that the cache refresh is successful, and then the status of the cache preheating task is updated to success, otherwise the status of the cache preheating task is updated to failure.

9. The responsive cache preheating method based on data and permission change notification according to claim 8, characterized in that: In S7, the cache preheating logic captures the overall exception, and if the exception is captured, the task status is updated to failure, waiting for the next cycle to be processed.

10. The responsive cache preheating method based on data and permission change notification according to claim 1, characterized in that: In S8, the key is the md5 value of the SQL generated by data compilation, ensuring one SQL is cached.

Citation Information

Patent Citations

  • Multistage queue delay task execution method and related device

    CN117850887A

  • Data query method and device based on cache preheating, equipment and medium

    CN118332018A