Cache warming system, method, computer device and storage medium

Abstract

This invention relates to the field of data processing technology and discloses a cache preheating system. The cache preheating system includes: an update module, used to generate a data update event based on user-updated data; the data update event includes the updated data; a data synchronization module, used to write the updated data into a first cache according to the key value of the query conditions when the data update event affects the query results, and broadcast the updated data to a query module through a first message queue; the data in the first cache is fully updated to the shared disk corresponding to the query module according to a first preset period; and a query module, used to complete and update the local memory data based on the data in the first message queue and the full data in the first cache on the shared disk during data recovery. Through the above method, this invention effectively improves data preheating efficiency.

Description

Technical Field

[0001] The embodiments of the present invention relate to the field of data processing technology, specifically to a cache preheating system, method, computer device, and computer-readable storage medium. Background Technology

[0002] Currently, for complex scenarios with high concurrency and low latency requirements, multi-level caching is a common technique for reducing the load on databases or downstream services. Based on multi-level caching, when a query is executed, the system first checks if there is data in the first-level cache; if so, it returns the data directly. If not, it checks the second-level cache; if the second-level cache also has no data, it queries the database or calls a query API. When an update operation is performed, the data in the cache must be deleted or updated.

[0003] However, in high-concurrency, high-performance scenarios, when services need to be redeployed, the first-level cache is empty, and a large number of requests will directly access the second-level cache and the underlying service. The inventors of this application have discovered that a large number of requests may overwhelm the underlying service, resulting in low efficiency of cache preheating during cold starts. Furthermore, due to update latency between various data storage modules, data inconsistencies may occur, leading to an avalanche effect or request timeouts within a short period of time, causing service unavailability and affecting service quality. Summary of the Invention

[0004] In view of the above problems, embodiments of the present invention provide a cache preheating system, method, computer device, and computer-readable storage medium to solve the problems of low cache preheating efficiency and data inconsistency during cold starts in the prior art.

[0005] According to one aspect of the present invention, a cache preheating system is provided, the cache preheating system comprising:

[0006] The update module is used to generate a data update event based on the data updated by the user; the data update event includes the updated data.

[0007] The data synchronization module is used to write the updated data into a first cache according to the key value of the query conditions when the data update event affects the query results, and broadcast the updated data to the query module through a first message queue; the data in the first cache is fully updated to the shared disk corresponding to the query module according to a first preset period.

[0008] The query module is used to complete and update the local memory data based on the data in the first message queue and the full data of the first cache in the shared disk during data recovery.

[0009] In one optional approach, the query module is used to complete and update the local memory data during data recovery based on the data in the first message queue and the full data of the first cache in the shared disk, specifically including:

[0010] Perform differential data analysis on the latest data in the first message queue and the latest full data in the first cache in the shared disk to obtain differential data analysis results;

[0011] Based on the difference data analysis results, the local memory data is updated by selecting either the full data from the first cache or the data from the first message queue, and the latest local memory data is asynchronously saved to the shared disk as the latest full backup data.

[0012] In an alternative embodiment, the update module is further configured to: parse the binary log of the database, generate a data update object, and send the data update object to the data update topic of a second message queue to generate a data update event when a user creates or updates data.

[0013] In an optional manner, the data synchronization module is further configured to: monitor the data update topic in the second message queue; and when a data update event occurs, determine whether the data update event affects the query results.

[0014] In an alternative embodiment, the data synchronization module is further configured to:

[0015] The updated data information is stored in a new set in the first cache at a second preset period; the updated data information includes the starting value of the queue offset in the second message queue.

[0016] In an alternative embodiment, the query module is further configured to:

[0017] Once the first service node of the query module has started, the data obtained after the first service node starts will be persisted to the shared disk as an incremental backup; and the latest memory data will be asynchronously saved to the shared disk as the latest full backup, so that it can be used by the other service nodes of the query module.

[0018] In an alternative embodiment, the query module is further configured to:

[0019] When the service node of the query module is shut down, determine whether the data in the second cache of the service node is complete;

[0020] In the complete case, the service node saves the second cached data to the shared disk and records the current progress information and current time information of the first message queue.

[0021] According to another aspect of the present invention, a cache preheating method is provided, comprising:

[0022] A data update event is generated based on the user-updated data; the data update event includes the updated data.

[0023] When the data update event affects the query results, the updated data is written into the first cache according to the key value of the query conditions, and the updated data is broadcast to the query module through the first message queue; the data in the first cache is fully updated to the shared disk corresponding to the query module according to the first preset period.

[0024] During data recovery, the local memory data is completed and updated based on the data in the first message queue and the full amount of data in the first cache in the shared disk.

[0025] According to another aspect of the present invention, a computer device is provided, including: a processor, a memory, a communication interface, and a communication bus, wherein the processor, the memory, and the communication interface communicate with each other through the communication bus;

[0026] The memory is used to store at least one executable instruction that causes the processor to perform the operation of the cache warm-up method.

[0027] According to another aspect of the present invention, a computer-readable storage medium is provided, the storage medium storing at least one executable instruction, which, when executed on a computer device, causes the computer device to perform the operation of the cache preheating method.

[0028] The cache preheating system of this invention includes an update module, a data synchronization module, and a query module. The update module generates a data update event based on user-updated data, the data update event including the updated data. The data synchronization module, when the data update event affects the query result, writes the updated data into a first cache according to the key value of the query conditions and broadcasts the updated data to the query module via a first message queue. The data in the first cache is fully updated to the shared disk corresponding to the query module according to a first preset period. The query module, during data recovery, completes and updates the local memory data based on the data in the first message queue and the full data in the first cache on the shared disk. In this way, data is stored in both the first message queue and the shared disk. When the data query module performs data recovery, it can combine the data in the first message queue and the shared disk to perform data recovery, effectively solving the problem of data inconsistency during cold start data recovery. Furthermore, by comparing the differences between the data in the first message queue and the shared disk, a more efficient data update method is determined, thereby improving data preheating efficiency.

[0029] The above description is merely an overview of the technical solutions of the embodiments of the present invention. In order to better understand the technical means of the embodiments of the present invention and to implement them in accordance with the contents of the specification, and to make the above and other objects, features and advantages of the embodiments of the present invention more apparent and understandable, specific embodiments of the present invention are described below. Attached Figure Description

[0030] The accompanying drawings are for illustrative purposes only and are not intended to limit the invention. Furthermore, the same reference numerals denote the same parts throughout the drawings. In the drawings:

[0031] Figure 1 A schematic diagram of the cache preheating system provided in an embodiment of the present invention is shown;

[0032] Figure 2 A schematic diagram of the structure of a cache preheating system provided in another embodiment of the present invention is shown;

[0033] Figure 3 This diagram illustrates the shutdown and startup process of the query module in the cache preheating system provided by an embodiment of the present invention.

[0034] Figure 4 A schematic flowchart of the cache preheating method provided in an embodiment of the present invention is shown;

[0035] Figure 5 A schematic diagram of the structure of a computer device provided in an embodiment of the present invention is shown. Detailed Implementation

[0036] Exemplary embodiments of the invention will now be described in more detail with reference to the accompanying drawings. Although exemplary embodiments of the invention are shown in the drawings, it should be understood that the invention may be implemented in various forms and should not be limited to the embodiments set forth herein.

[0037] Figure 1 A schematic diagram of the cache preheating system provided in an embodiment of the present invention is shown, as follows: Figure 1 As shown, the cache preheating system 100 includes an update module 110, a data synchronization module 120, and a query module 130. The query module 130 is responsible for query services; the update module 110 is responsible for update operations; and the data synchronization module 120 is responsible for refreshing the cached data in the query module 130 in a timely manner when the update operation causes the query results to change, so as to maintain data consistency.

[0038] The update module 110 is used to generate a data update event based on the data updated by the user; the data update event includes the updated data.

[0039] The data synchronization module 120 is used to write the updated data into a first cache according to the key value of the query conditions when the data update event affects the query results, and broadcast the updated data to the query module through a first message queue; the data in the first cache is fully updated to the shared disk corresponding to the query module according to a first preset period.

[0040] The query module 130 is used to complete and update the local memory data based on the data in the first message queue and the full amount of data in the first cache in the shared disk during data recovery.

[0041] In another embodiment of the present invention, such as Figure 2 As shown, the update module 110 communicates with the data synchronization module 120 via the database (DB), and the two communicate through a message queue. The data synchronization module 120 corresponds to a first cache (Redis), and communicates with the query module 130 via the first message queue. The query module 130 corresponds to a second cache (i.e., the memory data backup in the diagram). The update module 110 is used to parse the binary log of the database, generate a data update object, and send the data update object to the data update topic of the second message queue when a user creates or updates data, thereby generating a data update event. The data synchronization module 120 is also used to: listen to the data update topic in the second message queue; and when a data update event occurs, determine whether the data update event affects the query results. Specifically, in conjunction with... Figure 2When a user creates or updates data, a data update object is generated by parsing the database binary log using Canal, and then sent to the data update topic in the message queue, generating a data update event. The data synchronization module 120 listens to the data update topic in the message queue. When an update event occurs, it determines whether the update event affects the query results. If the update event affects the query results, it triggers the data synchronization update mechanism. Specifically, it determines whether the update affects the query module 130 by determining the information of the updated data in the data update event. The updated data information may include data type, data ID, etc. Based on the correspondence between the updated data information and the query conditions in the query module 130, it determines whether it affects the query module 130. When the query module 130 is affected, the data synchronization update mechanism is triggered. The data synchronization module 120 first obtains the data information of the data to be updated. On one hand, it writes the updated data into the first cache (Redis cache) for distributed storage according to the key-value pairs of the query conditions. On the other hand, it broadcasts the updated data to all service nodes of the query module 130 through the first message queue. After receiving the broadcast message, the query module 130 updates the local memory data in the second cache. In this embodiment of the invention, the data synchronization module 120 creates a new set in the first cache at a second preset period to store the information of the updated data, and creates a new set in Redis to store the information of data changes during this period, including the starting value of the offset in the corresponding second message queue, to facilitate data recovery. In addition, the data synchronization module 120 simultaneously persists all data in the first cache to a local file at a preset period and uploads it to the shared disk of the server corresponding to the deployment of the query module 130. The file also includes other information such as time for data recovery.

[0042] When the service node of the query module is shut down, the query module 130 is further configured to determine whether the local memory data in the second cache of the service node is complete. If complete, the service node saves the local memory data in the second cache to the shared disk and records the current progress information and current time information of the first message queue. In one embodiment of the present invention, the query module 130 can be in a cluster deployment mode. When the local memory data in the second cache is incomplete, the query module 130, when its service node is shut down, also determines whether other service nodes in the cluster have started backing up data from the first message queue to the second cache to update the local memory data. When no other service nodes are backing up, the time difference between the generation time of the local memory data in the second cache and the data updated in the first message queue is determined. If the time difference is less than a preset time difference threshold, no backup is performed. If the time difference is greater than or equal to the preset time difference threshold, the data updated in the first message queue is backed up to the local memory data in the second cache, and after the backup is completed, the local memory data in the second cache is saved to the shared disk, and the current progress information and current time information of the first message queue are recorded.

[0043] In embodiments of the present invention, such as Figure 3As shown, when data recovery is required, when the first service node of the query module 130 starts, it compares the timeliness between the latest local memory data saved by the query module 130 and the backup data uploaded to the shared disk by the data synchronization module 120. If the backup data uploaded to the shared disk by the data synchronization module 120 is newer, then the backup data uploaded to the shared disk by the data synchronization module 120 is used as the current original data. If the latest local memory data actively backed up by the service node of the query module 130 is newer, then the latest local memory data of the query module 130 is used as the current original data. The original data is divided into multiple sub-files to facilitate the query module 130's multi-threaded approach to accelerate data loading. The query module 130 obtains the differences between the current raw data, the latest data in the second message queue, and the latest data uploaded to the shared disk by the data synchronization module 120. It performs difference data analysis on the latest data in the first message queue and the latest full data in the first cache on the shared disk to obtain the difference data analysis results. Based on the difference data analysis results, it selects either the full data in the first cache or the data in the first message queue to complete and update the local memory data. Simultaneously, it asynchronously saves the latest local memory data as the latest full backup data to the shared disk. Specifically, the size of the data volume between the latest data in the first message queue and the latest full data in the first cache on the shared disk can be determined. Based on the data volume, it is determined whether to select the full data in the first cache or the data in the first message queue to complete and update the local memory data. In this embodiment, the smaller data volume can be selected for completion and update. This method can further accelerate data loading speed. In this module, once the first service node of the query module starts, the data acquired after its startup is persisted to a shared disk as an incremental backup. Simultaneously, the latest local memory data is asynchronously saved to the shared disk as a full backup, for use by other service nodes of the query module. When other service nodes of the query module 130 start, the data updated by the first service node is supplemented with an incremental completion component. This incremental completion is based on the current progress and time information of the first message queue, as well as the latest data in the first message queue. Adding the incremental completion component reduces network overhead when comparing the timeliness of backup data, further improving system warm-up performance. In high-concurrency, high-performance scenarios, performance may not meet requirements if a single user request involves multiple Redis cache or database operations, leading to business instability. Furthermore, cold starts during redeployment may cause a large number of requests to directly access Redis or the database, resulting in service unavailability. This embodiment of the invention improves service quality by preloading and minimizing the need for data synchronization to accelerate service recovery time.

[0044] The cache preheating system 100 of this embodiment of the invention includes an update module 110 that generates a data update event based on user-updated data; the data update event includes the updated data; a data synchronization module 120 that, when the data update event affects the query results, writes the updated data into a first cache according to the key value of the query conditions, and broadcasts the updated data to a query module 130 via a first message queue; the data in the first cache is fully updated to the shared disk corresponding to the query module according to a first preset period; and the query module 130 that, during data recovery, updates the local memory data based on the data in the first message queue or the full data in the first cache on the shared disk. This approach effectively solves the problem of cache preheating and data consistency during cold start data recovery, improving data preheating efficiency.

[0045] Figure 4 A flowchart illustrating the structure of the cache preheating method provided in an embodiment of the present invention is shown. This method is executed by the cache preheating system described in the above embodiment. Figure 4 As shown, the method includes the following steps:

[0046] Step 210: Generate a data update event based on the user-updated data; the data update event includes the updated data;

[0047] Step 220: When the data update event affects the query results, the updated data is written into the first cache according to the key value of the query conditions, and the updated data is broadcast to the query module through the first message queue; the data in the first cache is fully updated to the shared disk corresponding to the query module according to the first preset period;

[0048] Step 230: During data recovery, the local memory data is completed and updated based on the data in the first message queue and the full data of the first cache in the shared disk.

[0049] The specific implementation steps of the cache preheating method in this embodiment of the invention are largely the same as the specific working process of each module in the cache preheating system described above, and will not be repeated here.

[0050] In this embodiment of the invention, an update module generates a data update event based on user-updated data. The data update event includes the updated data. When the data update event affects the query results, the data synchronization module writes the updated data into a first cache according to the key value of the query conditions and broadcasts the updated data to the query module via a first message queue. The data in the first cache is fully updated to the shared disk corresponding to the query module according to a first preset period. During data recovery, the query module completes and updates the local memory data based on the data in the first message queue and the full data in the first cache on the shared disk. This method effectively solves the problem of cache warm-up and data consistency during cold start data recovery, improving data warm-up efficiency.

[0051] Figure 5 The diagram shows a structural schematic of a computer device provided in an embodiment of the present invention. The specific embodiments of the present invention do not limit the specific implementation of the computer device.

[0052] like Figure 5 As shown, the computer device may include: a processor 402, a communications interface 404, a memory 406, and a communications bus 408.

[0053] The processor 402, communication interface 404, and memory 406 communicate with each other via communication bus 408. Communication interface 404 is used to communicate with other network elements, such as clients or other servers. The processor 402 executes program 410, specifically performing the relevant steps described above in the embodiment of the cache preheating method.

[0054] Specifically, program 410 may include program code, which includes computer-executable instructions.

[0055] Processor 402 may be a central processing unit (CPU), an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement embodiments of the present invention. The computer device includes one or more processors, which may be processors of the same type, such as one or more CPUs; or processors of different types, such as one or more CPUs and one or more ASICs.

[0056] Memory 406 is used to store program 410. Memory 406 may include high-speed RAM memory, and may also include non-volatile memory, such as at least one disk storage device.

[0057] Specifically, program 410 can be called by processor 402 to cause the computer device to perform the following operations:

[0058] A data update event is generated based on the user-updated data; the data update event includes the updated data.

[0059] When the data update event affects the query results, the updated data is written into the first cache according to the key value of the query conditions, and the updated data is broadcast to the query module through the first message queue; the data in the first cache is fully updated to the shared disk corresponding to the query module according to the first preset period.

[0060] During data recovery, the local memory data is completed and updated based on the data in the first message queue and the full amount of data in the first cache in the shared disk.

[0061] The cache preheating method executed by the processor of the computer device in this embodiment of the invention is largely the same as the specific process steps of the above method embodiments, and will not be repeated here.

[0062] In this embodiment of the invention, an update module generates a data update event based on user-updated data. The data update event includes the updated data. When the data update event affects the query results, the data synchronization module writes the updated data into a first cache according to the key value of the query conditions and broadcasts the updated data to the query module via a first message queue. The data in the first cache is fully updated to the shared disk corresponding to the query module according to a first preset period. During data recovery, the query module completes and updates the local memory data based on the data in the first message queue and the full data in the first cache on the shared disk. This method effectively solves the problem of cache warm-up and data consistency during cold start data recovery, improving data warm-up efficiency.

[0063] This invention provides a computer-readable storage medium storing at least one executable instruction that, when executed on a computer device, causes the computer device to perform the cache preheating method in any of the above method embodiments.

[0064] Executable instructions can be used to cause computer devices to perform the following operations:

[0065] A data update event is generated based on the user-updated data; the data update event includes the updated data.

[0066] When the data update event affects the query results, the updated data is written into the first cache according to the key value of the query conditions, and the updated data is broadcast to the query module through the first message queue; the data in the first cache is fully updated to the shared disk corresponding to the query module according to the first preset period.

[0067] During data recovery, the local memory data is completed and updated based on the data in the first message queue and the full amount of data in the first cache in the shared disk.

[0068] When the computer-readable storage medium of this invention is run on a computer device, the cache preheating method executed is generally consistent with the specific process steps of the above method embodiments, and will not be repeated here.

[0069] In this embodiment of the invention, an update module generates a data update event based on user-updated data. The data update event includes the updated data. When the data update event affects the query results, the data synchronization module writes the updated data into a first cache according to the key value of the query conditions and broadcasts the updated data to the query module via a first message queue. The data in the first cache is fully updated to the shared disk corresponding to the query module according to a first preset period. During data recovery, the query module completes and updates the local memory data based on the data in the first message queue and the full data in the first cache on the shared disk. This method effectively solves the problem of cache warm-up and data consistency during cold start data recovery, improving data warm-up efficiency.

[0070] This invention provides a cache preheating device for performing the above-described cache preheating method.

[0071] This invention provides a computer program that can be called by a processor to cause a computer device to execute the cache preheating method in any of the above method embodiments.

[0072] This invention provides a computer program product, which includes a computer program stored on a computer-readable storage medium. The computer program includes program instructions that, when executed on a computer, cause the computer to perform the cache preheating method in any of the above method embodiments.

[0073] The algorithms or displays provided herein are not inherently related to any particular computer, virtual system, or other device. Various general-purpose systems can also be used in conjunction with the teachings herein. The required structure for constructing such systems is apparent from the above description. Furthermore, the embodiments of the present invention are not directed to any particular programming language. It should be understood that the content of the invention described herein can be implemented using various programming languages, and the above description of specific languages ​​is for the purpose of disclosing the best mode of implementation of the invention.

[0074] Numerous specific details are set forth in the specification provided herein. However, it will be understood that embodiments of the invention may be practiced without these specific details. In some instances, well-known methods, structures, and techniques have not been shown in detail so as not to obscure the understanding of this specification.

[0075] Similarly, it should be understood that, in order to streamline the invention and aid in understanding one or more of the various aspects of the invention, features of the embodiments of the invention are sometimes grouped together in a single embodiment, figure, or description thereof in the above description of exemplary embodiments of the invention. However, this disclosure should not be construed as reflecting an intention that the claimed invention requires more features than are expressly recited in each claim.

[0076] Those skilled in the art will understand that modules in the device of the embodiments can be adaptively changed and placed in one or more devices different from that embodiment. Modules, units, or components in the embodiments can be combined into a single module, unit, or component, and can be divided into multiple sub-modules, sub-units, or sub-components. Except where at least some of such features and / or processes or units are mutually exclusive, any combination can be used to combine all features disclosed in this specification (including the accompanying claims, abstract, and drawings) and all processes or units of any method or device so disclosed. Unless expressly stated otherwise, each feature disclosed in this specification (including the accompanying claims, abstract, and drawings) may be replaced by an alternative feature that serves the same, equivalent, or similar purpose.

[0077] It should be noted that the above embodiments are illustrative of the invention and not restrictive, and that those skilled in the art can devise alternative embodiments without departing from the scope of the appended claims. In the claims, any reference signs placed between parentheses should not be construed as limiting the claims. The word "comprising" does not exclude the presence of elements or steps not listed in the claims. The word "a" or "an" preceding an element does not exclude the presence of a plurality of such elements. The invention can be implemented by means of hardware comprising several different elements and by means of a suitably programmed computer. In the unit claims enumerating several means, several of these means may be embodied by the same item of hardware. The use of the words first, second, and third, etc., does not indicate any order. These words can be interpreted as names. The steps in the above embodiments, unless otherwise specified, should not be construed as limiting the order of execution.

Claims

1. A cache preheating system, characterized in that, The cache preheating system includes: The update module is used to generate a data update event based on the data updated by the user; the data update event includes the updated data. The data synchronization module is used to write the updated data into a first cache according to the key value of the query conditions when the data update event affects the query results, and broadcast the updated data to the query module through a first message queue; the data in the first cache is fully updated to the shared disk corresponding to the query module according to a first preset period. The query module is used to perform differential data analysis on the latest data in the first message queue and the latest full data in the first cache on the shared disk during data recovery. It determines the data volume between the latest data in the first message queue and the latest full data in the first cache on the shared disk, and selects to complete and update the local memory data based on the data volume, either through the full data in the first cache or the data in the first message queue.

2. The system according to claim 1, characterized in that, The update module is further used for: When a user creates or updates data, the binary log of the database is parsed to generate a data update object, and the data update object is sent to the data update topic in the second message queue to generate a data update event.

3. The system according to claim 2, characterized in that, The data synchronization module is also used for: Listen for the data update topic in the second message queue; When a data update event occurs, determine whether the data update event affects the query results.

4. The system according to claim 2, characterized in that, The data synchronization module is also used for: The updated data information is stored in a new set in the first cache at a second preset period; the updated data information includes the starting value of the offset in the second message queue.

5. The system according to claim 4, characterized in that, The query module is also used for: Once the first service node of the query module has started, the data obtained after the first service node starts will be persisted to the shared disk as an incremental backup. And asynchronously save the latest local memory data to a shared disk as the latest full backup, so that it can be used by other service nodes of the query module.

6. The system according to claim 4, characterized in that, The query module is also used for: When the service node of the query module is shut down, it is determined whether the local memory data in the second cache of the service node is complete; In the complete case, the local memory data in the second cache is saved to the shared disk, and the current progress information and current time information of the first message queue are recorded.

7. A cache preheating method, characterized in that, The method includes: A data update event is generated based on the user-updated data; the data update event includes the updated data. When the data update event affects the query results, the updated data is written into the first cache according to the key value of the query conditions, and the updated data is broadcast to the query module through the first message queue; the data in the first cache is fully updated to the shared disk corresponding to the query module according to the first preset period. During data recovery, a difference analysis is performed on the latest data in the first message queue and the latest full data in the first cache on the shared disk to determine the data volume between the latest data in the first message queue and the latest full data in the first cache on the shared disk. Based on the data volume, the local memory data is updated by selecting either the full data in the first cache or the data in the first message queue.

8. A computer device, characterized in that, include: The processor, memory, communication interface, and communication bus are provided, wherein the processor, memory, and communication interface communicate with each other via the communication bus. The memory is used to store at least one executable instruction that causes the processor to perform the operation of the cache warm-up method as described in claim 7.

9. A computer-readable storage medium, characterized in that, The storage medium stores at least one executable instruction, which, when executed on a computer device, causes the computer device to perform the cache preheating method as described in claim 7.

Images