A cache and log-based local queue data persistence method

Abstract

The application relates to a cache and log-based local queue data persistence method. The method comprises the following steps: when the system is started, three local queues are initialized and registered in a queue manager; service data is continuously pushed to a main queue and a copy queue, a main queue consumption thread pulls data, executes data processing services, and pushes the processed data to a to-be-deleted queue; copy data is written into a local log file, a copy queue consumption thread pulls data, and the copy data is cyclically backed up to a Redis cluster, and the corresponding data in the log file is deleted after backup; the to-be-deleted queue receives data output by the main queue data processing service, a consumption thread pulls data and compares the data with the data backed up in the Redis cluster, and the same data is deleted; when the system is restarted, the inventory data in the Redis cluster is inquired and loaded into the main queue for re-consumption; and the queue manager receives registration information of the queue and monitors the capacity of the local queue.

Description

Technical Field

[0001] This invention belongs to the technical field of data persistence methods, and particularly relates to a local queue data persistence method and data persistence system based on caching and logs. Background Technology

[0002] In distributed systems, local queues are a commonly used message passing mechanism for implementing asynchronous tasks and message processing. Local queues play a crucial role in computing systems and applications. However, with the increasing demands for data processing, traditional local queue systems face challenges when handling large amounts of data, exhibiting the following shortcomings and deficiencies:

[0003] First, when the system fails or loses power, unprocessed data in the local queue may be lost, resulting in data loss that is unrecoverable.

[0004] Secondly, traditional local queues rely on disk writes for data persistence, which results in low performance and susceptibility to disk failures. Moreover, existing methods for persistent storage and data recovery are often inefficient, increasing system overhead and latency.

[0005] Several solutions have emerged to address the above issues, as exemplified below:

[0006] 1. Database persistence: Data is stored in relational or non-relational databases, and data persistence and consistency are ensured through transactions and log recording. However, this approach is difficult to guarantee data consistency between the cache and the database, has high concurrency performance bottlenecks, and increases system complexity.

[0007] 2. Log recording: During system operation, key operations and data changes are recorded in log files. By writing data changes to persistent logs, even if the system fails, the information in the log files can still be used to restore and recover the data. However, this solution requires additional storage space to save the logs, and the recovery process is relatively time-consuming.

[0008] 3. Snapshot technology: Create a snapshot or mirror of the data so that the data can be restored to a specific point in time after a system failure. However, this solution also has the problems of excessive storage space consumption and long recovery time.

[0009] In summary, existing solutions all suffer from several insurmountable problems, exhibiting poor concurrency efficiency, ease of use, and versatility. Therefore, it is essential to design a simple and efficient data persistence method to address these issues. Summary of the Invention

[0010] To address the aforementioned problems with existing data persistence methods, this invention proposes a novel local queue data persistence method based on caching and logs.

[0011] Terminology Explanation

[0012] The technical terms used in the description of the method of this invention are explained as follows:

[0013] 1. Distributed Systems: A distributed system is a system composed of multiple independent computers or nodes that are connected through a network and work together to complete a common task. In a distributed system, nodes can communicate and coordinate through message passing, remote calls, or shared storage. These nodes share resources and workloads, and improve system performance, availability, and scalability through parallel processing and distributed computing.

[0014] 2. Queue: A queue is a common data structure used to manage a collection of elements according to the First-In-First-Out (FIFO) principle. It is similar to a queue in real life, where newly added elements are placed at the end of the queue, and elements are always removed from the queue starting from the head.

[0015] 3. Snapshot Technology: A snapshot is a method of capturing data or system state. In the computing field, snapshot technology is used to create a mirror image of data or a backup of a dataset at a specific point in time, so that the data can be quickly restored to a previous state when needed.

[0016] 4. In the following text, the local queue storing master data will be referred to as the master queue, and the data stored in the master queue will be referred to as master data (master queue data); the local queue storing replica data will be referred to as the replica queue, and the data stored in the replica queue will be referred to as replica data (replica queue data, master data replica); the local queue storing data to be deleted will be referred to as the queue to be deleted.

[0017] Specifically, this invention provides a local queue data persistence method based on caching and logging, the method comprising:

[0018] S1. When the system starts, initialize the log file path, and initialize three local queues and register them into the queue manager. The three local queues are the main queue, the replica queue, and the queue to be deleted.

[0019] S2. After the system starts, the generated business data will be continuously pushed to the main queue and the replica queue. The consumption thread of the main queue will continuously pull data from the main queue, then execute the data processing business of the main queue in a loop, and push the processed data to the queue to be deleted.

[0020] S3. Write the received replica data to the local log file. The consumer thread of the replica queue continuously pulls data from the replica queue and then backs up the replica data to the Redis cluster in a loop. After the backup is completed, delete the corresponding data in the log file.

[0021] S4. After receiving data from the main queue's data processing business output, the consumer thread continuously pulls data from the queue to be deleted and compares the pulled data with the data backed up in the Redis cluster, deleting the same data backed up in the Redis cluster.

[0022] S5. When the system restarts, after the main queue is initialized, it queries the existing data cached in the Redis cluster and loads it into the main queue for re-consumption;

[0023] S6. The queue manager receives the registration information of the queues and monitors the capacity of the registered local queues in real time. It controls the opening and closing of the expansion thread based on the queue capacity level.

[0024] Furthermore, in step S1 of the local queue data persistence method based on caching and logs of the present invention, among the three local queues:

[0025] The main queue is used to store the main data of the business;

[0026] The replica queue is used to store replicas of the master data;

[0027] The queue to be deleted is used to store data that has been processed by the main queue.

[0028] Furthermore, in the local queue data persistence method based on caching and logs of the present invention, the replica queue and the queue to be deleted serve the backup and recovery of the main queue data; the log file serves the backup and recovery of the replica queue data.

[0029] Furthermore, step S3 of the local queue data persistence method based on caching and logs of the present invention also includes: scanning the local log file when the system restarts, and reloading the data left in the local log file into the replica queue.

[0030] Furthermore, the existing data cached in the Redis cluster in step S5 of the local queue data persistence method based on caching and logs of the present invention is the data in the main queue that has not yet been consumed.

[0031] Furthermore, the queue registration information mentioned in step S6 of the local queue data persistence method based on caching and logs of the present invention includes the queue name, queue capacity, capacity alarm high watermark threshold, capacity alarm low watermark threshold, queue class, and expansion method.

[0032] The method of controlling the opening and closing of expansion threads based on queue capacity includes:

[0033] When the queue capacity reaches the high watermark threshold for capacity alarm, an expansion thread is started. The expansion method is called by reflection through the class of the queue and the expansion method to consume queue data and increase the available capacity of the queue.

[0034] When the queue capacity falls below the low watermark threshold for capacity alarm, the expansion thread is automatically interrupted.

[0035] Furthermore, the local queue data persistence method based on caching and logs of the present invention also includes: when an anomaly occurs during data processing, pushing the abnormal data to the fault tolerance module, and completing the fault tolerance of the abnormal data through a pre-set anomaly retry mechanism and anomaly compensation scheme.

[0036] In addition, the present invention provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the above-described method for local queue data persistence based on caching and logging.

[0037] On the other hand, the present invention also provides a local queue data persistence system based on caching and logging, the system comprising:

[0038] Data generation module: Used to generate and push master data and replica data, as well as consume data from the master queue;

[0039] The main data processing module is used to control and implement the consumption process of the main queue data, including pulling the main queue data, processing the main queue data in a loop, and pushing the processed data.

[0040] The replica data processing module is used to implement replica data processing business, write and delete log files, and insert replica data into the Redis cluster in batches.

[0041] The main data processing module to be deleted is used to receive the processed data information pushed by the main data processing module and delete the backups of these data in the Redis cluster.

[0042] Queue Registration and Management Module: Used for registering and managing local queues, enabling queue capacity monitoring and overload expansion;

[0043] Fault tolerance module: used to implement retries and exception compensation for data processing anomalies;

[0044] Redis Cluster: Used to store replicas of the master data. When the system restarts, the data is loaded into the master queue to prevent the master queue data from being lost due to system crashes.

[0045] Furthermore, the replica data processing module in the local queue data persistence system based on caching and logs of the present invention includes a replica data processing business submodule and a log file submodule.

[0046] In summary, the local queue data persistence method based on caching and logs of this invention has the following advantages:

[0047] 1. Data persistence: This method ensures that data in the local queue will not be lost in the event of system failure or power outage by writing data to a persistent cache.

[0048] 2. Efficient recovery: This method uses an interactive mechanism of local queues, caches, and log files to quickly and efficiently recover data that has not been stored, reducing system recovery time and overhead.

[0049] 3. System reliability: Improved data persistence increases system reliability and data consistency, reducing the risk of data loss.

[0050] 4. This method uses a local queue to store data, which reduces the I / O overhead of writing and querying compared to databases, snapshots and other technologies, thus improving the performance of concurrent data processing. Attached Figure Description

[0051] To more clearly illustrate the technical solution of the present invention, the accompanying drawings used in the description of the present invention will be briefly introduced below. Obviously, those skilled in the art can obtain other drawings based on these drawings without creative effort.

[0052] Figure 1 This is a schematic diagram of the data persistence system of the present invention.

[0053] Figure 2 This is a diagram illustrating the overall implementation framework of the data persistence method of the present invention.

[0054] Figure 3 This is a flowchart illustrating the implementation of the data persistence method of the present invention. Detailed Implementation

[0055] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below in conjunction with specific embodiments. Obviously, the described embodiments are only a part of the embodiments of this invention, and not all of them. This invention can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of this invention.

[0056] At the same time, it should be understood that the scope of protection of the present invention is not limited to the specific embodiments described below; it should also be understood that the terminology used in the embodiments of the present invention is for describing specific embodiments and not for limiting the scope of protection of the present invention.

[0057] Example: A method and system for local queue data persistence based on caching and logging.

[0058] The technical solution of the present invention will be described in detail below:

[0059] Firstly, the architecture of the data persistence system of the present invention is as follows: Figure 1 As shown, the system includes a data generation module, a master data processing module, a replica data processing module (comprising two sub-modules: replica data processing business and log files), a master data processing module to be deleted, a queue registration and management module, a fault tolerance module, and a Redis cluster. The specific functions of these modules are as follows:

[0060] 1. Data Generation Module: This module contains the specific implementation of the logic for generating and pushing master and replica data for the business processes involved in the system, as well as the consumption logic for the main queue. This module is primarily responsible for generating business master and replica data, and consuming data from the main queue.

[0061] 2. Master Data Processing Module: Contains the consumption flow control logic for the master queue data involved in the system's business. First, it cyclically pulls data from the master queue; then, it cyclically processes the pulled master data and hands it over to the consumption business for abstraction (the specific implementation is completed by the data generation module); finally, it pushes the processed data to the queue to be deleted. If an exception occurs during the process, the data is pushed to the fault tolerance module.

[0062] 3. Replica Data Processing Module: This module comprises two sub-modules: master data replica processing and log files. The master data replica processing workflow is as follows: First, it receives replicas of the master data pushed from the data generation module. Simultaneously (asynchronously for performance improvement, but with the risk of data loss), it writes the replica data to the local queue and the log file. The replica queue's consumption logic batches the master data replicas into the Redis cluster. After successful writing, the data in the local log file is deleted. The log file content is reloaded into the replica queue after a system restart to prevent the loss of unconsumed master data replicas due to system failure.

[0063] 4. Master data processing module to be deleted: This module is mainly responsible for receiving the processed data information pushed by the master data processing module and then deleting the backups of these data in the Redis cluster.

[0064] 5. Queue Registration and Management Module: Includes queue capacity monitoring and overload expansion logic, and is responsible for the registration and management of all local queues.

[0065] 6. Redis Cluster: Used to store replicas of the master data. Data is loaded into the master queue upon system restart to prevent data loss due to system crashes.

[0066] 7. Fault Tolerance Module: Includes retry logic and exception compensation logic for data processing anomalies.

[0067] In addition, the reason why this method and system use a local queue instead of an external message component is that the local queue processes data faster than the external message component (at least one less data storage and data query performance overhead).

[0068] Secondly, the specific implementation process of the local queue data persistence method based on caching and logs in this invention is as follows: Figure 2 and Figure 3 As shown, it includes the following steps:

[0069] Step 1. During system startup, in addition to initializing the log file path, three local queues are initialized and registered with the queue manager. The primary queue stores the main business data, the replica queue stores copies of the primary data, and the queue to be deleted stores data processed by the primary queue. The replica queue and the queue to be deleted serve the backup and recovery of the primary queue data; the log file serves the backup and recovery of the replica queue data. This three-queue collaborative scheme effectively mitigates the risk of local queue data loss without sacrificing the concurrent performance of the primary queue data processing.

[0070] Step 2. After the system starts, the business processes involved will generate some business data, which will be pushed to the main queue and the replica queue. The data in the replica queue is a deep copy of the data in the main queue. As long as the business system remains operational, it will continuously push data to these two queues. The consumption threads in the main queue continuously pull data from it, then loop through and execute the processing business logic for the main queue data, and finally push the processed data to the deletion queue.

[0071] Step 3. When the replica data processing module receives the pushed replica data, it needs to write a copy of the data to a local log file. The consumption thread of the replica queue continuously pulls data from the replica queue and then backs up the primary data replica to the Redis cluster in a loop. After the backup is complete, the corresponding data in the log file is deleted. When the system restarts, it scans the local log file and reloads any data left in the local log file into the replica queue. The backup, deletion, and restoration operations of the log file ensure that the replica queue data is not lost. Parameters can be set to control whether the log file is enabled or disabled. If disabled, there is a very small probability that the replica queue will lose data (if the data consumption rate of the replica queue is less than the data processing rate of the primary queue, resulting in system crash).

[0072] Step 4. The queue to be deleted receives data from the main queue's data processing business output; then, the consumer thread continuously pulls data from the queue to be deleted; then, the pulled data is compared with the data backed up in the Redis cluster, and the data backed up in the Redis cluster is deleted.

[0073] Step 5. The Redis cluster caches backups of the main queue data. This data is continuously added by the replica queue consumption logic and deleted by the deletion queue consumption logic. The existing data represents the data in the main queue that has not yet been consumed. When the system restarts, after the main queue initialization is complete, it queries the existing data cached in the Redis cluster and loads it into the main queue for re-consumption, thus preventing data loss in the main queue.

[0074] Step 6. The queue manager receives the queue's registration information. This information includes the queue's name, capacity, high-water mark threshold for capacity alerts, low-water mark threshold for capacity alerts, the queue's class, and the method for resizing. The queue manager monitors the capacity of registered local queues in real time. When the queue capacity reaches the high-water mark threshold, it initiates a resizing thread. This thread uses reflection to call the resizing method, consuming queue data and increasing the available capacity, based on parameters such as the queue's class and the resizing method. When the queue capacity falls below the low-water mark threshold, the resizing thread automatically terminates.

[0075] Step 7. Any data processing involved in the above process may encounter anomalies. When an anomaly occurs, the abnormal data will be pushed to the fault tolerance module, which will use a pre-set anomaly retry mechanism and anomaly compensation scheme to achieve fault tolerance for the corresponding anomaly.

[0076] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention in any way. Any person skilled in the art can make some modifications or alterations to the above-disclosed technical content to obtain equivalent embodiments without departing from the scope of the technical solution of the present invention. Any simple modifications, equivalent changes and alterations made to the above embodiments based on the technical essence of the present invention without departing from the scope of the technical solution of the present invention should be included within the scope of protection of the claims of the present invention.

Claims

1. A local queue data persistence method based on caching and logging, characterized in that, The method includes: S1. Upon system startup, the log file path is initialized, and three local queues are initialized and registered with the queue manager. These three local queues are the primary queue, the replica queue, and the queue to be deleted. The main queue is used to store the main data of the business; The replica queue is used to store replicas of the master data; The queue to be deleted is used to store data that has been processed by the main queue; S2. After the system starts, the generated business data will be continuously pushed to the main queue and the replica queue. The consumption thread of the main queue will continuously pull data from the main queue, then execute the data processing business of the main queue in a loop, and push the processed data to the queue to be deleted. S3. The received replica data is written to the local log file. The consumer thread of the replica queue continuously pulls data from the replica queue and then backs up the replica data to the Redis cluster in a loop. After the backup is completed, the corresponding data in the log file is deleted. When the system restarts, the local log file is scanned and the data left in the local log file is reloaded into the replica queue. S4. After the queue to be deleted receives data from the main queue's data processing business output, the consumer thread continuously pulls data from the queue to be deleted and compares the pulled data with the data backed up in the Redis cluster, deleting the same data backed up in the Redis cluster; S5. When the system restarts, after the main queue is initialized, the existing data cached in the Redis cluster is queried and loaded into the main queue for re-consumption; the existing data cached in the Redis cluster refers to the data in the main queue that has not yet been consumed. S6. The queue manager receives queue registration information, including the queue name, queue capacity, high-water mark threshold for capacity alarms, low-water mark threshold for capacity alarms, the class of the queue, and the expansion method; and monitors the capacity of registered local queues in real time, controlling the starting and stopping of expansion threads based on queue capacity levels, including: When the queue capacity reaches the high watermark threshold for capacity alarm, an expansion thread is started. The expansion method is called by reflection through the class of the queue and the expansion method to consume queue data and increase the available capacity of the queue. When the queue capacity falls below the low watermark threshold for capacity alarm, the expansion thread is automatically interrupted.

2. The local queue data persistence method based on caching and logging according to claim 1, characterized in that, The replica queue and the queue to be deleted serve the backup and recovery of the main queue data; the log file serves the backup and recovery of the replica queue data.

3. The local queue data persistence method based on caching and logging according to claim 1, characterized in that, The local queue data persistence method further includes: when an anomaly occurs during data processing, pushing the abnormal data to the fault tolerance module, and completing the fault tolerance of the abnormal data through a pre-set anomaly retry mechanism and anomaly compensation scheme.

4. A computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the local queue data persistence method based on caching and logging as described in any one of claims 1-3.

5. A local queue data persistence system based on caching and logging, characterized in that, The system implements the steps of the local queue data persistence method based on caching and logging as described in any one of claims 1-3 during runtime, and the system includes: Data generation module: Used to generate and push master data and replica data, as well as consume data from the master queue; The main data processing module is used to control and implement the consumption process of the main queue data, including pulling the main queue data, processing the main queue data in a loop, and pushing the processed data. The replica data processing module is used to implement replica data processing business, write and delete log files, and insert replica data into the Redis cluster in batches. The main data processing module to be deleted is used to receive the processed data information pushed by the main data processing module and delete the backups of these data in the Redis cluster. Queue Registration and Management Module: Used for registering and managing local queues, enabling queue capacity monitoring and overload expansion; Fault tolerance module: used to implement retries and exception compensation for data processing anomalies; Redis Cluster: Used to store replicas of the master data. When the system restarts, the data is loaded into the master queue to prevent the master queue data from being lost due to system crashes.

6. The local queue data persistence system based on caching and logging according to claim 5, characterized in that, The replica data processing module includes a replica data processing business submodule and a log file submodule.

Images