Consistency updating method and related equipment

Abstract

The embodiment of the invention provides a consistency updating method and related equipment, and belongs to the technical field of software development. The method comprises the following steps: updating business data through a database transaction and writing the business data into a cache operation log; scanning a current log of the cache operation log by adopting a timed task; converting the current logs into queue messages in batches, and sending the queue messages to a message queue; monitoring the message queue through the consumer cluster to obtain a queue message; analyzing the queue message through a consumer cluster to obtain an operation type, a cache key and a cache value; and calling a corresponding cache client interface according to the operation type through the consumer cluster, and executing cache operation to redis cache through the cache client interface according to the operation type, the cache key and the cache value. According to the embodiment of the invention, decoupling of business operation and cache updating can be realized, and the consistency and reliability of data are improved.

Description

Technical Field

[0001] This application relates to the field of software development technology, and in particular to a consistent update method and related equipment. Background Technology

[0002] In related technologies, traditional distributed systems perform cache update operations immediately after database transaction commit. This approach leads to extended transaction time, directly impacting core business operations when the cache service becomes unavailable. It suffers from high system coupling, poor fault tolerance, and lacks an effective asynchronous compensation mechanism, failing to meet the need for eventual consistency in high-concurrency scenarios.

[0003] In summary, the technical problems existing in the relevant technologies need to be improved. Summary of the Invention

[0004] The main objective of this application is to propose a consistent update method and related equipment, which aims to decouple business operations from cache updates.

[0005] To achieve the above objectives, one aspect of this application proposes a consistency update method, the method comprising: Update business data through database transactions and write the business data to the cache operation log; A scheduled task is used to scan the current log of the cached operation log, where the current log is in an unprocessed state. The current logs are converted into queue messages in batches, and the queue messages are sent to the message queue. The message queue is monitored by the consumer cluster to obtain the queue messages; The consumer cluster parses the queue messages to obtain the operation type, cache key, and cache value; The consumer cluster calls the corresponding cache client interface according to the operation type. The cache client interface performs a cache operation to the Redis cache according to the operation type, the cache key, and the cache value. The parameters of the cache client interface include the cache key and the cache value. The cache operation is an idempotent operation.

[0006] In some embodiments, updating business data through a database transaction and writing the business data to a cache operation log includes: A write request is initiated by the client, and a database transaction is started by the business service. The business data is updated based on the write request through a database transaction.

[0007] In some embodiments, the method further includes: If the cache client interface successfully executes the cache operation, the processing status of the current log corresponding to the queue message in the cache operation log is modified to "processed". Delete the queue message indicating successful execution from the message queue; If the cache client interface fails to perform the cache operation, the failure record in the current log corresponding to the queue message is incremented by one; If the number of failed records in the current log exceeds the retry threshold, the current log will be transferred to the dead letter queue.

[0008] In some embodiments, the method further includes: The verification task verifies whether the cache operation log is synchronized with the data cached in Redis. If the cache operation log is not synchronized with the data cached in Redis, then return to the step of scanning the current log of the cache operation log using a scheduled task.

[0009] In some embodiments, the step of batch converting the current logs into queue messages includes: Change the current log processing status to "processing"; The contents of the current log are encapsulated into a queue message.

[0010] In some embodiments, the idempotent operation includes write and delete operations. The step of calling the corresponding cache client interface through the consumer cluster according to the operation type, and executing the cache operation to the Redis cache through the cache client interface, includes: If the operation type is a write type, then the write interface is called through the consumer cluster, and the write operation is executed to the Redis cache through the cache client interface. The parameters of the write interface include the cache key and the cache value. If the operation type is deletion, the deletion interface is called through the consumer cluster, and the deletion operation is executed to the Redis cache through the cache client interface. The parameters of the deletion interface include the cache key.

[0011] To achieve the above objectives, another aspect of this application provides a consistency update apparatus, the apparatus comprising: The transaction update module is used to update business data through database transactions and write the business data into the cache operation log; The scheduled scanning module is used to scan the current log of the cached operation log using a scheduled task, wherein the processing status of the current log is unprocessed. The message conversion module is used to convert the current logs into queue messages in batches and send the queue messages to the message queue. The queue listening module is used to listen to the message queue through the consumer cluster and obtain the queue messages; The parsing module is used to parse the queue messages through the consumer cluster to obtain the operation type, cache key, and cache value; The caching module is used to call the corresponding caching client interface through the consumer cluster according to the operation type, and to perform caching operations to the Redis cache through the caching client interface according to the operation type, the cache key and the cache value. The parameters of the caching client interface include the cache key and the cache value, and the caching operation is an idempotent operation.

[0012] To achieve the above objectives, another aspect of this application provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the method described above.

[0013] To achieve the above objectives, another aspect of the embodiments of this application proposes a computer-readable storage medium storing a computer program that, when executed by a processor, implements the methods described above.

[0014] To achieve the above objectives, another aspect of the embodiments of this application proposes a computer program product, including a computer program that, when executed by a processor, implements the aforementioned method.

[0015] The embodiments of this application include at least the following beneficial effects: This application provides a consistent update method, apparatus, electronic device, storage medium, and program product. This solution updates business data through database transactions and writes the business data to the cache operation log, which helps improve data consistency and makes the modification of business data atomic. A scheduled task scans the current log of the cache operation log, and the processing status of the current log is unprocessed. The current log is batch-converted into queue messages and sent to the message queue. Even under abnormal conditions such as network jitter or node failure, each cache operation instruction can be executed at least once, and the queue message is eventually successfully obtained by listening to the message queue through the consumer cluster. The queue message is parsed by the consumer cluster to obtain the operation type, cache key, and cache value. The consumer cluster calls the corresponding cache client interface based on the operation type. The cache client interface then executes the cache operation to the Redis cache based on the operation type, cache key, and cache value. This makes the cache update operation asynchronous, completely decoupling the business service from the cache service. Failures on one side will not directly affect the other's core process, improving data consistency and reliability. Through the combination of "message persistence + retries + status marking", a highly reliable, self-healing eventual consistency closed loop is built. As long as the database transaction succeeds, the cache will be updated correctly. Attached Figure Description

[0016] Figure 1 This is a flowchart of the consistency update method provided in the embodiments of this application; Figure 2 yes Figure 1 The flowchart of step S101 in the text; Figure 3 This is a flowchart of the steps for determining the operation result of the consistency update method provided in the embodiments of this application; Figure 4 This is a flowchart of the consistency verification steps of the consistency update method provided in this application embodiment; Figure 5 yes Figure 1 The flowchart of step S106 in the process; Figure 6 This is a flowchart illustrating a specific implementation of the consistency update method provided in this application when applied to a message queue-based system that guarantees eventual consistency between the cache and the database. Figure 7 This is a system architecture diagram of a message queue-based system for ensuring eventual consistency between cache and database, provided in an embodiment of this application. Figure 8 This is a business process diagram of a message queue-based system for ensuring eventual consistency between cache and database, provided in an embodiment of this application. Figure 9This is a schematic diagram of the consistency update device provided in the embodiments of this application; Figure 10 This is a schematic diagram of the hardware structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0017] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of this application and are not intended to limit it. In the following description, when referring to the accompanying drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with those of this application; they are merely examples of apparatuses and methods consistent with some aspects of the embodiments of this application as detailed in the appended claims.

[0018] It is understood that the terms “first,” “second,” etc., used in this application may be used herein to describe various concepts, but unless otherwise stated, these concepts are not limited by these terms. These terms are only used to distinguish one concept from another. For example, without departing from the scope of the embodiments of this application, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Depending on the context, the words “if,” “when,” or “in response to a determination” as used herein may be interpreted as “when…” or “when…” or “in response to a determination.”

[0019] As used in this application, the terms "at least one", "multiple", "each", "any", etc., "at least one" includes one, two or more, "multiple" includes two or more, "each" refers to each of the corresponding multiples, and "any" refers to any one of the multiples.

[0020] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit the scope of this application.

[0021] Before providing a detailed description of the embodiments of this application, some of the nouns and terms involved in the embodiments of this application will be explained first. The nouns and terms involved in the embodiments of this application are subject to the following interpretations.

[0022] 1) ACID (ACID properties): refers to the four properties of Atomicity, Consistency, Isolation, and Durability. In this solution, atomicity is guaranteed by transaction logs and the same transaction commits of database operations.

[0023] 2) MQ (Message Queue): This solution uses RabbitMQ middleware, which acts as an asynchronous communication hub to ensure reliable transmission of cached operation logs to the consumer cluster and supports fault tolerance mechanisms such as automatic retry / dead letter queue.

[0024] 3) XA (Distributed Transaction Protocol): The traditional two-phase commit scheme, this scheme avoids the performance bottleneck of XA through eventual consistency design.

[0025] 4) TPS (Transactions Per Second): A key performance indicator, representing the number of transactions processed per second.

[0026] In related technologies, traditional distributed systems perform cache update operations immediately after database transaction commit. This approach leads to extended transaction time, directly impacting core business operations when the cache service becomes unavailable. It suffers from high system coupling, poor fault tolerance, and lacks an effective asynchronous compensation mechanism, failing to meet the need for eventual consistency in high-concurrency scenarios.

[0027] In summary, the technical problems existing in the relevant technologies need to be improved.

[0028] In view of this, this application provides a consistent update method and related equipment. This scheme updates business data through database transactions and writes the business data to the cache operation log, which helps improve data consistency and makes the modification of business data atomic. A scheduled task scans the current log of the cache operation log, where the current log's processing status is unprocessed. The current log is batch-converted into queue messages and sent to the message queue. Even under abnormal conditions such as network jitter or node failure, each cache operation instruction is executed at least once and eventually successfully listens to the message queue through the consumer cluster. The consumer cluster parses the queue messages to obtain the operation type, cache key, and cache value. The consumer cluster calls the corresponding cache client interface according to the operation type, and the cache client interface executes the cache operation to the Redis cache according to the operation type, cache key, and cache value. This makes the cache update operation asynchronous, completely decoupling the business service and the cache service. Failures of one will not directly affect the other's core process, improving data consistency and reliability. Through the combination of "message persistence + retry + status marking," a highly reliable, self-healing eventual consistency closed loop is constructed. As long as the database transaction succeeds, the cache will be correctly updated.

[0029] The consistency update method provided in this application relates to the field of software development technology. The consistency update method provided in this application can be applied to a terminal, a server, or software running on a terminal or server. In some embodiments, the terminal can be a smartphone, tablet, laptop, desktop computer, smart speaker, smartwatch, or in-vehicle terminal, but is not limited to these. The server can be configured as an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. The server can also be a node server in a blockchain network. The software can be an application implementing the consistency update method, but is not limited to the above forms.

[0030] This application can be used in a wide variety of general-purpose or special-purpose computer system environments or configurations. Examples include: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics devices, network PCs, minicomputers, mainframe computers, and distributed computing environments including any of the above systems or devices. This application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform specific tasks or implement specific abstract data types. This application can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0031] It should be noted that in all specific embodiments of this application, when processing data related to user identity or characteristics, such as user information, user behavior data, user historical data, and user location information, user permission or consent is obtained first. Furthermore, the collection, use, and processing of this data comply with relevant laws, regulations, and standards. In addition, when embodiments of this application require access to sensitive personal information of users, separate permission or consent from the user is obtained through pop-ups or redirection to confirmation pages. Only after obtaining the user's separate permission or consent is the necessary user-related data required for the proper functioning of these embodiments acquired.

[0032] Figure 1 This is an optional flowchart of the consistency update method provided in the embodiments of this application. Figure 1The method may include, but is not limited to, steps S101 to S106.

[0033] Step S101: Update business data through database transactions and write the business data to the cache operation log.

[0034] In some embodiments, a write request is initiated by the client, and a database transaction is started by the business service; the business data is updated based on the write request through the database transaction.

[0035] Optionally, a transaction log table (i.e., a cached operation log) can be created in the business database.

[0036] For example, taking the "product inventory deduction" scenario of an e-commerce platform as an example, in this scenario, after a user places an order, the system needs to complete the following operations: 1. Database operation: Deduct 5 units from the inventory of product `001` in the business table.

[0037] 2. Cache operation: Update the latest inventory value of product `001` in the Redis cache.

[0038] The order service receives a request to create an order. The order service opens a database transaction locally and then performs the following operations: Insert the order data into the orders business table.

[0039] Update the inventory deduction data to the inventory business table.

[0040] Construct a cache operation log record and write it to the transaction log table.

[0041] In this embodiment, updating business data through database transactions and writing the business data to the cache operation log ensures that if there are changes in the database, the task of updating the cache is triggered, which helps improve data consistency and makes the modification of business data atomic.

[0042] Step S102: Use a scheduled task to scan the current log of the cached operation log.

[0043] Specifically, the current log processing status is unprocessed.

[0044] In some embodiments, a scheduled task is used to scan the pending logs in the cached operation log.

[0045] Optionally, a scheduled task can be executed at preset time intervals to scan the cached operation log for unprocessed current logs.

[0046] In this embodiment, a scheduled task is used to scan the current log of the cached operation log. Through asynchronous processing, the performance of the business service is no longer directly affected by the response speed and stability of the cache service, and the overall throughput and availability of the system are fundamentally improved.

[0047] Step S103: Convert the current logs into queue messages in batches and send the queue messages to the message queue.

[0048] Understandably, the scheduled task will convert the current logs into a batch of recognizable message formats and send the converted queue messages to the message queues corresponding to the message formats.

[0049] In some embodiments, the processing status of the current log is changed to "processing"; the content of the current log is encapsulated into a queue message.

[0050] Optionally, for each unprocessed log record, the scheduled task performs the following operations: Update its status to 1 (processing) to prevent repeated polling.

[0051] Encapsulate the content of this log entry into a message body (such as JSON format).

[0052] Send the message to the specified message queue (such as RabbitMQ), which ensures that the message is delivered at least once.

[0053] In this embodiment, the current logs are converted into queue messages in batches and the queue messages are sent to the message queue. Even under abnormal circumstances such as network jitter or node failure, each cache operation instruction can be executed at least once and eventually succeed.

[0054] Step S104: Obtain queue messages by listening to the message queue through the consumer cluster.

[0055] Optionally, the consumer cluster includes cache workers.

[0056] In some embodiments, one or more independent CacheWorker services listen to the message queue.

[0057] Optionally, when there is a new message in the queue, the message queue will push the queue message to the cache worker according to a load balancing strategy (such as round-robin).

[0058] In this embodiment, the consumer cluster listens to the message queue to obtain queue messages, and the message queue ensures that each message is executed at least once.

[0059] Step S105: Parse the queue message through the consumer cluster to obtain the operation type, cache key, and cache value.

[0060] In some embodiments, when a new message is received, the cache worker parses the message content to obtain the cache key, operation type, and value to be operated on.

[0061] It is understandable to monitor the processing speed and failure rate of consumers. Alerts should be issued promptly when a large number of failures or message backlog occurs.

[0062] In this embodiment, the consumer cluster parses the queue message to obtain the operation type, cache key, and cache value, which prepares for subsequent cache operations.

[0063] Step S106: The consumer cluster calls the corresponding cache client interface according to the operation type, and performs the cache operation to Redis cache according to the operation type, cache key and cache value through the cache client interface.

[0064] Specifically, caching operations include operation type, cache key, and cache value. The parameters of the cache client interface include cache key and cache value. Idempotent operations include write operations and delete operations. Caching operations are idempotent, and the result of executing an idempotent operation multiple times is the same as the result of executing it once, preventing multiple executions from affecting data accuracy. The cache client interface includes write interface and delete interface.

[0065] It is understandable that if the operation type is write, then the cache operation is a write operation; if the operation type is delete, then the cache operation is a delete operation.

[0066] In some embodiments, if the operation type is write, the write interface is called through the consumer cluster, and the write operation is performed to the Redis cache through the cache client interface. The parameters of the write interface include the cache key and the cache value. Optionally, if the operation type is deletion, the deletion interface is called through the consumer cluster, and the deletion operation is performed to the Redis cache through the cache client interface. The parameters of the deletion interface include the cache key.

[0067] Furthermore, if the cache client interface successfully performs the cache operation, the processing status of the current log corresponding to the queue message in the cache operation log is modified to "processed"; the successfully executed queue message is deleted from the message queue; if the cache client interface fails to perform the cache operation, the failure record in the current log corresponding to the queue message is incremented by one; if the failure record in the current log exceeds the retry threshold, the current log is transferred to the dead letter queue.

[0068] Optionally, the verification task can be used to check whether the cache operation log is synchronized with the data cached in Redis; if the cache operation log is not synchronized with the data cached in Redis, the step of scanning the current log of the cache operation log using a scheduled task can be returned.

[0069] In this embodiment, the consumer cluster calls the corresponding cache client interface according to the operation type. The cache client interface executes the cache operation to the Redis cache according to the operation type, cache key and cache value, making the cache update operation asynchronous and completely decoupling the business service and the cache service. The failure of one party will not directly affect the other's core process, improving data consistency and reliability. Through the combination of "message persistence + retry + status marking", a highly reliable, self-healing eventual consistency closed loop is built. As long as the database transaction is successful, the cache will eventually be updated correctly.

[0070] Steps S101 to S106 of this embodiment update business data through database transactions and write the business data to the cache operation log, which helps improve data consistency and makes the modification of business data atomic. A scheduled task scans the current log of the cache operation log, where the current log's processing status is unprocessed. The current log is batch-converted into queue messages and sent to the message queue. Even under abnormal conditions such as network jitter or node failure, each cache operation instruction is executed at least once and eventually successfully listens to the message queue through the consumer cluster. The consumer cluster parses the queue message to obtain the operation type, cache key, and cache value. The consumer cluster calls the corresponding cache client interface according to the operation type, and the cache client interface executes the cache operation to the Redis cache according to the operation type, cache key, and cache value. This makes the cache update operation asynchronous, completely decoupling the business service and the cache service. Failures of one will not directly affect the other's core process, improving data consistency and reliability. Through the combination of "message persistence + retry + status marking," a highly reliable, self-healing eventual consistency closed loop is constructed. As long as the database transaction succeeds, the cache will be correctly updated.

[0071] Please see Figure 2 In some embodiments, step S101 may include, but is not limited to, steps S201 to S202: Step S201: Initiate a write request through the client and start a database transaction through the business service.

[0072] In step S201 of some embodiments, a cached transaction log table is established in the business database, which includes fields such as operation type, cached key value, and transaction status.

[0073] Optionally, if the log volume is enormous, the transaction log table can be partitioned by time. For example, partitioning by month is beneficial for managing and cleaning up historical data.

[0074] Step S202: Update business data based on write requests through database transactions.

[0075] In step S202 of some embodiments, business data is synchronously updated within the transaction and cache update operation is performed directly.

[0076] Please see Figure 3 In some embodiments, the consistency update method provided in this application further includes a step of determining the operation result, which may include, but is not limited to, steps S301 to S304: Step S301: If the cache client interface successfully performs the cache operation, then the processing status of the current log corresponding to the queue message in the cache operation log is modified to "processed".

[0077] In step S301 of some embodiments, the set / delete atomic operation is executed through the cache client interface. After successful processing, the log status is updated to completed, and the failure record is marked as retry_count+1.

[0078] Step S302: Delete the successful execution message from the message queue.

[0079] In step S302 of some embodiments, the queue message of successful execution is deleted from the message queue, so that the message queue only contains the queue message corresponding to the current log with the status of processing.

[0080] Step S303: If the cache client interface fails to perform the cache operation, increment the failure record in the current log corresponding to the queue message by one.

[0081] In step S303 of some embodiments, if the cache client interface fails to perform a cache operation, the current log processing status is changed to unprocessed, and the process waits for the next log scan to retry.

[0082] Specifically, if the current log fails for the first time, the failure record in the current log is set to 1; if it is not the first failure, the failure record is incremented by 1.

[0083] Step S304: If the number of failed records in the current log exceeds the retry threshold, the current log is transferred to the dead letter queue.

[0084] In step S304 of some embodiments, after the maximum number of retries is exceeded, the system is transferred to a dead-letter queue, triggering a monitoring alarm and awaiting manual intervention for investigation.

[0085] Specifically, if the retries fail after 3 attempts, the message is transferred to the dead-letter queue, triggering a monitoring alarm to notify manual intervention.

[0086] Please see Figure 4In some embodiments, the consistency update method provided in this application further includes a consistency verification step, which may include, but is not limited to, steps S401 to S402: Step S401: Verify whether the cache operation log and the data cached in Redis are synchronized through the verification task.

[0087] In step S401 of some embodiments, a verification task is started periodically during off-peak business hours to compare the data differences between the cache operation log and the Redis cache.

[0088] Optionally, the proofreading task can be performed using full proofreading or incremental proofreading.

[0089] Step S402: If the cache operation log is not synchronized with the data cached in Redis, return to the step of scanning the current log of the cache operation log using a scheduled task.

[0090] In step S402 of some embodiments, if the cache operation log is not synchronized with the data cached in Redis, the step of scanning the current log of the cache operation log using a scheduled task is returned.

[0091] Furthermore, if the cache does not exist, a SET operation is performed to write the cache operation log data to Redis.

[0092] In addition, for data that exists in the cache but has been deleted from the database, perform a DELETE operation to remove it from the cache.

[0093] Please see Figure 5 In some embodiments, step S106 may also include, but is not limited to, steps S501 to S502: Step S501: If the operation type is write, the write interface is called through the consumer cluster, and the write operation is executed to the Redis cache through the cache client interface.

[0094] Specifically, the parameters of the write interface include the cache key and the cache value.

[0095] In step S501 of some embodiments, the consumer cluster concurrently consumes messages and performs atomic operations such as redisTemplate.opsForValue().set / delete.

[0096] Specifically, the cache client interface is invoked based on the operation type: If it is a SET, then redisClient.set(cache_key, cache_value) will be executed.

[0097] If it is a DEL, then redisClient.del(cache_key) will be executed.

[0098] In step S502, if the operation type is deletion, the deletion interface is called through the consumer cluster, and the deletion operation is executed in the Redis cache through the cache client interface.

[0099] Specifically, the parameters for deleting the interface include the cache key.

[0100] In step S502 of some embodiments, if the operation type is DEL, then redisClient.del(cache_key) is executed.

[0101] Figure 6 This is a flowchart illustrating a specific implementation of the consistency update method provided in this application when applied to a message queue-based system for ensuring eventual consistency between the cache and the database. The system architecture diagram of the message queue-based system for ensuring eventual consistency between the cache and the database is shown below. Figure 7 As shown, Figure 6 The methods may include, but are not limited to, the following steps: Step 1: Business processing and log recording.

[0102] In some embodiments, the OrderService receives a request to create an order. It first opens a database transaction locally, and then performs the following operations: Insert the order data into the orders business table.

[0103] Update the inventory deduction data to the inventory business table.

[0104] Construct a cache operation log entry and write it to the cache_transaction_log table. This entry contains: cache_key = "inventory:product_id:1001", operation_type = 1 (SET), cache_value = "{\"stock\": 95}" (the new inventory value).

[0105] Specifically, when the client initiates a write request, the business service starts a database transaction.

[0106] For example, a transaction log table named cache_transaction_log is created in the business database, with the following structure: CREATE TABLE cache_transaction_log ( id BIGINT AUTO_INCREMENT PRIMARY KEY, `transaction_id VARCHAR(64) NOT NULL COMMENT 'Global Transaction ID'` cache_key VARCHAR(512) NOT NULL COMMENT 'Cache key', operation_type TINYINT NOT NULL COMMENT 'Operation type: 1-Set, 2-Delete', cache_value TEXT COMMENT 'Cached value (JSON format)', status TINYINT DEFAULT 0 COMMENT 'Status: 0 - Unprocessed, 1 - Processing, 2 - Completed, 3 - Failed', retry_count INT DEFAULT 0 COMMENT 'Number of retries', create_time DATETIME DEFAULT CURRENT_TIMESTAMP, update_time DATETIME DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP, INDEX idx_status (status), INDEX idx_create_time (create_time), INDEX idx_transaction_id (transaction_id) COMMENT 'Cached transaction log table'; For example, the business process diagram of a message queue-based system for ensuring eventual consistency between cache and database is as follows: Figure 8 As shown.

[0107] Step 2, commit the transaction.

[0108] Specifically, business data is updated synchronously within the transaction and written to the cache operation log (insert into cache_transaction_log).

[0109] In some embodiments, the order service commits a database transaction. Since business table operations and log recording are completed within the same database transaction, they are atomic: either both succeed or both fail. If the transaction commit fails, the entire operation is rolled back, and the cache is not updated.

[0110] Step 3: Asynchronous log polling.

[0111] Specifically, after the transaction is committed, a scheduled task scans log records with a pending status (select ...where status='PENDING' limit 100).

[0112] In some embodiments, a separate background scheduled task (e.g., implemented via Spring Boot's @Scheduled) runs at fixed time intervals (e.g., every 5 seconds). It queries the cache_transaction_log table for all records with status = 0 (unprocessed).

[0113] Step 4, send the message.

[0114] In some embodiments, for each unprocessed log record, the scheduled task performs the following operations: Update its status to 1 (processing) to prevent repeated polling.

[0115] Encapsulate the contents of this log entry (including cache_key, operation_type, cache_value) into a message body (such as JSON format).

[0116] Send the message to the specified message queue (such as RabbitMQ), which ensures that the message is delivered at least once.

[0117] Specifically, logs are converted into MQ messages in batches, and the publisher confirms mechanism is used to ensure the reliability of message delivery.

[0118] If a message fails to be sent, the scheduled task will try to send it again during the next polling, since the status of the record is still 0.

[0119] Step 5: Message consumption and caching operations.

[0120] Specifically, the consumer cluster consumes messages concurrently and executes atomic operations such as redisTemplate.opsForValue().set / delete.

[0121] In some embodiments, one or more independent CacheWorker services listen to this message queue. When a new message is received, it performs the following operations: Parse the message content to obtain the cache key, operation type, and value to be operated on.

[0122] Invoke the cache client interface based on the operation type: If it is a SET, then redisClient.set(cache_key, cache_value) will be executed; If it is a DEL, then redisClient.del(cache_key) will be executed.

[0123] If a caching operation fails (e.g., due to Redis network jitter), the consumer will not send an ACK; the message queue will redeliver the message once the connection is restored. Simultaneously, the consumer can update the `retry_count` field in the database. If the number of retries exceeds a predetermined threshold (e.g., 3 times), the `status` will be set to 3 (failure), and an alarm will be triggered to notify manual intervention.

[0124] It's important to note that due to the "at least once" delivery semantics of message queues, consumers may receive duplicate messages. Therefore, caching operations need to be designed to be idempotent. For example, executing `SET inventory:product_id:1001 95` multiple times yields the same result as executing it once.

[0125] Step 6, Status Update and Confirmation.

[0126] In some embodiments, if the caching operation is successful, the cache worker sends an update command to the database, updating the status of the corresponding log record to 2 (success). Finally, the consumer sends an acknowledgment (ACK) to the message queue, indicating that the message has been successfully processed, and the message queue then deletes the message.

[0127] Specifically, after successful processing, the log status is updated to "completed", and the failure record is marked as "retry_count+1". If it still fails after 3 retries, it is transferred to the dead letter queue, triggering a monitoring alarm to notify manual intervention.

[0128] In addition, a verification task is started every day at midnight to compare the differences between cache_transaction_log and Redis data.

[0129] This invention aims to solve the consistency problem between database transactions and cache operations through an asynchronous message queue processing mechanism, eliminating the impact of traditional synchronous dual-write schemes on system performance. It employs a combination of transaction logs and message delivery to decouple business operations from cache updates, improving system throughput while ensuring eventual consistency. A three-level fault tolerance mechanism (automatic retry / dead-letter queue / manual intervention) ensures data reliability in abnormal scenarios, enabling ordinary distributed systems to achieve tens of thousands of transactions per second without relying on complex transaction managers.

[0130] Specifically, this invention aims to solve the technical problems of difficulty in maintaining strong consistency and low performance of database transactions and caching operations in distributed systems. Its key technical point lies in providing an innovative method that incorporates caching operations into the scope of database transaction management and ultimately guarantees data consistency. The core points are as follows: (1) Atomicity recording mechanism based on database transaction log Key points: In business services, cache operations (such as SET and DELETE) that need to be performed are recorded as a log entry and committed together with the actual business data update operations in the same database transaction.

[0131] Core principle: Leveraging the inherent ACID properties of relational databases (especially atomicity), this ensures absolute atomicity between the modification of business data and the "recording" of cache operations. That is, if the business operation succeeds, the cache operation log will be persisted; if the business operation fails and rolls back, the log will be cleared. This provides a single, reliable data source for subsequent cache updates.

[0132] (2) Asynchronous and decoupled message queue relay mechanism Key points: Design a background scheduler independent of the business service to periodically poll the transaction log table, retrieve log records with a status of "unprocessed", and encapsulate them into messages to send to the message middleware.

[0133] Core Feature: This mechanism completely decouples the two core processes of business processing and cache updates. The performance of business services is no longer directly affected by the response speed and stability of the cache service, fundamentally improving the overall throughput and availability of the system.

[0134] (3) Eventual consistency guarantee mechanism with fault tolerance and retry capabilities Key features: An independent message consumer listens to the message queue, executes the caching operations specified in the message, and updates the log status in the database upon successful operation. This mechanism integrates the retry delivery feature of the message queue with the idempotency handling on the consumer side, ensuring that even under abnormal conditions such as network jitter or node failure, each caching operation instruction is executed at least once and ultimately succeeds.

[0135] Core: By combining message persistence, retries, and status marking, a highly reliable, self-healing, eventually consistent closed loop is constructed. It guarantees that as long as the database transaction succeeds, the cache will eventually be updated correctly, thus achieving a deterministic leap from "potentially inconsistent" to "eventual consistency" at the technical level.

[0136] (4) System-level overall architecture Key points: The system includes a complete system for implementing the method, characterized by the collaborative workflow of a business service module, a transaction log table, a message middleware, a log polling scheduler, and a cache operation consumer module.

[0137] The core aspects include not only the method itself, but also the system architecture that implements the method, the functional division and collaboration between modules, which is conducive to the implementation of the technical solution.

[0138] It should be noted that the beneficial effects of this application are as follows: Performance breakthrough: By processing transaction logs asynchronously, the time taken for database transactions is significantly reduced, and the TPS of a single node is greatly improved.

[0139] Enhanced reliability: The three-level fault tolerance mechanism (automatic retry / dead letter queue / automatic repair) ensures the eventual consistency of data and can automatically handle and repair most abnormal scenarios.

[0140] Intelligent Operation and Maintenance: Automated verification tasks enable minute-level difference detection (1 million records / second processing capacity), and combined with an automatic resubmission mechanism, the MTTR is shortened to the minute level.

[0141] Elastic architecture: The message queue buffer layer supports dynamic scaling of consumers, which greatly improves resource utilization and reduces the time required for cluster expansion.

[0142] Monitoring system: It integrates the collection of 17 indicators (success rate, latency, backlog, etc.) and Grafana dashboards, which can basically cover the automatic early warning of all faults.

[0143] For example, a comparison table between the present invention and conventional solutions is shown in Table 1 below:

[0144] Please see Figure 9 This application also provides a consistency update apparatus that can implement the above method. The apparatus includes: Transaction update module 901 is used to update business data through database transactions and write the business data into the cache operation log; The timed scanning module 902 is used to scan the current log of the cached operation log using a timed task, wherein the processing status of the current log is unprocessed. Message conversion module 903 is used to convert the current log into queue messages in batches and send the queue messages to a message queue; The queue listening module 904 is used to listen to the message queue through the consumer cluster and obtain the queue messages; Parsing module 905 is used to parse the queue message through the consumer cluster to obtain the operation type, cache key and cache value; The caching module 906 is used to call the corresponding caching client interface through the consumer cluster according to the operation type, and to perform a caching operation to the Redis cache through the caching client interface according to the operation type, the cache key and the cache value. The parameters of the caching client interface include the cache key and the cache value, and the caching operation is an idempotent operation.

[0145] It is understood that the content of the above method embodiments is applicable to the present device embodiments. The specific functions implemented by the present device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0146] This application also provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the above-described method. This electronic device can be any smart terminal, including tablet computers, in-vehicle computers, etc.

[0147] It is understood that the content of the above method embodiments is applicable to this device embodiment. The specific functions implemented by this device embodiment are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0148] Please see Figure 10 , Figure 10 The hardware structure of an electronic device according to another embodiment is illustrated. The electronic device includes: The processor 1001 can be implemented using a general-purpose CPU (Central Processing Unit), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this application. The memory 1002 can be implemented as a read-only memory (ROM), static storage device, dynamic storage device, or random access memory (RAM). The memory 1002 can store the operating system and other applications. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 1002 and is called and executed by the processor 1001 using the methods described in the embodiments of this application. Input / output interface 1003 is used to implement information input and output; The communication interface 1004 is used to enable communication and interaction between this device and other devices. Communication can be achieved through wired means (such as USB, network cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.). Bus 1005 transmits information between various components of the device (e.g., processor 1001, memory 1002, input / output interface 1003, and communication interface 1004); The processor 1001, memory 1002, input / output interface 1003 and communication interface 1004 are connected to each other within the device via bus 1005.

[0149] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described method.

[0150] It is understood that the content of the above method embodiments is applicable to this storage medium embodiment. The specific functions implemented in this storage medium embodiment are the same as those in the above method embodiments, and the beneficial effects achieved are also the same as those achieved in the above method embodiments.

[0151] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the above-described method.

[0152] It is understood that the content of the above method embodiments is applicable to the embodiments of this program product. The specific functions implemented by the embodiments of this program product are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0153] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. Furthermore, memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory may optionally include memory remotely located relative to the processor, and these remote memories can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0154] The consistency update method, apparatus, electronic device, storage medium, and program product provided in this application update business data through database transactions and write the business data to the cache operation log, which helps improve data consistency and makes the modification of business data atomic. A scheduled task scans the current log of the cache operation log, and the processing status of the current log is unprocessed. The current log is converted into queue messages in batches and sent to the message queue. Even under abnormal conditions such as network jitter or node failure, each cache operation instruction is executed at least once and eventually successfully listens to the message queue through the consumer cluster. The consumer cluster parses the queue messages to obtain the operation type, cache key, and cache value. The consumer cluster calls the corresponding cache client interface according to the operation type, and the cache client interface executes the cache operation to the Redis cache according to the operation type, cache key, and cache value. The cache update operation is made asynchronous, completely decoupling the business service and the cache service. Failures of one will not directly affect the core process of the other, improving data consistency and reliability. Through the combination of "message persistence + retry + status marking", a highly reliable, self-healing eventual consistency closed loop is constructed. As long as the database transaction succeeds, the cache will be updated correctly.

[0155] The embodiments described in this application are for the purpose of more clearly illustrating the technical solutions of the embodiments of this application, and do not constitute a limitation on the technical solutions provided by the embodiments of this application. As those skilled in the art will know, with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by the embodiments of this application are also applicable to similar technical problems.

[0156] Those skilled in the art will understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of this application, and may include more or fewer steps than shown, or combine certain steps, or different steps.

[0157] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0158] Those skilled in the art will understand that all or some of the steps in the methods disclosed above, as well as the functional modules / units in the systems and devices, can be implemented as software, firmware, hardware, or suitable combinations thereof.

[0159] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0160] It should be understood that in this application, "at least one (item)" means one or more, and "more than" means two or more. "And / or" is used to describe the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one (item) of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one (item) of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0161] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of the units described above is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0162] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0163] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0164] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes multiple instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing programs, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0165] The preferred embodiments of the present application have been described above with reference to the accompanying drawings, but this does not limit the scope of the claims of the present application. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and substance of the embodiments of the present application shall be within the scope of the claims of the present application.

Claims

1. A consistent update method, characterized in that, The method includes the following steps: Update business data through database transactions and write the business data to the cache operation log; A scheduled task is used to scan the current log of the cached operation log, where the current log is in an unprocessed state. The current logs are converted into queue messages in batches, and the queue messages are sent to the message queue. The message queue is monitored by the consumer cluster to obtain the queue messages; The consumer cluster parses the queue messages to obtain the operation type, cache key, and cache value; The consumer cluster calls the corresponding cache client interface according to the operation type. The cache client interface performs a cache operation to the Redis cache according to the operation type, the cache key, and the cache value. The parameters of the cache client interface include the cache key and the cache value. The cache operation is an idempotent operation.

2. The method according to claim 1, characterized in that, The step of updating business data through database transactions and writing the business data to the cache operation log includes: A write request is initiated by the client, and a database transaction is started by the business service. The business data is updated based on the write request through a database transaction.

3. The method according to claim 1, characterized in that, The method further includes: If the cache client interface successfully executes the cache operation, the processing status of the current log corresponding to the queue message in the cache operation log is modified to "processed". Delete the queue message indicating successful execution from the message queue; If the cache client interface fails to perform the cache operation, the failure record in the current log corresponding to the queue message is incremented by one; If the number of failed records in the current log exceeds the retry threshold, the current log will be transferred to the dead letter queue.

4. The method according to claim 1, characterized in that, The method further includes: The verification task verifies whether the cache operation log is synchronized with the data cached in Redis. If the cache operation log is not synchronized with the data cached in Redis, then return to the step of scanning the current log of the cache operation log using a scheduled task.

5. The method according to claim 1, characterized in that, The step of converting the current logs into queue messages in batches includes: Change the current log processing status to "processing"; The contents of the current log are encapsulated into a queue message.

6. The method according to claim 1, characterized in that, The idempotent operations include write and delete operations. The step of calling the corresponding cache client interface through the consumer cluster based on the operation type, and executing the cache operation to the Redis cache through the cache client interface, includes: If the operation type is a write type, then the write interface is called through the consumer cluster, and the write operation is executed to the Redis cache through the cache client interface. The parameters of the write interface include the cache key and the cache value. If the operation type is deletion, the deletion interface is called through the consumer cluster, and the deletion operation is executed to the Redis cache through the cache client interface. The parameters of the deletion interface include the cache key.

7. A consistency update device, characterized in that, The device includes: The transaction update module is used to update business data through database transactions and write the business data into the cache operation log; The scheduled scanning module is used to scan the current log of the cached operation log using a scheduled task, wherein the processing status of the current log is unprocessed. The message conversion module is used to convert the current logs into queue messages in batches and send the queue messages to the message queue. The queue listening module is used to listen to the message queue through the consumer cluster and obtain the queue messages; The parsing module is used to parse the queue messages through the consumer cluster to obtain the operation type, cache key, and cache value; The caching module is used to call the corresponding caching client interface through the consumer cluster according to the operation type, and to perform caching operations to the Redis cache through the caching client interface according to the operation type, the cache key and the cache value. The parameters of the caching client interface include the cache key and the cache value, and the caching operation is an idempotent operation.

8. An electronic device, characterized in that, The electronic device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the method according to any one of claims 1 to 6.

9. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the method of any one of claims 1 to 6.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the method of any one of claims 1 to 6.

Images