Data update methods and systems for database slave nodes

By starting multiple consumer threads on the database slave node to process update log records in parallel, the problem of low data update efficiency of the slave node under high concurrency writes is solved, and efficient data synchronization and consistency are achieved.

CN123115153APending Publication Date: 2026-11-17FUJIAN HUAYU EDUCATION TECH CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202610995888.X
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2026-07-06
Publication Date
2026-11-17

AI Technical Summary

Technical Problem

In high-concurrency write scenarios, traditional databases suffer from low data update efficiency on slave nodes, leading to increased master-slave synchronization latency and failing to meet high-concurrency requirements.

Method used

When an update log is received from a node, multiple consumer threads are started. The record attributes are parsed and assigned to the corresponding target consumer threads to process the update log records in parallel, ensuring that the same record is processed by the same thread and different records are executed in parallel by different threads.

Benefits of technology

It improves the data update efficiency of slave nodes, reduces master-slave synchronization latency, and ensures the availability and data consistency of the system in high-concurrency write scenarios.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN123115153A_ABST
    Figure CN123115153A_ABST
Patent Text Reader

Abstract

This application provides a method and system for updating data on a database slave node. The method includes: when a slave node receives an update log, starting at least two consumer threads; for each record in the update log, parsing the record attributes of the record, determining the target consumer thread corresponding to the record based on the record attributes, and assigning the record to the corresponding target consumer thread; and having each target consumer thread process the assigned record in parallel to update the data in the slave node. This application enables different records to be assigned to different threads for concurrent execution, fully utilizing multi-core CPU resources, significantly improving the throughput of change log processing on the slave node, and enhancing the data update efficiency of the slave node.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This invention relates to the field of database technology, and in particular to a method and system for updating data on slave nodes of a database. Background Technology

[0002] MySQL, a widely used open-source relational database, relies on its master-slave replication mechanism as a key technology for ensuring high data availability and achieving read-write separation. The master node handles write operations (such as insert, delete, and update operations) and records all data changes in a binary log (binlog) file. Slave nodes synchronize data with the master node by retrieving and executing the SQL statements from this log file.

[0003] Data synchronization between master and slave nodes primarily relies on the binlog mechanism. The master node writes data changes sequentially into the binlog and transmits the log file to the slave node. Upon receiving the log file, the slave node parses and executes the SQL statements line by line. While this serial execution method maintains transaction integrity and data consistency, it is inefficient. In high-concurrency write scenarios, the slave node can only replay binlog events sequentially in a single thread, which can easily lead to increased master-slave synchronization latency and other problems. Summary of the Invention

[0004] The technical problem to be solved by the present invention is to provide a data update method and system for database slave nodes, which can improve the data update efficiency of database slave nodes.

[0005] To solve the above-mentioned technical problems, the technical solution adopted by the present invention is as follows: A method for updating data on a slave node of a database, the method comprising: When an update log is received from a node, at least two consumer threads are started. For each record in the update log, the record attributes of the record are parsed, and the target consumer thread corresponding to the record is determined based on the record attributes. The record is then assigned to the corresponding target consumer thread. The allocated records are processed in parallel by each of the target consumer threads to update the data in the slave nodes.

[0006] To solve the above-mentioned technical problems, another technical solution adopted by the present invention is as follows: A data update system for a database slave node includes a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that the processor executes the computer program to implement the various steps of any of the data update methods for a database slave node as described above.

[0007] The beneficial effects of this invention are as follows: For each change record in the update log, the record is assigned to the corresponding target consumer thread according to its attributes. Each record represents a data change operation, ensuring that the same change record always enters the same thread, thereby maintaining the execution order of each record and avoiding the problem of chaotic changes to the same data, thus ensuring the accuracy of data updates in the slave node. Based on multiple preset consumer threads, different records can be assigned to different threads for concurrent execution, fully utilizing multi-core CPU resources to significantly improve the throughput of change log processing in the slave node and increase the data update efficiency of the slave node. Therefore, this application can effectively reduce master-slave synchronization latency while ensuring data synchronization quality, guaranteeing the availability of the system in high-concurrency write scenarios. Attached Figure Description

[0008] Figure 1 This is a flowchart illustrating a data update method for a database slave node according to an embodiment of the present invention; Figure 2 This is another flowchart of a database slave node data update method according to an embodiment of the present invention; Figure 3 This is a schematic diagram of a database slave node data update system according to an embodiment of the present invention. Detailed Implementation

[0009] Definitions:

[0010] To explain in detail the technical content, objectives, and effects of the present invention, the following description is provided in conjunction with the embodiments and accompanying drawings.

[0011] In existing technologies, traditional database slave nodes use a single-threaded serial replay update log mode for data updates. This mode can strictly guarantee the transaction execution order and data consistency, avoiding problems such as data corruption and dirty data caused by concurrent updates. However, in high-concurrency internet business scenarios, the master node writes frequently and the update log data volume is large. The single-threaded serial processing mode will greatly limit the data update efficiency of the slave node, causing the slave node log processing speed to lag behind the master node data write speed, resulting in serious master-slave data latency.

[0012] To at least address the aforementioned issues, after receiving the update log from the master node, the slave node starts at least two consumer threads. It then parses the record attributes of each record in the update log, determines the corresponding target consumer thread based on the parsed attributes, and distributes the record to the appropriate thread. Each target consumer thread processes its assigned record in parallel, ultimately completing the data update for the slave node. This approach allows different records to be assigned to different threads for concurrent execution, fully utilizing multi-core CPU resources to significantly improve the throughput of change log processing on the slave node and enhance its data update efficiency.

[0013] The following describes in detail a method for updating data on a database slave node according to the present invention. Please refer to [link / reference]. Figure 1 The method 100 includes steps 110 to 130.

[0014] In step 110, when an update log is received from a slave node, at least two consumer threads are started.

[0015] In one alternative implementation, the slave node continuously listens to the master node's log push channel and receives the binlog update logs synchronously transmitted by the master node in real time. Once an update log is detected, a pre-initialized thread pool is immediately started, with at least two consumer threads.

[0016] In step 120, for each record in the update log, the record attributes of the record are parsed, and the target consumer thread corresponding to the record is determined according to the record attributes, and the record is assigned to the corresponding target consumer thread.

[0017] In one optional implementation, the primary key identifier of the record and the table name to which the record belongs are used as the record attributes. Each record in the update log is traversed from the node, and the corresponding table name and primary key identifier are parsed and extracted as record attributes. The record attribute is then used as the key to query the local cache for a corresponding unique thread identifier. If it exists, the consumer thread corresponding to that thread identifier is directly identified as the target consumer thread. If it does not exist, a hash operation is performed on the record attribute, and the initial target consumer thread is obtained by modulo matching with the total number of consumer threads. Subsequently, the number of records to be processed in the queue of the initial target consumer thread is queried. If the number does not exceed a preset load threshold, the record is directly assigned to that thread queue. If the load threshold is exceeded, the load-compliant consumer threads are repeatedly re-filtered and verified to obtain the final target consumer thread and complete the record allocation. Simultaneously, the mapping relationship between record attributes and thread identifiers in the cache is updated synchronously.

[0018] In step 130, the allocated records are processed in parallel by each of the target consumer threads to update the data in the slave nodes.

[0019] In one optional implementation, the system assigns the log records that have completed load verification and been distributed to each final target consumer thread to multiple independent target consumer threads for parallel processing. Each consumer thread retrieves the corresponding log records from its own queue in an orderly manner and performs log replay operations one by one. The log records of the same row of data are always executed serially by a single consumer thread to ensure data update sequence and transaction consistency. The log records of different rows of data are executed concurrently by different consumer threads. Finally, the batch data update of the slave nodes is completed through multi-threaded collaborative processing.

[0020] As described above, by starting at least two consumer threads when a slave node receives an update log, matching and distributing the log records according to their attributes, and replaying the log records through multi-threaded parallel execution to update the slave node's data, log records of different rows can be distributed to different consumer threads for concurrent execution. This improves the slave node's log processing throughput and data update efficiency, resolving issues such as master-slave latency in high-concurrency write scenarios. Furthermore, by binding target threads to record attributes, the update task for a single data row in the log is strictly limited to a dedicated consumer thread, ensuring the update sequence and transaction consistency of single data rows, improving the efficiency of database master-slave synchronization, and guaranteeing the accuracy of slave node data updates.

[0021] In one alternative implementation, step 120 includes steps 121 to 123.

[0022] Step 121: Using the record attribute as the key, query whether there is a corresponding thread identifier in the cache, where the thread identifier uniquely corresponds to the consumer thread.

[0023] In one optional implementation, the system uses the parsed record attributes as a retrieval key to query the local cache of the slave node, determining whether a corresponding thread identifier is stored under that key. Each thread identifier is uniquely bound to a single consumer thread, and there are no duplicate correspondences. Since the cache stores the mapping relationship between the records to be processed and the consumer threads, querying the cache can accurately determine whether the data row to which the currently pending log record belongs has been pre-bound to a fixed consumer thread.

[0024] Step 122: If it exists, then the consumer thread corresponding to the queried thread identifier is determined as the target consumer thread.

[0025] In one optional implementation, if the cached query results contain a thread identifier corresponding to the current record attribute, the thread identifier is read directly, and the consumer thread uniquely corresponding to the thread identifier is directly determined as the target consumer thread for processing the current log record, without recalculating the matching thread.

[0026] Step 123: If it does not exist, perform a hash operation on the record attribute to obtain a hash value, determine the target consumer thread based on the hash value, and store the record attribute as the key and the thread identifier of the target consumer thread as the value in the cache.

[0027] In one optional implementation, if the thread identifier corresponding to the current record attribute is not found in the cache, it means that the current data row is not bound to a processing thread. At this time, the system performs a hash operation on the record attribute to generate a corresponding hash value, and determines the corresponding target consumer thread based on the hash value. At the same time, the key-value pair is stored in the cache with the record attribute as the key and the thread identifier of the matched target consumer thread as the value, thus completing the binding of the current record with the consumer thread.

[0028] As described above, by caching the mapping relationship between currently pending log records and consumer threads, when allocating the initial target consumer thread for the current log record, the cache is quickly searched to see if a thread identifier matching the current log record exists. If so, the existing mapping relationship is reused, eliminating redundant hash calculations and improving task allocation efficiency. For data rows without cached records, the initial thread allocation is completed by hashing the record attributes and synchronously stored in the cache. This allows the same row of data to be continuously bound to the same consumer thread, ensuring that single-row data updates can be executed serially and effectively avoiding data corruption caused by concurrent processing.

[0029] In one alternative implementation, step 123 includes step 1231.

[0030] Step 1231: Perform a modulo operation on the hash value with the total number of consumer threads, and determine the target consumer thread corresponding to the record based on the modulo operation result.

[0031] In one optional implementation, the system obtains the total number of all consumer threads that have been started by the current slave node, performs a modulo operation on the total number of threads using the hash value calculated through the record attribute, and obtains the modulo result. Each modulo result corresponds to a consumer thread with a fixed thread identifier in advance. The system matches a unique consumer thread according to this correspondence, thereby determining the initial target consumer thread corresponding to the current record, that is, the first target consumer thread initially allocated.

[0032] As described above, matching the target consumer thread by taking the modulo of the hash value with the total number of consumer threads can establish a mapping relationship between log records and consumer threads, and can evenly distribute different records to each consumer thread, achieving a balanced allocation of initial tasks and avoiding tasks accumulating on some consumer threads.

[0033] In one alternative implementation, step 124 includes steps 1241 to 1242.

[0034] Step 1241: Query the number of pending records in the pending queue of the target consumer thread.

[0035] In one alternative implementation, before determining the initial target consumer thread and formally distributing log records, the system detects the thread load of the initial target consumer thread, accesses the pending queue of the target consumer thread in real time, counts the total number of log records in the queue that have not yet been processed, and obtains the current backlog of tasks for the thread in real time.

[0036] Step 1242: Assign the records to the target consumer thread according to the number of records to be processed.

[0037] In one optional implementation, the system compares the number of pending records obtained from the query with a preset load threshold to determine the load level of the current target consumer thread. Based on the actual load status of the thread, the system selects the corresponding record allocation strategy. When the thread load is normal, the record allocation is completed directly. When the thread is overloaded, the system triggers the reselection of a consumer thread with a lighter load, thereby achieving dynamic and reasonable allocation of the current log records to the appropriate consumer thread and avoiding the accumulation of single-threaded tasks.

[0038] As described above, by querying the number of records to be recorded in the queue of the target consumer thread in real time, the current load status of the target consumer thread can be determined. This can be used as the basis for record allocation, which can identify the backlog of thread tasks in advance and avoid continuously dispatching new tasks to high-load threads.

[0039] In one alternative implementation, step 1242 includes steps 12421 to 12422.

[0040] Step 12421: If the number of records to be processed does not exceed the preset load threshold, the records are allocated to the processing queue of the target consumer thread.

[0041] In one optional implementation, the system compares the number of pending records of the current target consumer thread with a preset load threshold. If the number of records in the thread's pending queue does not exceed the preset load threshold, it is determined that the thread has sufficient load and there is no risk of task backlog. It can normally accept new log records and directly allocate the current pending update log records to the pending queue corresponding to the target consumer thread, waiting for the target consumer thread to schedule and execute replay processing.

[0042] Step 12422: If the number of records to be processed exceeds the load threshold, a new target consumer thread is re-determined for the records, and the step of allocating the records to the target consumer thread according to the number of records to be processed is executed again until the number of records to be processed does not exceed the preset load threshold, and the final target consumer thread is obtained, and the records are allocated to the final target consumer thread.

[0043] In one optional implementation, if the number of pending records of the currently matched target consumer thread exceeds a preset load threshold, the thread is determined to be in an overloaded state, and no more current log records are allocated to the thread. The system then re-selects and matches new target consumer threads for the current log records and repeatedly executes the thread load detection and record allocation steps to continuously verify the queue load of the newly matched threads until a consumer thread with a number of pending records that meets the load requirements is selected. This thread is then identified as the final target consumer thread, and the allocation of the current log records is completed.

[0044] As described above, based on the preset load threshold, tasks are directly distributed when the thread load is normal. When the thread is overloaded, new target consumer threads are matched for the current log record and the thread load is checked in a loop until a target consumer thread with a lighter load is matched. This can balance the running pressure of each consumer thread and prevent task backlog in a single thread, so as to ensure the overall efficiency and running stability of log processing in high-concurrency scenarios.

[0045] In an alternative implementation, step 12422 is followed by step 12423.

[0046] Step 12423: Using the record attribute of the record as the key, update the corresponding thread identifier in the cache to the thread identifier of the final target consumer thread.

[0047] In one optional implementation, after determining the final target consumer thread that complies with load regulations, the system uses the record attribute corresponding to the current log record as the retrieval key to traverse the local cache of the slave node, locate the thread identifier originally bound to the row of data, and update the original thread identifier stored in the cache to the thread identifier of the final target consumer thread. The mapping relationship between the record and the consumer thread is updated in real time to ensure that all subsequent log records of the row of data are routed to the final target consumer thread for processing.

[0048] As described above, when a new target consumer thread is matched for the current record, the mapping relationship between the record attributes and the thread identifier in the cache is updated synchronously, so that all subsequent log records of this row of data are routed to the new target consumer thread for processing, thereby ensuring that the data update order of a single record remains unchanged and thus guaranteeing the data consistency between the master and slave nodes.

[0049] In an alternative implementation, step 12424 is included after step 12422.

[0050] Step 12424: When it is detected that the number of pending records of the first target consumer thread to which the record is assigned has fallen below the load threshold, the record is reassigned to the first target consumer thread, and the corresponding thread identifier in the cache is found using the record attribute of the record as the key, and the thread identifier is updated to the thread identifier of the first target consumer thread.

[0051] In one optional implementation, the system continuously monitors the load status of the queue of the first target consumer thread that was initially matched in real time. When the number of records to be processed by the first target consumer thread falls below a preset load threshold and the load returns to normal, the system reallocates subsequent log records of that row of data to the first target consumer thread. Simultaneously, the system retrieves the cache using the record attribute as the key and updates the corresponding thread identifier in the cache to the thread identifier of the first target consumer thread, achieving dynamic thread allocation switching and load balancing. If the current record has not yet been processed when the allocation is switched, the record continues to be processed in the existing final target consumer thread until it is completed. Only newly added log records after that row of data are uniformly reallocated to the first target consumer thread that has recovered to normal load, and the thread identifier mapping relationship in the cache is updated synchronously.

[0052] As described above, by monitoring the load changes of the initially overloaded consumer thread in real time, and redistributing the corresponding record back to the initial thread after the load of the initially allocated target consumer thread returns to normal, and synchronously updating the mapping relationship in the cache, the overall load distribution can be dynamically adjusted, thereby improving resource utilization.

[0053] In an alternative implementation, step 120 is followed by step 140.

[0054] Step 140: After the target consumer thread finishes processing the record, delete the key-value pair consisting of the record attribute and the corresponding thread identifier of the record from the cache.

[0055] In one optional implementation, after the target consumer thread completes the replay operation of the current log record and the corresponding data update of the slave node, the system retrieves the corresponding mapping relationship in the cache based on the record attribute corresponding to the current record, and deletes the key-value pair in the cache consisting of the record attribute and the corresponding thread identifier. As described above, promptly deleting the corresponding key-value pairs from the cache after record processing cleans up invalid mapping data, reduces cache resource consumption, avoids cache redundancy accumulation, and ensures that only the thread mapping relationship of the currently processed row data remains in the cache. This guarantees the accuracy of subsequent thread hash allocation, load verification, and dynamic switching logic. Furthermore, deleting the key-value pairs corresponding to completed records from the cache signifies the completion of the current update task for that row. When a new update log is received, and the row shows further data changes in the new log, the system will re-execute the hash allocation process, ensuring real-time synchronization between cached data and the actual task status.

[0056] Figure 2 This is another flowchart illustrating a data update method for a database slave node according to an embodiment of the present invention. This embodiment uses a MySQL database master-slave replication architecture as an application scenario, referring to... Figure 2 This embodiment includes steps 201 to 209.

[0057] Step 201: Receive update logs and start several consumer threads.

[0058] In one optional implementation, the slave nodes of the MySQL database receive binlog update logs synchronously transmitted from the master node, while the master node retains the original log writing and transmission logic. Upon receiving the update logs, the slave node automatically starts 10 pre-defined consumer threads, constructing a parallel processing thread pool. Each consumer thread is independently configured with a waiting queue for receiving and processing update log records.

[0059] Step 202: Parse the update log and extract record attributes.

[0060] In one optional implementation, the slave node parses the received binlog update logs one by one. For each data change record, the record attributes are extracted. The record attributes include the table name (e.g., data table name) to which the current record belongs and the primary key identifier. The table name and the primary key identifier are combined into a string, and the combined string is used as the unique identifier of the record for matching consumer threads.

[0061] Step 203: Initially determine the first target consumer thread.

[0062] In one optional implementation, the parsed table name and primary key identifier are used as the search key to query whether a corresponding thread identifier exists in the local Map structure cache. If a thread identifier corresponding to the key exists in the cache, the consumer thread corresponding to that thread identifier is directly determined as the first target consumer thread. If a thread identifier corresponding to the key exists in the cache, it means that the row data to which the current log record belongs has been assigned to a designated consumer thread for processing in the historical update round (i.e., when the slave node processes the update logs received before), but has not yet been processed. The system has bound a dedicated consumer thread to the current record.

[0063] If no corresponding key-value pair is found in the cache, it indicates that the row data to which this log record belongs is either being updated for the first time or the previous update has been fully processed, and the system has not yet bound a dedicated consumer thread to this row data. In this case, a hash operation is performed on the combined string of the table name and primary key identifier to obtain a hash value. This hash value is then moduloed with the total number of threads (e.g., setting 10 consumer threads to be started from slave nodes). The corresponding consumer thread is matched based on the modulo result, completing the allocation of the first consumer thread. Simultaneously, the table name and primary key identifier are used as the key, and the thread identifier of the first matched target consumer thread is used as the value, stored in the local cache to establish a mapping relationship between records and consumer threads.

[0064] Step 204: Real-time monitoring of the load status of the consumer thread.

[0065] In one optional implementation, after determining the first target consumer thread, the node queries in real time the number of existing records in the pending queue of the target consumer thread and compares it with a preset load threshold (e.g., 1000 records). If the number of pending records for the first target consumer thread is less than the preset threshold, it indicates that the load is normal, and the current log record is directly assigned to the pending queue of that consumer thread. If the number of pending records for the first target consumer thread is greater than or equal to the load threshold, the thread is deemed overloaded. The first target consumer thread is abandoned, and other consumer threads are selected again through hash matching and cache retrieval. The load detection steps are repeated, and the queue backlog of each newly matched thread is checked one by one until a thread with the number of pending records not exceeding the threshold is selected and determined as the final target consumer thread.

[0066] Step 205: Update the mapping relationship between the records in the cache and the consumer thread.

[0067] In one optional implementation, after determining the final target consumer thread, the slave node updates the corresponding thread identifier in the cache using the table name and primary key identifier of the current record as the key, and replaces the thread identifier of the original first target consumer thread with the thread identifier of the final target consumer thread, so as to ensure that all subsequent log records of this row of data (i.e. the current log record) are preferentially matched with the latest processing thread.

[0068] Step 206: Each consumer thread performs data updates in parallel.

[0069] In one optional implementation, a single consumer thread pulls allocated log records from its own queue and serially executes log replay operations within its own queue to complete data operations such as adding, modifying, deleting, and updating local data on the node. All log records for the same row of data are always processed serially by the same thread to ensure data update sequence. Log records for different rows of data are processed in parallel by different threads to improve overall update efficiency.

[0070] Step 207: Clear the cache after the data update task is completed.

[0071] In one optional implementation, after the target consumer thread completes all replay operations for a single log record and the local data update takes effect, the system automatically retrieves the key-value pair corresponding to the table name and primary key identifier of the record in the cache and performs a deletion operation to remove the mapping relationship between the processed record and the consumer thread stored in the cache, and only retains the mapping relationship between the unprocessed record and the consumer thread, so as to avoid cache data redundancy and ensure the accuracy of subsequent thread allocation.

[0072] Step 208: Monitor the recovery status of overloaded consumer threads in real time.

[0073] In one optional implementation, the system continuously monitors the queue load status of all consumer threads in real time. When the previously overloaded first target consumer thread completes its queue backlog task and the number of pending records falls below the load threshold, a rollback mechanism is triggered. All log records initially allocated to that thread are reassigned to that consumer thread. For log records reassigned to the first target consumer thread, the system searches for the corresponding thread identifier in the cache based on the log record's table name and primary key, and updates that thread identifier to match the first target consumer thread's thread identifier, thereby achieving dynamic balancing of thread load.

[0074] Please refer to Figure 3 The present invention also provides a data update system 300 for a database slave node, including a memory 301, a processor 302, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the various steps of the data update method for a database slave node as described above.

[0075] In summary, this application provides a method and system for updating data on a database slave node. When a slave node receives update logs, it starts at least two consumer threads to distribute tasks and replay them in parallel based on log record attributes, thereby updating local data. This not only improves log processing throughput and data update efficiency through multi-threaded concurrency, solving the master-slave latency problem under high-concurrency writes, but also ensures the accuracy of update sequence, transactions, and data synchronization for individual data rows by assigning update tasks to dedicated consumer threads through attribute binding.

[0076] By caching the mapping relationship between log records and consumer threads, existing mappings are prioritized for retrieval and reuse during task allocation, eliminating redundant hash calculations and accelerating allocation efficiency. For data rows without cached records, attribute hashing is used to complete the initial thread allocation and write the data to the cache, ensuring that single rows of data are always processed serially by the same thread, avoiding data corruption caused by concurrency. During allocation, the hash value is modulo the total number of threads to achieve initial balanced task distribution and prevent local task backlog.

[0077] Load is assessed by real-time querying of the consumer thread's pending queue length, and tasks are allocated based on a preset load threshold. If a thread's load is normal, the record is directly assigned; if overloaded, a lower-load consumer thread is reassigned to balance thread load and ensure overall operational stability. If the target consumer thread is changed, the cache mapping is updated synchronously to maintain the data update order. When the original consumer thread's load recovers, the corresponding record can be reassigned back to the initial consumer thread, and the cache mapping updated, dynamically optimizing load and improving resource utilization.

[0078] After a consumer thread finishes processing a record, the system promptly deletes the key-value pair corresponding to that record from the cache, cleans up redundant data, reduces resource consumption, keeps the cached data synchronized with the actual task status, and ensures the reliable operation of subsequent allocation, verification, and thread switching logic.

[0079] The above description is merely an embodiment of the present invention and does not limit the patent scope of the present invention. Any equivalent modifications made based on the content of the present invention specification and drawings, or direct or indirect applications in related technical fields, are similarly included within the patent protection scope of the present invention.

Claims

1. A method for updating data on a slave node of a database, characterized in that, include: When an update log is received from a node, at least two consumer threads are started. For each record in the update log, the record attributes of the record are parsed, and the target consumer thread corresponding to the record is determined based on the record attributes. The record is then assigned to the corresponding target consumer thread. The allocated records are processed in parallel by each of the target consumer threads to update the data in the slave nodes.

2. The method according to claim 1, characterized in that, Determining the target consumer thread corresponding to the record based on the record attributes includes: Using the record attribute as the key, query whether there is a corresponding thread identifier in the cache, where the thread identifier uniquely corresponds to the consumer thread; If it exists, the consumer thread corresponding to the queried thread identifier will be determined as the target consumer thread; If it does not exist, a hash operation is performed on the record attribute to obtain a hash value. The target consumer thread is determined based on the hash value, and the record attribute is used as the key, while the thread identifier of the target consumer thread is used as the value and stored in the cache.

3. The method according to claim 2, characterized in that, Determining the target consumer thread based on the hash value includes: The hash value is moduloed by the total number of consumer threads, and the target consumer thread corresponding to the record is determined based on the modulo operation result.

4. The method according to claim 1, characterized in that, The step of assigning the record to the corresponding target consumer thread includes: Query the number of pending records in the pending queue of the target consumer thread; The records are allocated to the target consumer thread based on the number of records to be processed.

5. The method according to claim 4, characterized in that, The step of allocating the records to the target consumer thread according to the number of records to be processed includes: If the number of records to be processed does not exceed the preset load threshold, the records are allocated to the processing queue of the target consumer thread. If the number of records to be processed exceeds the load threshold, a new target consumer thread is re-determined for the records, and the step of allocating the records to the target consumer thread according to the number of records to be processed is executed again until the number of records to be processed does not exceed the preset load threshold, and a final target consumer thread is obtained, and the records are allocated to the final target consumer thread.

6. The method according to claim 5, characterized in that, After obtaining the final target consumer thread, the process also includes: Using the record attribute of the record as the key, update the corresponding thread identifier in the cache to the thread identifier of the final target consumer thread.

7. The method according to claim 5, characterized in that, Also includes: When the number of pending records of the first target consumer thread to which the record is assigned falls below the load threshold, the record is reassigned to the first target consumer thread, and the corresponding thread identifier in the cache is found using the record attribute of the record as the key, and the thread identifier is updated to the thread identifier of the first target consumer thread.

8. The method according to claim 2, characterized in that, After assigning the record to the corresponding target consumer thread, the method further includes: After the target consumer thread finishes processing the record, it deletes the key-value pair consisting of the record attribute and the corresponding thread identifier from the cache.

9. The method according to claim 1, characterized in that, Also includes: The primary key identifier of the record and the table name to which the record belongs are used as the record attributes.

10. A data update system for a slave node of a database, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the various steps of the data update method for a slave node of a database as described in any one of claims 1 to 9.