Heterogeneous data synchronization and proofreading collaboration method based on multi-target feedback control

Abstract

Embodiments of the present disclosure disclose a heterogeneous data synchronization proofreading collaboration method based on multi-target feedback control. A specific embodiment of the method comprises: a producer thread delivers each change event of a collected relational database to a message queue; the producer thread determines a repair request information set; a consumer thread determines an initial normal queue and an initial high priority queue; the consumer thread updates the initial normal queue and the initial high priority queue; the consumer thread dynamically adjusts a batch window time and a normal queue scheduling weight; the consumer thread collects a batch message set; the consumer thread performs deduplication processing on the batch message set; and the consumer thread updates a normal queue, a high priority queue, a version number information set stored in a version number memory, and a target data source. The embodiment is related to the technical field of heterogeneous data synchronization, and realizes lightweight, high-reliability, self-adaptive, and low-cost heterogeneous data synchronization.

Description

Technical Field

[0001] The embodiments disclosed herein relate to the field of heterogeneous data synchronization technology, specifically to a heterogeneous data synchronization, calibration, and collaborative method based on multi-objective feedback control. Background Technology

[0002] In business systems, relational databases are typically used as the primary storage. However, to support efficient search and aggregation analysis, data needs to be synchronized to non-relational warehouses such as Elasticsearch (a search server). Existing synchronization solutions have the following technical problems: First, heavy framework dependency and state bloat: Current mainstream solutions rely on streaming computing frameworks, which require maintaining a large amount of state (such as user sessions). The size of the state will grow linearly with the amount of active data, resulting in high memory and disk overhead and high operation and maintenance costs.

[0003] Second, concurrency conflicts and out-of-order overwriting issues: Update messages for the same primary key in different relational tables may arrive out of order, or duplicate consumption may occur due to message queue rebalancing, causing the target document to be overwritten by old data.

[0004] Third, severe write amplification: In ERP (Enterprise Resource Planning) systems, a single transaction may update the same row of records multiple times, generating multiple change logs. If each log triggers a write operation, it will cause a surge in write pressure on the target storage.

[0005] Fourth, there is a lack of fallback mechanisms for data consistency: even with real-time synchronization links, data inconsistencies can still occur due to program defects, network failures, etc. Existing verification mechanisms either operate independently of real-time synchronization, leading to data rollback, or suffer from excessively long delays in repairing message backlogs.

[0006] Fifth, lack of systematic adaptive capability: Existing solutions typically employ fixed-parameter batching windows, fixed-priority scheduling, and fixed-version-number storage strategies, which cannot dynamically adjust resource allocation based on real-time traffic, backlog levels, and data consistency status.

[0007] The information disclosed in this background section is only intended to enhance the understanding of the background of the inventive concept, and therefore may contain information that does not form prior art known to those skilled in the art. Summary of the Invention

[0008] The summary portion of this disclosure is intended to provide a brief overview of the concepts, which will be described in detail in the detailed description portion. This summary portion is not intended to identify key or essential features of the claimed technical solutions, nor is it intended to limit the scope of the claimed technical solutions.

[0009] Some embodiments of this disclosure propose a heterogeneous data synchronization and collation collaborative method based on multi-objective feedback control to solve the technical problems mentioned in the background section above.

[0010] In a first aspect, some embodiments of this disclosure provide a heterogeneous data synchronization and collation collaboration method based on multi-objective feedback control. The method includes: a producer thread submitting each change event collected from a relational database to a message queue, and sending the message queue to a consumer thread; the producer thread determining a repair request information set based on a pre-determined row-level version number information set stored in a version number memory and the relational database, and sending the repair request information set to the consumer thread; the consumer thread, in response to receiving the message queue and the repair request information set sent by the producer thread, determining an initial ordinary queue and an initial high-priority queue; and the consumer thread, based on the row-level version number information set stored in the version number memory... The row-level version number information set is used to update the initial ordinary queue and the initial high-priority queue to generate ordinary queues and high-priority queues. The consumer thread dynamically adjusts the batching window time and the ordinary queue scheduling weight based on the ordinary queue and the high-priority queue. The consumer thread collects batching message sets based on the batching window time, the ordinary queue scheduling weight, and the ordinary and high-priority queues. The consumer thread deduplicates the batching message sets to generate deduplicated batching message sets. Based on the deduplicated batching message sets, the consumer thread updates the ordinary queue, the high-priority queue, the row-level version number information set stored in the version number memory, and the target data source.

[0011] The various embodiments disclosed above have the following beneficial effects: Through the heterogeneous data synchronization and collation collaborative method based on multi-objective feedback control in some embodiments of this disclosure, firstly, write amplification is significantly reduced: dynamic adaptive windows combined with batch deduplication can reduce the number of writes by more than 90% in high-frequency update scenarios; secondly, real-time performance and consistency are balanced: priority queues ensure that collation repair is not delayed due to message backlog; version number lag feedback enables rapid repair of severely inconsistent data; thirdly, adaptive optimization: multi-objective feedback control automatically balances write pressure, synchronization latency, and collation latency, eliminating the need for manual parameter tuning; fourthly, eventual consistency is guaranteed: collation tasks and real-time synchronization share version number control, completely avoiding data rollback; fifthly, resource consumption is low: stateless consumers avoid state bloat. Therefore, lightweight, highly reliable, adaptive, and low-cost heterogeneous data synchronization is achieved. Attached Figure Description

[0012] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. Throughout the drawings, the same or similar reference numerals denote the same or similar elements. It should be understood that the drawings are schematic, and elements are not necessarily drawn to scale.

[0013] Figure 1 This is a flowchart of some embodiments of the heterogeneous data synchronization and collation collaborative method based on multi-objective feedback control according to the present disclosure; Figure 2 This is a timing flowchart of a consumer thread according to some embodiments of the heterogeneous data synchronization and calibration coordination method based on multi-objective feedback control disclosed herein. Detailed Implementation

[0014] Embodiments of this disclosure will now be described in more detail with reference to the accompanying drawings. While some embodiments of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this disclosure. It should be understood that the accompanying drawings and embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of protection of this disclosure.

[0015] It should also be noted that, for ease of description, only the parts relevant to the invention are shown in the accompanying drawings. Unless otherwise specified, the embodiments and features described in this disclosure can be combined with each other.

[0016] It should be noted that the concepts of "first" and "second" mentioned in this disclosure are used only to distinguish different devices, modules or units, and are not used to limit the order of functions performed by these devices, modules or units or their interdependencies.

[0017] It should be noted that the terms "a" and "a plurality of" used in this disclosure are illustrative rather than restrictive, and those skilled in the art should understand that, unless otherwise expressly indicated in the context, they should be understood as "one or more".

[0018] The names of messages or information exchanged between multiple devices in the embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of such messages or information.

[0019] This disclosure will now be described in detail with reference to the accompanying drawings and embodiments.

[0020] Figure 1 A flowchart 100 is shown, illustrating some embodiments of a heterogeneous data synchronization and collation collaborative method based on multi-objective feedback control according to this disclosure. This heterogeneous data synchronization and collation collaborative method based on multi-objective feedback control includes the following steps: Step 101: The producer thread delivers each change event collected from the relational database to the message queue and sends the message queue to the consumer thread.

[0021] In some embodiments, the producer thread delivers each change event collected from the relational database to a message queue and sends the message queue to the consumer thread. The relational database may include at least one relational data set. The relational data set may include a row-level identifier, a row-level version number, and relational row-level data. A change event can represent a change in the relational data stored in the relational database. The change event may include row-level change information. The row-level change information may include a row-level identifier, a row-level version number, and row-level change data. The message queue may include a set of row-level change information, and the row-level change information in the set may include a row-level identifier, a row-level version number, and row-level change data. In practice, firstly, the producer thread can encapsulate each change event collected from the relational database into a message. The message can represent row-level change information. Secondly, the producer thread can use a preset partition delivery algorithm to deliver each encapsulated message to a specified partition of the message queue. For example, the preset partition delivery algorithm could be: using the row-level identifier of the message as the primary key value and performing a hash modulo operation to deliver the message to a specified partition of the message queue. As an example, the message queue includes five partitions, and the row-level identifier of the message is "10". The preset partition delivery algorithm is: performing a hash modulo operation on the row-level identifier (10) of the message and the number of partitions (5) of the message queue to deliver the message to a specified partition (partition 0) of the message queue. Then, the producer thread can send each message from any partition in the message queue to any consumer thread. Thus, the data of one partition can be consumed by one consumer thread.

[0022] Here, a relational database can be a database used to store relational data. At least one relational data element within the relational database can represent a row of data stored in the relational database. The row-level identifier included in at least one relational data element can uniquely identify a row of data stored in the relational database, and the row-level version number can represent the version number of the current row of data. Relational row-level data can represent a row of data stored in the relational database. For example, relational row-level data can include, but is not limited to, at least one of the following: username, user address, and phone number.

[0023] Before performing any of the operations involving the collection, storage, or use of user personal information (such as username, user address, and telephone number) disclosed in this disclosure, the relevant organizations or individuals shall fulfill their obligations, including conducting personal information security impact assessments, informing the personal information subjects, and obtaining prior authorization and consent from the personal information subjects.

[0024] The row-level change information in the message queue can be formatted data representing changes to relational data stored in a relational database. The row-level identifier uniquely identifies a row of data stored in the relational database, the row-level version number represents the version number of the current row, and the row-level change data represents the data after the relational row-level data has been changed.

[0025] As an example, the row-level change information in the row-level change information set can be binlog data. The relational database can be a MySQL database.

[0026] Producer threads can be threads used to deliver changed data from a relational database to a message queue. Consumer threads can be threads used to synchronize, verify, and coordinate the data in the message queue.

[0027] Step 102: The producer thread determines the repair request information set based on the row-level version number information set stored in the pre-determined version number memory and the relational database, and sends the repair request information set to the consumer thread.

[0028] In some embodiments, the producer thread determines a repair request information set based on a pre-determined version number memory storing a set of row-level version numbers and the relational database, and sends the repair request information set to the consumer thread. Here, the version number memory can be a pre-determined memory for storing the set of row-level version numbers of the target data source. The row-level version number information in the set may include: a row-level identifier and a current version number. The target data source can be a server storing data written by the consumer thread. The target data source can store a consumer dataset. The consumer data in the consumer dataset may include: a consumer row-level dataset. The consumer row-level data in the consumer row-level dataset may include, but is not limited to: a row-level identifier, a current version number, and current consumer row-level data. The row-level identifier included in the consumer row-level data in the consumer row-level dataset can uniquely identify a consumer row-level data set, the current version number can represent the current version number of the consumer row-level data, and the current consumer row-level data can represent a row of data stored in the target data source. For example, the current consumer row-level data may include, but is not limited to, at least one of the following: username, user address, and phone number.

[0029] As an example, the target data source could be Elasticsearch (a search server).

[0030] In practice, the aforementioned producer thread can determine the repair request information set based on the row-level version number information set stored in a pre-determined version number memory and the aforementioned relational database through the following steps: The first step is to perform the following sub-steps for each row-level version number in the above row-level version number set: The first sub-step involves identifying the relational data in the aforementioned relational database that meets the preset row-level identifier conditions as the target relational data. The target relational data may include: a row-level identifier, a row-level version number, and relational row-level data. The preset row-level identifier condition may be that the row-level identifier included in the relational data is the same as the row-level identifier included in the row-level version number information.

[0031] The second sub-step involves determining that the row-level version number included in the target relational data and the current version number included in the row-level version number information meet a preset verification condition, and then identifying the target relational data as repair request information. The preset verification condition may be that the row-level version number included in the target relational data is different from the current version number included in the row-level version number information.

[0032] The second step is to define the identified repair request information as a repair request information set.

[0033] Step 103: In response to receiving the message queue and repair request information set sent by the producer thread, the consumer thread determines the initial ordinary queue and the initial high-priority queue.

[0034] In some embodiments, in response to receiving the message queue and the repair request information set sent by the producer thread, the consumer thread determines an initial ordinary queue and an initial high-priority queue. In practice, in response to receiving the message queue and the repair request information set sent by the producer thread, firstly, the consumer thread determines the message queue as the initial ordinary queue. Secondly, the consumer thread adds the repair request information set to the initial high-priority queue. The initial high-priority queue is initially empty.

[0035] Here, the aforementioned initial ordinary queue may include: an initial ordinary information set. The initial ordinary information in the initial ordinary information set may include: row-level identifier, row-level version number, and row-level change data. The aforementioned initial high-priority queue may include: an initial high-priority information set. The initial high-priority information in the initial high-priority information set may include: row-level identifier, row-level version number, and relational row-level data.

[0036] Step 104: The consumer thread updates the initial ordinary queue and the initial high-priority queue based on the row-level version number information set stored in the version number memory to generate the ordinary queue and the high-priority queue.

[0037] In some embodiments, the consumer thread updates the initial ordinary queue and the initial high-priority queue based on the row-level version number information set stored in the version number memory to generate the ordinary queue and the high-priority queue.

[0038] In practice, the aforementioned consumer thread can update the initial ordinary queue and the initial high-priority queue based on the row-level version number information set stored in the version number memory through the following steps to generate the ordinary queue and the high-priority queue: The first step, for each piece of initial general information included in the above initial general queue, is to perform the following first determination sub-step: The first sub-step involves identifying the row-level version number information that meets the preset ordinary row-level identifier conditions from the aforementioned row-level version number information set as the first row-level version number information. The preset ordinary row-level identifier conditions can be: the row-level identifier included in the row-level version number information is the same as the row-level identifier included in the aforementioned initial ordinary information.

[0039] The second sub-step involves deleting the initial ordinary information from the initial ordinary queue in response to determining that the row-level version number included in the first row-level version number information and the row-level version number included in the initial ordinary information satisfy a preset ordinary version condition. The preset ordinary version condition may be: the row-level version number included in the first row-level version number information is greater than or equal to the row-level version number included in the initial ordinary information.

[0040] The second step is to define the initial ordinary queue after deletion as an ordinary queue.

[0041] Third, for each initial high-priority information included in the above initial high-priority queue, perform the following second determination sub-step: The first sub-step involves identifying the row-level version numbers that satisfy a preset high-priority identifier from the aforementioned row-level version number information set as the second row-level version number information. The preset high-priority identifier may be that the row-level identifier included in the row-level version number information is the same as the row-level identifier included in the initial high-priority information.

[0042] The second sub-step involves deleting the initial high-priority information from the initial high-priority queue in response to determining that the row-level version number included in the second row-level version number information meets a preset high-priority version condition with respect to the row-level version number included in the initial high-priority information. The preset high-priority version condition may be that the row-level version number included in the second row-level version number information is greater than or equal to the row-level version number included in the initial high-priority information.

[0043] The fourth step is to determine the initial high-priority queue after deletion as a high-priority queue.

[0044] Therefore, a regular queue can represent messages whose row-level version numbers differ from those of data with the same row-level identifier in the message queue, as confirmed by the consumer thread. A high-priority queue can represent messages whose row-level version numbers differ from those of data with the same row-level identifier in the relational database, as confirmed by the producer thread.

[0045] Step 105: The consumer thread dynamically adjusts the batching window time and the scheduling weight of the ordinary queue based on the ordinary queue and the high-priority queue.

[0046] In some embodiments, the consumer thread dynamically adjusts the batching window time and the scheduling weight of the ordinary queue based on the ordinary queue and the high-priority queue.

[0047] In practice, the aforementioned consumer thread can dynamically adjust the batch processing window time and the scheduling weight of the ordinary queue based on the aforementioned ordinary queue and high-priority queue through the following steps: The first step involves collecting data such as the length of the ordinary queue, message arrival rate, version number lag value, target storage write latency, initial batching window time, and initial ordinary queue scheduling weight. In practice, firstly, the consumer thread can determine the length of the ordinary queue as the ordinary queue length. Secondly, the consumer thread can determine the message arrival rate as the extraction rate of the consumer thread. The extraction rate can be the rate at which the consumer thread extracts messages from the ordinary queue per unit time (e.g., 1 second). Next, the consumer thread can determine the largest initial version number lag value among the various initial version number lag values ​​corresponding to the high-priority queue as the version number lag value. The initial version number lag value can correspond to the collation row-level information in the collation row-level information set included in the high-priority queue. The initial version number lag value can be generated through the following steps: determining the difference between the row-level version number included in the collation row-level information and the current version number included in the row-level version number information as the initial version number lag value. Then, the aforementioned consumer thread can determine the target storage write latency as the time from writing data to the target data source to receiving a signal from the target data source (indicating successful writing). Next, it determines the preset batching time as the initial batching window time. Finally, it determines the preset queue weight as the initial normal queue scheduling weight. For example, the preset batching time could be 100ms, and the preset queue weight could be 5.

[0048] The second step involves updating the initial batching window time in response to the determination that the above-mentioned ordinary queue length, message arrival rate, version number lag value, and target storage write latency meet preset abnormal conditions, thereby generating a batching window time. The preset abnormal conditions may include: a first preset abnormal condition, a second preset abnormal condition, a third preset abnormal condition, a fourth preset abnormal condition, and a fifth preset abnormal condition. The first preset abnormal condition may be: the ordinary queue length is greater than a first preset ordinary queue length threshold. For example, the first preset ordinary queue length threshold may be 1000. The second preset abnormal condition may be: the ordinary queue length is less than a second preset ordinary queue length threshold. For example, the second preset ordinary queue length threshold may be 100. The third preset abnormal condition may be: the version number lag value is greater than a preset version number lag threshold. For example, the preset version number lag threshold may be 50. The fourth preset abnormal condition may be: the target storage write latency is greater than a preset storage write latency threshold. For example, the preset storage write latency threshold may be 100ms. The fifth preset abnormal condition may be: the message arrival rate is greater than a first message arrival rate, and the first message arrival rate is greater than a second message arrival rate. For example, the first message arrival rate could be the message arrival rate corresponding to the previous unit of time in the current unit of time. The second message arrival rate could be the message arrival rate corresponding to the unit of time in the previous unit of time in which the first message arrival rate corresponds.

[0049] In practice, the aforementioned consumer thread can update the initial batching window time in response to determining that the above-mentioned ordinary queue length, message arrival rate, version number lag value, and target storage write delay meet preset exception conditions through the following sub-steps, in order to generate the batching window time: The first sub-step involves determining the batching window time as the minimum of the first initial batching window time and the first preset batching window time, in response to the determination that the length of the normal queue meets the first preset exception condition. The first initial batching window time can be the product of the initial batching window time and the first exception weight. For example, the first exception weight can be 1.2. The first preset batching window time can be 1000ms.

[0050] The second sub-step, in response to determining that the length of the ordinary queue meets the second preset exception condition, determines the minimum value of the second initial batching window time and the second preset batching window time as the batching window time. The second initial batching window time can be the product of the initial batching window time and the second exception weight. For example, the second exception weight can be 0.9. The second preset batching window time can be 50ms.

[0051] The third sub-step, in response to determining that the aforementioned version number lag value meets the third preset anomaly condition, determines the maximum value of the third initial batching window time and the third preset batching window time as the batching window time. The third initial batching window time can be the product of the initial batching window time and the third anomaly weight. For example, the third anomaly weight can be 0.5. The third preset batching window time can be 50ms.

[0052] The fourth sub-step, in response to determining that the target storage write latency meets the fourth preset anomaly condition, determines the minimum value of the fourth initial batching window time and the fourth preset batching window time as the batching window time. The fourth initial batching window time can be the product of the initial batching window time and the fourth anomaly weight. For example, the fourth anomaly weight can be 1.3. The fourth preset batching window time can be 2000ms.

[0053] The fifth sub-step, in response to determining that the message arrival rate meets the fifth preset anomaly condition, determines the minimum value of the fifth initial batching window time and the fifth preset batching window time as the batching window time. The fifth initial batching window time can be the product of the initial batching window time and the fifth anomaly weight. For example, the fifth anomaly weight can be 1.1. The fifth preset batching window time can be 1000ms.

[0054] The third step involves updating the initial ordinary queue scheduling weights in response to the determination that the length of the ordinary queue meets the preset queue anomaly conditions, thereby generating ordinary queue scheduling weights. The preset queue anomaly conditions may include a first preset queue anomaly condition and a second preset queue anomaly condition. The first preset queue anomaly condition may be that the length of the ordinary queue is greater than a first queue anomaly threshold. For example, the first queue anomaly threshold may be 1000. The second preset queue anomaly condition may be that the length of the ordinary queue is less than a second queue anomaly threshold. For example, the second queue anomaly threshold may be 100.

[0055] In practice, the aforementioned consumer thread can update the initial ordinary queue scheduling weight in response to determining that the length of the ordinary queue meets the preset queue exception condition through the following sub-steps, in order to generate the ordinary queue scheduling weight: The first sub-step, in response to determining that the above-mentioned ordinary queue length meets the first preset queue exception condition, determines the maximum value of the first initial ordinary queue scheduling weight and the first preset ordinary queue scheduling weight as the ordinary queue scheduling weight. The first initial ordinary queue scheduling weight can be the difference between the initial ordinary queue scheduling weight and the first preset ordinary weight. For example, the first preset ordinary weight can be 1.

[0056] The second sub-step, in response to determining that the above-mentioned ordinary queue length meets the second preset queue exception condition, determines the ordinary queue scheduling weight as the minimum value between the second initial ordinary queue scheduling weight and the second preset ordinary queue scheduling weight. The second initial ordinary queue scheduling weight can be the difference between the initial ordinary queue scheduling weight and the second preset ordinary weight. For example, the second preset ordinary weight can be 1. The second preset ordinary queue scheduling weight can be 20.

[0057] Therefore, through multi-objective feedback control, the batching window time and the scheduling weight of ordinary queues can be dynamically adjusted: when queue backlog is severe, the batching window is automatically increased and the weight of ordinary queues is reduced; when version numbers are severely lagging, the verification priority is forcibly increased and the batching window is reduced; when target storage latency increases, write concurrency is actively limited. This achieves adaptive optimization: multi-objective feedback control automatically balances write pressure, synchronization latency, and verification latency without requiring manual parameter tuning.

[0058] Step 106: The consumer thread collects the batch message set based on the batch window time, the scheduling weight of the ordinary queue, and the ordinary queue and the high priority queue.

[0059] In some embodiments, the consumer thread collects a batch message set based on the batching window time, the ordinary queue scheduling weight, the ordinary queue, and the high-priority queue. The batch message set may include: a row-level identifier, a row-level version number, and batch row-level data. In practice, firstly, in response to determining that the high-priority queue is empty, a batch message set corresponding to the ordinary queue is collected based on the batching window time and the ordinary queue scheduling weight, and a first preset collection identifier is determined as the collection identifier corresponding to the batch message set. Secondly, in response to determining that the high-priority queue is not empty, each proofreading row-level information included in the high-priority queue is determined as a batch message, resulting in a batch message set, and a second preset collection identifier is determined as the collection identifier corresponding to the batch message set. Here, collecting the batch message set corresponding to the ordinary queue based on the batching window time and the ordinary queue scheduling weight can be: collecting the number of ordinary proofreading row-level information items included in the ordinary queue within the batching window time, up to the number of ordinary queue scheduling weights, as the batch message set. Here, the batch-level messages included in the batch message set can represent: relational row-level data included in the collation row-level information of the high-priority queue. The batch-level messages included in the batch message set can also represent: row-level change data included in the ordinary collation row-level information of the ordinary queue. For example, a first preset collection identifier can represent that the batch message set was collected through the ordinary queue. A second preset collection identifier can represent that the batch message set was collected through the high-priority queue.

[0060] Step 107: The consumer thread performs deduplication processing on the batch message set to generate a deduplicated batch message set.

[0061] In some embodiments, the consumer thread performs deduplication processing on the batch message set to generate a deduplicated batch message set.

[0062] In practice, the aforementioned consumer thread can perform deduplication processing on the batch message set using the following steps to generate a deduplicated batch message set: The first step is to classify the aforementioned batch message set to generate batch message group sets. In practice, the consumer thread can use a preset classification algorithm to classify the batch message set to generate batch message group sets. For example, the preset classification algorithm could be to identify batch messages with the same row-level identifier as batch message groups.

[0063] The second step involves identifying the batch-deduplicated message that meets the preset maximum version condition within each batch-deduplicated message group set. The preset maximum version condition can be that the batch-deduplicated message is the one with the highest row-level version number included in the batch-deduplicated message group.

[0064] The third step is to define each batch deduplication message as a batch deduplication message set.

[0065] Step 108: The consumer thread updates the row-level version number information set stored in the ordinary queue, high-priority queue, version number storage and the target data source based on the batch deduplication message set.

[0066] In some embodiments, the consumer thread updates the ordinary queue, the high-priority queue, the row-level version number information set stored in the version number memory, and the target data source based on the batch deduplication message set.

[0067] In practice, the aforementioned consumer thread can update the aforementioned ordinary queue, the aforementioned high-priority queue, the row-level version number information set stored in the aforementioned version number storage, and the aforementioned target data source based on the aforementioned batch deduplication message set through the following steps: The first step involves sending the batch deduplication message set to the target data source to write the batch deduplication message set to the target data source and receiving a write completion signal from the target data source. The write completion signal can be a signal indicating that the target data source has successfully written the batch deduplication message set. In practice, writing the batch deduplication message set to the target data source can be achieved by the consumer thread sending the batch deduplication message set to the target data source. Upon receiving the batch deduplication message set from the consumer thread, the target data source, for each batch deduplication message in the set, updates the consumer data in its stored consumer dataset corresponding to the batch deduplication message with the corresponding batch deduplication message. Here, the consumer data in the consumer dataset corresponding to the batch deduplication message can be consumer row-level data whose row-level identifiers are the same as those in the batch deduplication message.

[0068] The second step involves updating the row-level version number information set stored in the ordinary queue, the high-priority queue, and the version number memory based on the batch deduplication message set, in response to receiving the write completion signal from the target data source. In practice, the execution entity can update the row-level version number information set in the ordinary queue, the high-priority queue, and the version number memory through the following sub-steps in response to receiving the write completion signal from the target data source and based on the batch deduplication message set: The first sub-step involves sending the aforementioned batch deduplication message set to the aforementioned version number memory to update the row-level version number information set stored in the version number memory, and receiving an update completion signal from the version number memory. The update completion signal indicates that the version number memory has completed updating the row-level version number information set. Here, in response to receiving the batch deduplication message set, for each batch deduplication message in the batch deduplication message set, the version number memory updates the current version number included in the row-level version number information set corresponding to the aforementioned batch deduplication message to the row-level version number included in the aforementioned batch deduplication message, thereby updating the row-level version number information set stored in the version number memory. Here, the row-level version number information in the row-level version number information set corresponding to the aforementioned batch deduplication message can be: row-level version number information where the row-level identifier included in the row-level version number information set is the same as the row-level identifier included in the batch deduplication message.

[0069] The second sub-step involves, in response to receiving update completion information from the version number memory and in response to determining that the batch deduplication message set meets a preset general condition, deleting the general verification row-level information corresponding to the batch deduplication message from the general queue for each batch deduplication message in the batch deduplication message set, thereby updating the general queue. The preset general condition may be that the acquisition identifier corresponding to the batch deduplication message set is a first preset acquisition identifier. The general verification row-level information corresponding to the batch deduplication message in the general queue may be general verification row-level information whose row-level identifiers are the same as those in the batch deduplication message.

[0070] The third sub-step involves, in response to receiving update completion information from the version number storage and in response to determining that the batch deduplication message set does not meet preset general conditions, deleting the collation row-level information corresponding to the batch deduplication message from the high-priority queue for each batch deduplication message in the batch deduplication message set, thereby updating the high-priority queue. The collation row-level information corresponding to the batch deduplication message in the high-priority queue can be: collation row-level information in the high-priority queue whose row-level identifier is the same as that in the batch deduplication message.

[0071] Optionally, the consumer thread can also use the updated ordinary queue, the updated high-priority queue, and the updated row-level version number information set as the ordinary queue, the high-priority queue, and the row-level version number information set, respectively, for the purpose of performing the above dynamic adjustment steps again.

[0072] As an example, the timing flowchart of the consumer thread is as follows: Figure 2 As shown, Figure 2A timing flowchart of a consumer thread is shown, representing some embodiments of the heterogeneous data synchronization and calibration coordination method based on multi-objective feedback control according to the present disclosure.

[0073] The above description is merely a selection of preferred embodiments of this disclosure and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of the invention involved in the embodiments of this disclosure is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-described inventive concept. For example, technical solutions formed by substituting the above-described features with (but not limited to) technical features with similar functions disclosed in the embodiments of this disclosure.

Claims

1. A heterogeneous data synchronization and collaborative method based on multi-objective feedback control, characterized in that, include: The producer thread delivers each change event collected from the relational database to a message queue, and sends the message queue to the consumer thread; The producer thread determines a repair request information set based on a pre-determined set of row-level version number information stored in a version number memory and the relational database, and sends the repair request information set to the consumer thread. In response to receiving the message queue and the repair request information set sent by the producer thread, the consumer thread determines the initial ordinary queue and the initial high-priority queue. The consumer thread updates the initial ordinary queue and the initial high-priority queue based on the row-level version number information set stored in the version number memory, so as to generate the ordinary queue and the high-priority queue. The consumer thread dynamically adjusts the batch processing window time and the scheduling weight of the ordinary queue based on the ordinary queue and the high-priority queue. The consumer thread collects the batch message set based on the batch window time, the scheduling weight of the ordinary queue, the ordinary queue, and the high-priority queue; The consumer thread performs deduplication processing on the batch message set to generate a deduplicated batch message set. The consumer thread updates the ordinary queue, the high-priority queue, the row-level version number information set stored in the version number storage, and the target data source based on the batch deduplication message set.

2. The heterogeneous data synchronization and collaborative method based on multi-objective feedback control according to claim 1, characterized in that, The relational database includes: at least one relational data, and the relational data in the at least one relational data includes: row-level identifier and row-level version number, and the row-level version number information in the row-level version number information set includes: row-level identifier and current version number; And the set of row-level version number information stored in the predetermined version number memory and the relational database, to determine the set of repair request information, including: For each row-level version number in the row-level version number set, perform the following processing steps: The relational data in the relational database that meets the preset row-level identifier condition is identified as the target relational data. The target relational data includes: row-level identifier and row-level version number. The preset row-level identifier condition is: the row-level identifier included in the relational data is the same as the row-level identifier included in the row-level version number information. In response to determining that the row-level version number included in the target relational data and the current version number included in the row-level version number information meet the preset verification conditions, the target relational data is determined as repair request information; The identified repair request information is defined as a repair request information set.

3. The heterogeneous data synchronization and collaborative method based on multi-objective feedback control according to claim 2, characterized in that, The initial ordinary queue includes: an initial ordinary information set, the initial ordinary information in the initial ordinary information set includes: row-level identifier and row-level version number; the initial high-priority queue includes: an initial high-priority information set, the initial high-priority information in the initial high-priority information set includes: row-level identifier and row-level version number. The process of updating the initial ordinary queue and the initial high-priority queue based on the row-level version number information set stored in the version number memory to generate ordinary queues and high-priority queues includes: For each piece of initial general information included in the initial general queue, the following first determination step is performed: The row-level version number information that meets the preset ordinary row-level identifier condition in the row-level version number information set is determined as the first row-level version number information. The preset ordinary row-level identifier condition is that the row-level identifier included in the row-level version number information is the same as the row-level identifier included in the initial ordinary information. In response to determining that the row-level version number included in the first row-level version number information and the row-level version number included in the initial ordinary information meet a preset ordinary version condition, the initial ordinary information is deleted from the initial ordinary queue. The initial ordinary queue after deletion is designated as an ordinary queue; For each initial high-priority piece of information included in the initial high-priority queue, the following second determination step is performed: The row-level version number information that satisfies the preset high-priority identification condition in the row-level version number information set is determined as the second row-level version number information. The preset high-priority identification condition is that the row-level identifier included in the row-level version number information is the same as the row-level identifier included in the initial high-priority information. In response to determining that the row-level version number included in the second row-level version number information and the row-level version number included in the initial high-priority information meet the preset high-priority version condition, the initial high-priority information is deleted from the initial high-priority queue. The initial high-priority queue after deletion is designated as a high-priority queue.

4. The heterogeneous data synchronization and collaborative method based on multi-objective feedback control according to claim 1, characterized in that, The dynamic adjustment of the batching window time and the scheduling weight of the ordinary queue based on the ordinary queue and the high-priority queue includes: Collect data such as ordinary queue length, message arrival rate, version number lag value, target storage write latency, initial batch window time, and initial ordinary queue scheduling weight. In response to determining that the length of the normal queue, the message arrival rate, the version number lag value, and the target storage write delay meet preset abnormal conditions, the initial batching window time is updated to generate a batching window time. In response to determining that the length of the ordinary queue meets the preset queue exception conditions, the initial ordinary queue scheduling weight is updated to generate an ordinary queue scheduling weight.

5. The heterogeneous data synchronization and collaborative method based on multi-objective feedback control according to claim 1, characterized in that, The step of deduplicating the batch message set to generate a deduplicated batch message set includes: The batch message set is classified to generate a batch classified message group set; For each batch category message group in the batch category message group set, the batch category message that meets the preset maximum version condition in the batch category message group is determined as the batch deduplication message. The identified batch deduplication messages are defined as a batch deduplication message set.

6. The heterogeneous data synchronization and collaborative method based on multi-objective feedback control according to claim 1, characterized in that, The step of updating the ordinary queue, the high-priority queue, the row-level version number information set stored in the version number storage, and the target data source based on the batch deduplication message set includes: Send the batch deduplication message set to the target data source to write the batch deduplication message set to the target data source, and receive the write completion signal sent by the target data source; In response to receiving a write completion signal from the target data source, the row-level version number information set stored in the ordinary queue, the high-priority queue, and the version number memory is updated based on the batch deduplication message set.

7. The heterogeneous data synchronization and collaborative method based on multi-objective feedback control according to claim 1, characterized in that, The method further includes: The consumer thread uses the updated ordinary queue, the updated high-priority queue, and the updated row-level version number information set as the ordinary queue, the high-priority queue, and the row-level version number information set, respectively, for re-execution of the dynamic adjustment step.

Images