Data synchronization method, system, and storage medium

Abstract

The application discloses a data synchronization method and system and a storage medium, relates to the technical field of data synchronization, and comprises the following steps: acquiring a current error value between a current data synchronization delay and a standard data synchronization delay determined based on an observation index, and determining an accumulated error value and an error value change amount based on the current error value; calculating a data synchronization interval adjustment amount according to the current error value, the accumulated error value and the error value change amount; adjusting a current data synchronization interval according to the data synchronization interval adjustment amount to generate a target data synchronization interval; and transmitting to-be-synchronized data in a source database to a target database in the target data synchronization interval. The application adjusts the data synchronization interval required by the system by using multi-dimensional observation indexes, and solves the problem that a fixed data synchronization interval cannot balance the real-time performance and load balancing of data synchronization.

Description

Technical Field

[0001] This application relates to the field of data synchronization technology, and in particular to data synchronization methods, systems and storage media. Background Technology

[0002] In near real-time bidirectional data synchronization scenarios for databases, the synchronization strategy employed by the relevant technologies is fixed, resulting in a fixed data synchronization interval. If the data synchronization interval is too short, it increases the burden on resources and the database, leading to delayed data synchronization or resource exhaustion; if the data synchronization interval is too long, it causes data synchronization delays, affecting database consistency. Therefore, a fixed data synchronization interval cannot balance the real-time performance of data synchronization with load balancing.

[0003] The above content is only used to help understand the technical solution of this application and does not represent an admission that the above content is prior art. Summary of the Invention

[0004] The main purpose of this application is to provide a data synchronization method, system, and storage medium, which aims to solve the technical problem that a fixed data synchronization interval cannot balance the real-time performance and load balancing of data synchronization.

[0005] To achieve the above objectives, this application proposes a data synchronization method, the steps of which include: Obtain the current error value between the current data synchronization delay determined based on the observation index and the standard data synchronization delay, and determine the cumulative error value and the change in error value based on the current error value; Based on the current error value, the cumulative error value, and the change in the error value, the adjustment amount of the data synchronization interval for the current data synchronization delay is calculated. The current data synchronization interval is adjusted according to the data synchronization interval adjustment amount to generate the target data synchronization interval; The data to be synchronized in the source database is transmitted to the target database at the target data synchronization interval.

[0006] For example, the step of calculating the data synchronization interval adjustment amount for the current data synchronization delay based on the current error value, the cumulative error value, and the change in the error value includes: Determine the first weight value corresponding to the current error value, the cumulative error value, and the change in the error value, respectively; The current error value, the cumulative error value, and the change in the error value are weighted and fused according to the first weight value, and the result is used as the data synchronization interval adjustment amount.

[0007] For example, the step of obtaining the current error value between the current data synchronization delay determined based on the observation index and the standard data synchronization delay includes: The data synchronization latency, CPU utilization, and task queue length are determined based on the observed indicators. The data synchronization latency, CPU utilization, and task queue length are normalized to determine the normalized values ​​corresponding to each observation indicator. The data synchronization delay, CPU utilization, and task queue length are respectively weighted and fused according to the second weight values ​​and the normalized values, and the calculated result is used as the current data synchronization delay. The current error value is determined based on the difference between the current data synchronization delay and the standard data synchronization delay.

[0008] For example, after the step of adjusting the current data synchronization interval according to the data synchronization interval adjustment amount to generate the target data synchronization interval, the method further includes: When the target data synchronization interval exceeds the first preset data synchronization interval, the target data synchronization interval is updated to the first preset data synchronization interval. When the target data synchronization interval is less than the second preset data synchronization interval, the target data synchronization interval is updated to the second preset data synchronization interval; When the target data synchronization interval is greater than or equal to the second preset data synchronization interval and less than or equal to the first preset data synchronization interval, the step of synchronizing the data to be synchronized to the target database at the target data synchronization interval is executed. The second preset data synchronization interval is less than the first preset data synchronization interval.

[0009] For example, before the step of obtaining the current error value between the current data synchronization delay determined based on the observation index and the standard data synchronization delay, the method further includes: If operation description information targeting the source database is detected, the operation description information is converted into operation description information in a preset format. The operation intent information is obtained based on the operation description information in the preset format; The data to be synchronized is determined based on the aforementioned operational intent information; A data synchronization task is generated based on the data to be synchronized and stored in the task queue.

[0010] For example, the step of generating a data synchronization task based on the data to be synchronized and storing it in a task queue includes: Obtain the execution status corresponding to the data to be synchronized; When the execution state is the first execution state, the step of generating a data synchronization task based on the data to be synchronized and storing it in the task queue is performed.

[0011] For example, after the step of obtaining the execution status corresponding to the data to be synchronized, the method further includes: When the execution state is the second execution state, discard the data to be synchronized, and return to the step of detecting the existence of operation description information for the source database and converting the operation description information into operation description information in a preset format; When the execution state is the third execution state, the step of generating a data synchronization task based on the data to be synchronized and storing it in the task queue is performed.

[0012] For example, the step of adjusting the current data synchronization interval according to the data synchronization interval adjustment amount to generate the target data synchronization interval includes: A composite health index is generated based on the observed indicators; When the composite health index is greater than or equal to the first composite health index, the result of adding the data synchronization interval adjustment amount and the current data synchronization interval is taken as the target data synchronization interval; When the composite health index is less than the first composite health index, the result of adding the data synchronization interval adjustment amount and the current data synchronization interval is used as the initial data synchronization interval, and the initial data synchronization interval with a preset multiple is used as the target data synchronization interval. When the composite health index is less than the second composite health index, the step of adjusting the current data synchronization interval according to the data synchronization interval adjustment amount to generate the target data synchronization interval is paused, and the second composite health index is less than the first composite health index.

[0013] In addition, to achieve the above objectives, this application also proposes a data synchronization system, which includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the data synchronization method described above.

[0014] In addition, to achieve the above objectives, this application also proposes a data synchronization device, which includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the data synchronization method described above.

[0015] In addition, to achieve the above objectives, this application also proposes a storage medium, which is a computer-readable storage medium, on which a computer program is stored, and which, when executed by a processor, implements the steps of the data synchronization method described above.

[0016] In addition, to achieve the above objectives, this application also provides a computer program product, which includes a computer program that, when executed by a processor, implements the steps of the data synchronization method described above.

[0017] The present application proposes one or more technical solutions that have at least the following technical effects: The current data synchronization delay is calculated by monitoring the observed indicators; the current error value between the current data synchronization delay and the standard data synchronization delay is determined; the current error value is used to evaluate the system synchronization state when data synchronization is performed at the current data synchronization interval; the current error value is then input into a PID (Proportional-Integral-Derivative) model, which determines the cumulative error value and the change in error value based on the current error value; the data synchronization interval adjustment is then calculated based on the current error value, the cumulative error value, and the change in error value; and the target data synchronization interval required for the current system synchronization state is determined based on the data synchronization interval adjustment. When data synchronization is performed at the calculated target data synchronization interval, the contradiction between data consistency and system balance is balanced. The present application dynamically adjusts the data synchronization interval based on the system's data synchronization state, solving the problem of excessively high synchronization delay or system resource exhaustion caused by a fixed data synchronization interval. Attached Figure Description

[0018] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0019] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0020] Figure 1 This is a schematic diagram of the module structure of the data synchronization system in the first embodiment of the data synchronization method of this application; Figure 2 This is a diagram illustrating the computational principle architecture of the PID model in the first embodiment of the data synchronization method of this application. Figure 3This is a flowchart illustrating the data synchronization method of the second embodiment of the data synchronization method of this application; Figure 4 This is a detailed flowchart of step S10 in the second embodiment of the data synchronization method of this application; Figure 5 This is a detailed flowchart of step S20 in the second embodiment of the data synchronization method of this application; Figure 6 This is a flowchart illustrating the data synchronization method of the second embodiment of the data synchronization method of this application; Figure 7 This is a flowchart of the data synchronization method in the third embodiment of the data synchronization method of this application; Figure 8 This is a detailed flowchart of step S80 in the third embodiment of the data synchronization method of this application; Figure 9 This is a schematic diagram of the device structure of the hardware operating environment involved in the data synchronization method in this application embodiment.

[0021] The purpose, features, and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0022] It should be understood that the specific embodiments described herein are merely illustrative of the technical solutions of this application and are not intended to limit this application.

[0023] In related technologies, during near real-time bidirectional data synchronization of databases, the synchronization strategy employed is fixed, resulting in a fixed data synchronization interval. When the data synchronization interval is too short, it increases the burden on resources and the database, leading to delayed data synchronization or resource exhaustion; conversely, when the data synchronization interval is too long, it causes data synchronization delays, affecting database consistency. Therefore, a fixed data synchronization interval cannot balance the conflict between the real-time nature of data synchronization and load balancing.

[0024] This application provides a solution that calculates the current error value between the current data synchronization delay and the standard data synchronization delay, and obtains the cumulative error value and the change in error value based on the current error value. A PID controller performs proportional, integral, and derivative weighted calculations on the above error to dynamically generate a data synchronization interval adjustment amount, thereby adjusting to obtain the target data synchronization interval. Thus, the system automatically shortens the data synchronization interval to ensure real-time performance when the data synchronization delay is high, and automatically extends the data synchronization interval to reduce resource consumption when the system load is high, achieving a dynamic adaptive balance between real-time data synchronization and system load balancing.

[0025] To better understand the technical solution of this application, a detailed description will be provided below in conjunction with the accompanying drawings and specific implementation methods.

[0026] First Embodiment Reference Figure 1 , Figure 1 This application provides a data synchronization system 10, which includes: The dynamic load monitoring engine 11 is used to obtain the current error value between the current data synchronization delay determined based on the observed indicators and the standard data synchronization delay; The data synchronization interval adjustment calculation module 12 is used to determine the cumulative error value and the change in error value based on the current error value, and to calculate the data synchronization interval adjustment amount for the current data synchronization delay based on the current error value, the cumulative error value and the change in error value. The data synchronization module 13 adjusts the current data synchronization interval according to the data synchronization interval adjustment amount, generates a target data synchronization interval, and transmits the data to be synchronized from the source database to the target database at the target data synchronization interval.

[0027] In this embodiment, the data synchronization system 10 is communicatively connected to the source database and the target database, respectively, and transmits the data to be synchronized from the source database to the target database at a target data synchronization interval. The target data synchronization interval is determined based on multi-dimensional data such as the current system load status, task queue length, and data synchronization latency. When data synchronization is performed at the target data synchronization interval, a dynamic adaptive balance between real-time data synchronization and system load balancing can be achieved.

[0028] In one optional implementation, the data synchronization system 10 includes a dynamic load monitoring engine 11. The dynamic load monitoring engine 11 is used to acquire observation indicators and determine the current data synchronization delay of the data synchronization system 10 based on the observation indicators. Then, it determines the current error value based on the current data synchronization delay and the standard data synchronization delay. The current error value is used to characterize the degree of deviation between the current state of the data synchronization system 10 and the standard state. The standard state is that the current data synchronization delay of the data synchronization system 10 tends to zero and the current load is balanced. If the current error value is larger, the current data synchronization delay is too high or the current load is too high. If the current data synchronization delay is too high, the data synchronization interval needs to be reduced. If the current load is too high, the data synchronization interval needs to be increased.

[0029] It should be noted that the observed metrics include, but are not limited to, data synchronization latency, CPU utilization, and task queue length. Data synchronization latency is the time interval between the times the same data is written to the source database and the target database. For example, if the data is written to the source database at 14:42:18 and synchronized to the target database at 14:42:18, then the data synchronization latency is 1 minute. CPU utilization quantifies the current computing resource load of the system and is a core indicator reflecting whether the system is under high load or overload. When CPU utilization exceeds a threshold, it indicates that system resources are strained and require further attention. To reduce the synchronization frequency and increase the synchronization interval, it is necessary to avoid further increasing the load and causing system lag or crashes. The task queue length refers to the total number of data synchronization tasks currently waiting to be executed. It is used to quantitatively characterize the backlog of data synchronization and the system processing pressure, and is a core observation indicator for judging whether synchronization tasks are congested and whether the synchronization frequency needs to be adjusted. The longer the task queue length, the more data is waiting to be synchronized, the more the system cannot process it, and the greater the synchronization pressure. That is, when the task queue is too long, it means that the synchronization frequency is too high and the tasks are generated too quickly, and the system cannot execute the synchronization tasks in time. It is necessary to increase the synchronization interval and reduce the synchronization frequency to alleviate the queue backlog. It is understood that this application comprehensively evaluates the real-time performance of data synchronization and the resource load status of the system based on data synchronization latency, CPU utilization, and task queue length, providing a basis for dynamically adjusting the data synchronization interval and achieving an adaptive balance between data synchronization real-time performance and system load balancing.

[0030] In another optional implementation, the data synchronization interval adjustment calculation module 12 includes a PID model 121. The PID model 121 is used to calculate the data synchronization interval adjustment required by the data synchronization system 10 based on the current error value. Specifically, the PID model 121 determines the cumulative error value and the change in error value based on the current error value, and then calculates the data synchronization interval adjustment for the current data synchronization delay based on the current error value, the cumulative error value, and the change in error value. The calculation formula for calculating the data synchronization interval adjustment is as follows:

[0031] in, T is the data synchronization interval adjustment amount, Kp, Ki, and Kd are the first weight values ​​corresponding to the current error value, cumulative error value, and error value change, respectively, e(t) is the current error value, ∑(t) is the cumulative error value, and e(t-1) is the error value corresponding to the previous acquisition time.

[0032] It should be noted that, referring to Figure 2 , Figure 2The calculation principle architecture diagram of PID model 121 is shown. Specifically, the input to PID model 121 is the current data synchronization delay and the standard data synchronization delay, so that PID model 121 can calculate the data synchronization interval adjustment amount based on the current data synchronization delay and the standard data synchronization delay using proportional, integral and proportional components. The output of PID model 121 is the data synchronization interval adjustment amount. Specifically, the PID model 121 includes a proportional, integral, and derivative component. After inputting the current error value, different components are called to calculate the corresponding results. The data synchronization interval adjustment is generated by weighted fusion of the results generated by each component. The proportional component directly responds to the current error value, which is determined by states such as CPU utilization exceeding a threshold or a sudden increase in synchronization delay. By directly increasing the data synchronization interval, immediate intervention in abnormal system states is achieved. The integral component is used to accumulate historical deviations throughout the entire cycle to determine the cumulative error value. The cumulative error value is determined by states such as continuous backlog of historical deviation task queues and long-term high load. The cumulative error value is obtained by accumulating the data synchronization delay corresponding to each data synchronization task. The integral component gradually eliminates the steady-state residual error of the system through cumulative adjustment, making the system conform to the ideal operating indicators. The derivative component predicts the dynamic trend of error changes (such as accelerated increase in synchronization delay and continuous increase in load). Through advance damping adjustment, it suppresses system overshoot and oscillation in advance, ensuring the overall stability of the system.

[0033] It is understandable that the data synchronization interval adjustment calculated by PID model 121 can be a positive value. When the data synchronization interval adjustment is positive, the data synchronization interval is increased and the data synchronization frequency is reduced to relieve the CPU load pressure of the system. When the data synchronization interval adjustment is negative, the data synchronization interval is reduced and the data synchronization frequency is increased to reduce the data synchronization delay and ensure real-time performance.

[0034] It should be noted that the PID model 121 also includes a closed-loop feedback loop. After calculating the data synchronization interval adjustment amount, the target operating state of the system is determined based on the data synchronization interval adjustment amount. The target operating state includes the actual data synchronization delay of the data synchronization system 10 after adjusting the data synchronization interval. The actual synchronization delay includes the actual data synchronization delay, the actual CPU utilization, and the actual task queue length. The closed-loop feedback loop transmits the actual operating state of the data synchronization system 10 (including the actual data synchronization delay, the actual CPU utilization, and the actual task queue length) back to the input end as the actual input basis for comparison with the standard data synchronization delay, so as to recalculate the data synchronization interval adjustment amount until the data synchronization system 10 meets the stable operation requirements.

[0035] It should be noted that the relevant technologies for capturing data changes employ methods such as triggers, log parsing, and application-layer dual-write. During implementation, it was found that adding triggers to the data synchronization system 10 results in high execution overhead due to triggers being non-lightweight operations, impacting the performance of the data synchronization system 10. Log parsing, on the other hand, is logically complex, cumbersome to configure, lacks versatility, has high implementation costs, and is difficult to adapt to different databases and ORM frameworks. Using application-layer dual-write for data change capture requires embedding synchronization logic code into the business code, leading to high invasiveness and significant modification requirements. Therefore, this application embodiment includes a data change monitoring module 14, which silently captures data change events in real time.

[0036] In an optional implementation, the data synchronization system 10 further includes a data change monitoring module 14, which monitors whether data changes have occurred in the source database. The data change monitoring module includes an ORM intent recognition component 141, which comprises a unified SQL interception layer, a unified SQL parsing layer, and a multi-ORM framework adaptation plugin layer. In this configuration, the data synchronization system 10 does not require modification of the business code; it intercepts data operations solely through native framework extension points, thereby accurately detecting and capturing data change events.

[0037] In one optional implementation, upon system startup, the multi-ORM framework adapter plugin layer automatically scans and loads the plugins corresponding to the ORM frameworks used for the current data operations based on configuration files or the Spring container. For example, if the ORM framework used for database operations is MyBatis, the MyBatis adapter plugin is loaded; if JPA / Hibernate is currently used, the JPA adapter plugin is loaded. Each plugin then completes the registration and initialization of its own interceptors.

[0038] Furthermore, upon detecting a data change operation targeting the source database, a unified SQL interception layer is used to obtain the operation description information for this data change operation. This unified SQL interception layer silently captures the operation description information without intruding on the business logic. The operation description information includes, but is not limited to, the operation type (e.g., INSERT, UPDATE, or DELETE); the operation object (e.g., the name of the data table involved (or the table name corresponding to the entity class name); the operation parameters (e.g., the specific data value or primary key ID after the change); and the transaction context (e.g., obtaining the transaction status of the current thread and binding the transaction ID through the Spring transaction synchronization manager).

[0039] Furthermore, after capturing the operation description information, it is sent to a unified SQL parsing layer. This unified SQL parsing layer is responsible for standardizing and uniformly parsing the heterogeneous operation description information captured by different ORM frameworks. Specifically, it converts the operation description information into operation description information in a preset format, and then generates operation intent information in a preset format based on the operation description information in the preset format. The determined operation intent information includes standardized table names, operation types, primary key values, and modified data. For example, in the current ORM framework MyBatis, the native SQL string of MyBatis is parsed into an abstract syntax tree (AST), and the table name, condition fields, and values ​​are extracted. In the current ORM framework JPA / Hibernate, the entity state changes in JPA / Hibernate are mapped to standard data change entities.

[0040] In an optional implementation, the data change monitoring module further includes a Spring transaction synchronization manager 142, which is communicatively connected to the ORM intent recognition component 141. When a data change is detected, i.e., when there is data to be synchronized, a data synchronization task is not directly generated. Instead, the Spring transaction synchronization manager 142 performs execution status detection based on the transaction execution status corresponding to the data to be synchronized, and generates a data synchronization task based on the detection result. Specifically, when the execution status is the first execution status, which is the transaction commit status, the current data change operation is confirmed to be effective, and the data to be synchronized is stored in the task queue, waiting for subsequent synchronization. When the execution status is the second execution status, which is the transaction rollback status, the data to be synchronized is discarded, and no synchronization task is generated, ensuring the consistency of data synchronization. When the execution status is the third execution status, which is the no-transaction status, a synchronization task is directly generated based on the data to be synchronized and stored in the task queue.

[0041] It is understood that the data synchronization system 10 in this embodiment of the application constructs a unified ORM intent recognition component 141 and combines the ORM intent recognition component 141 with the Spring transaction synchronization manager 142 to capture data change events in real time. Based on the execution status corresponding to the data change event, a corresponding data synchronization task is generated. The ORM intent recognition component 141 supports pluggable adaptation to multiple ORM frameworks, reducing development costs under different ORM framework environments. At the same time, by using SQL interceptors to intercept data change events, data changes can be captured without using triggers, parsing database logs, or modifying business code, reducing system overhead and improving data capture efficiency.

[0042] Furthermore, after determining the data synchronization interval adjustment amount, the data synchronization module 13 obtains the target data synchronization interval based on the current data synchronization interval and the data synchronization interval adjustment amount, and then obtains the data to be synchronized from the source database at the target data synchronization interval and sends the data to be synchronized to the target database, so as to achieve data synchronization between the source database and the target database while ensuring the load balance of the system.

[0043] In this embodiment, a unified SQL interception layer is constructed to achieve silent capture of data change events without the need for additional triggers, database log parsing, or additional code, thus improving the efficiency of data change event capture. Simultaneously, a unified SQL parsing layer is constructed to parse operation description information under different ORM frameworks, supporting pluggable adaptation to different ORM frameworks, reducing development costs in multi-ORM environments, and improving the versatility of the data synchronization system 10. When a data change event is detected, the execution status of the data to be synchronized is detected in conjunction with the Spring transaction synchronization manager 142 to generate a corresponding data synchronization task. The entire data capture process is asynchronous and decoupled, avoiding blocking of business logic. When there are tasks to be synchronized in the task queue, the dynamic load monitoring engine 11 is used to monitor the running status of the data synchronization system 10 in real time, and the data synchronization interval is dynamically adjusted through multi-dimensional observation indicators to achieve an adaptive balance between data synchronization real-time performance and system load balancing.

[0044] Second Embodiment Reference Figure 3 Based on the first embodiment, this application proposes a data synchronization method, which includes S10~S40: Step S10: Obtain the current error value between the current data synchronization delay determined based on the observation index and the standard data synchronization delay, and determine the cumulative error value and the change in error value based on the current error value; In this application embodiment, the data synchronization system described in the first embodiment is applied. The data synchronization system adjusts the data synchronization interval in real time based on the actual operating status of the system to achieve an adaptive balance between data synchronization real-time performance and system load balancing.

[0045] In one optional embodiment, the actual operating state of the system is determined by the current data synchronization delay. The closer the current data synchronization delay is to the standard data synchronization delay, the smaller the deviation between the actual and standard operating states of the system; conversely, the greater the deviation between the current and standard data synchronization delay, the greater the deviation. The current data synchronization delay is determined comprehensively based on data synchronization latency, CPU utilization, and task queue length. If the current data synchronization delay is significantly higher than the standard data synchronization delay, it indicates a smaller current data synchronization interval, higher CPU utilization, or a longer task queue length. Conversely, if the current data synchronization delay is significantly lower than the standard data synchronization delay, it indicates a larger current data synchronization interval, lower CPU utilization, or a shorter task queue length. In this case, the system's resource utilization is low, and the consistency between databases is low.

[0046] It is understandable that determining the current data synchronization delay solely based on the data synchronization latency, without considering system load, would significantly increase the burden on resources and the database, further affecting data synchronization or resource exhaustion. Therefore, this application's embodiments use data synchronization latency as the primary control objective, with CPU utilization and task queue length as constraints. For example, when CPU utilization is too high or the task queue length exceeds a threshold, the data synchronization interval is increased to prioritize reducing system load. Through comprehensive consideration of multi-dimensional indicators, an adaptive balance between real-time data synchronization and system load balancing is achieved. Based on this, referring to... Figure 4 Step S10 includes steps S11 to S14: Step S11: Determine the data synchronization latency, CPU utilization, and task queue length based on the observed indicators; Step S12: Normalize the data synchronization latency, CPU utilization and task queue length to determine the normalized values ​​corresponding to each observation indicator. Step S13: Perform a weighted fusion calculation based on the second weight values ​​corresponding to the data synchronization delay, CPU utilization, and task queue length, respectively, and the normalized value, and use the calculated result as the current data synchronization delay. Step S14: Determine the current error value based on the difference between the current data synchronization delay and the standard data synchronization delay.

[0047] It is understandable that the observation metrics are multidimensional, and there are multiple metrics affecting the data synchronization strategy. Furthermore, these metrics interact with each other. For example, increased CPU utilization may lead to increased synchronization time, which inevitably affects the task queue length. Similarly, if CPU utilization is too high, even with low synchronization latency, the synchronization interval needs to be increased accordingly to appropriately reduce the load impact. Optionally, the observation metrics include, but are not limited to, data synchronization latency, CPU utilization, and task queue length. They may also include changed data priority; if the changed data priority is high, the data synchronization interval should be appropriately reduced to ensure timely synchronization of the data to be synchronized to the target database. Other system resource layer metrics besides CPU utilization may also include memory utilization; if memory utilization is significantly higher than a threshold, the data synchronization interval should be appropriately increased to reduce the load impact. Other system resource layer metrics may also include the number of threads in the thread pool, which is directly proportional to the data synchronization interval—the more threads, the larger the data synchronization interval. Other system resource layer metrics may also include active database sessions and the number of database connections, etc.

[0048] In one alternative implementation, the data synchronization status of the data synchronization system is monitored in real time to obtain at least one observation index, a second weight value corresponding to different observation indices is determined, and each observation index is weighted and fused according to the second weight value to generate the current data synchronization delay.

[0049] It should be noted that different observation metrics correspond to different units of measurement. For example, CPU utilization is a percentage, task queue length is an absolute value, and data synchronization latency is in milliseconds or seconds. Therefore, after obtaining the observation metrics, they need to be normalized to convert them into values ​​with the same units of measurement. Specifically, the observation metrics are converted into values ​​in a unified quantization space, such as [0, 1]. For example, if the observation metric is CPU utilization, the percentage value corresponding to CPU utilization is converted into the corresponding normalized value, with 100% corresponding to 1 and 50% corresponding to a normalized value of 0.5. Similarly, if the observation metric is task queue length, the corresponding normalized value can be determined based on the ratio of the task queue length to the maximum preset queue length. For example, if the task queue length is 5 and the maximum preset queue length is 10, the corresponding normalized value is 0.5. Furthermore, if the observation metric is data synchronization latency, the corresponding normalized value can be determined based on the ratio of the data synchronization latency to the maximum allowable latency error. For example, if the data synchronization latency is 10ms and the maximum allowable latency error is 100ms, the corresponding normalized value is 0.1.

[0050] Understandably, different types of observation metrics impose different constraints on data synchronization. For example, a larger data synchronization latency means a greater current data synchronization delay, thus requiring a reduction in the data synchronization interval. However, if CPU utilization is high, reducing the data synchronization interval will lead to a continuous increase in CPU utilization, resulting in system resource exhaustion. Therefore, after generating normalized values ​​for each observation metric, the normalized values ​​are directionally corrected according to the type of the observation metric. If the observation metric is a positive constraint type, such as data synchronization latency, the determined correction direction is positive, and the corrected data synchronization latency is +0.1. If the observation metric is a negative constraint type, such as CPU utilization, the determined correction direction is negative, and the corrected normalized value for CPU utilization is -0.1.

[0051] Furthermore, after normalizing and correcting the observation indicators, normalized values ​​for each indicator are obtained, and second weight values ​​for each indicator are acquired. A weighted fusion calculation is then performed based on the second weight values ​​and the normalized values, and the calculated result is used as the current data synchronization delay. For example, the current data synchronization delay is calculated using the following formula:

[0052] Among them, Total error The current data synchronization latency is represented by E. 0.4, 0.3, and 0.3 are the second weight values ​​corresponding to the data synchronization latency, CPU utilization, and task queue length, respectively. delay (-Ecpu) is the normalized value corresponding to the data synchronization latency, (-Ecpu) is the normalized value after direction correction corresponding to the CPU utilization, and (-Equeue) is the normalized value after direction correction corresponding to the task queue length.

[0053] Furthermore, after calculating the current data synchronization delay, the current error value is determined based on the difference between the current data synchronization delay and the standard data synchronization delay, where the standard data synchronization delay can be 0. For example, if the current data synchronization delay is 0.5 and the standard data synchronization delay is 0.1, then the current error value is 0.4.

[0054] It is understandable that a larger error value indicates a larger current data synchronization interval, and a smaller error value indicates a smaller current data synchronization interval.

[0055] Furthermore, when calculating the current error value, it is input into the PID model so that the PID model can calculate the cumulative error value and the change in error value based on the current error value. The cumulative error value is the sum of the error values ​​from each run of the data synchronization system, and the change in error value is the difference between the current error value and the previous error value. It is understandable that if the data synchronization interval is quickly adjusted based solely on the current error value without considering the cumulative error, the system may fail to reach its ideal state for an extended period. The change in error value is used to assess the error change trend, enabling early suppression of system overshoot and oscillation, ensuring a smooth and safe adjustment process. Ignoring the change in error value can easily lead to repeated system oscillations and severe overshoot; if synchronization is too fast, it can cause excessive load. Therefore, this embodiment combines the current error value, the cumulative error value, and the change in error value to adjust the data synchronization interval. This allows for timely adjustment of the data synchronization interval using the current error value, gradual reduction of the error using the cumulative error value to prevent error accumulation, and stable adjustment of the data synchronization interval using the change in error value.

[0056] Further, proceed with steps S20~S40: Step S20: Calculate the data synchronization interval adjustment amount for the current data synchronization delay based on the current error value, the cumulative error value, and the change in the error value. Step S30: Adjust the current data synchronization interval according to the data synchronization interval adjustment amount to generate the target data synchronization interval; Step S40: Transmit the data to be synchronized from the source database to the target database at the target data synchronization interval.

[0057] In one optional implementation, the PID model calculates the data synchronization interval adjustment based on the calculated current error value, cumulative error value, and change in the error value, referring to... Figure 5 Step S20 includes: Step S21: Determine the current error value, the cumulative error value, and the first weight value corresponding to the change in the error value, respectively; Step S22: Perform a weighted fusion calculation on the current error value, the cumulative error value, and the change in the error value according to the first weight value, and use the calculated result as the data synchronization interval adjustment amount.

[0058] Specifically, the PID model includes a proportional, integral, and derivative component. The proportional component rapidly adjusts based on the current error; the integral component accumulates all errors within the historical period to eliminate long-term steady-state deviations; and the derivative component predicts the deviation trend based on the changes in error between adjacent periods to suppress system overshoot and oscillations. The outputs of the three components are weighted and summed to obtain the data synchronization interval adjustment amount, which is used to dynamically adjust the data synchronization interval to achieve stable adaptive control with real-time synchronization and balanced system load. The first weight values ​​corresponding to the current error value, the accumulated error value, and the change in error value are 0.5, 0.1, and 0.05, respectively.

[0059] For example, the formula for calculating the data synchronization interval adjustment in the PID model is:

[0060] in, T is the data synchronization interval adjustment amount, Kp, Ki, and Kd are the first weight values ​​corresponding to the current error value, cumulative error value, and error value change, respectively, e(t) is the current error value, ∑(t) is the cumulative error value, and e(t-1) is the error value corresponding to the previous acquisition time.

[0061] It should be noted that the data synchronization interval adjustment calculated by the PID model can be a positive value. When the data synchronization interval adjustment is positive, the data synchronization interval is increased and the data synchronization frequency is decreased to relieve the CPU load pressure on the system. When the data synchronization interval adjustment is negative, the data synchronization interval is decreased and the data synchronization frequency is increased to reduce data synchronization delay and ensure real-time performance.

[0062] Furthermore, after calculating the data synchronization interval adjustment amount, the current data synchronization interval is adjusted based on the data synchronization adjustment amount. Specifically, when the data synchronization interval growth amount is positive, the current data synchronization interval is added to the data synchronization interval adjustment amount to obtain the target data synchronization interval, thereby increasing the data synchronization interval and relieving the system load pressure; when the data synchronization interval growth amount is negative, the current data synchronization interval is subtracted from the absolute value of the data synchronization interval adjustment amount to obtain the target data synchronization interval, thereby reducing the data synchronization interval, increasing data synchronization efficiency, reducing data latency, and ensuring the real-time performance of data synchronization.

[0063] Understandably, PID models are prone to over-adjustment, overshoot oscillation, and loss of synchronization frequency control during closed-loop regulation. Specifically, over-adjustment can easily lead to excessively small or large target data synchronization intervals. Too small an interval can exhaust system resources. If no lower limit is set for the synchronization interval, the PID model will continuously reduce it when the error is large, causing the synchronization frequency to rise indefinitely. This results in a sharp increase in CPU utilization, task queue congestion, and database overload, ultimately causing system lag, slow response, or even service crashes. Conversely, excessively large synchronization intervals can lead to data synchronization real-time failure. If no upper limit is set, the PID model will continuously increase the interval under high system load, causing the synchronization frequency to drop indefinitely. This leads to continuously increasing data synchronization latency and long-term inconsistencies between the source and target databases, failing to meet near-real-time synchronization requirements. Furthermore, when the PID model experiences overshoot and system instability, the closed-loop control is prone to overshoot, repeated fluctuations, and loss of control over the adjustment amplitude during dynamic adjustment, causing the synchronization interval to fluctuate wildly and resulting in unstable system operation. By limiting the amplitude of data synchronization intervals, drastic fluctuations can be prevented and the system can operate smoothly. Based on this, this application proposes to address the problem of unreasonable target data synchronization intervals by limiting the amplitude of data synchronization intervals.

[0064] In one alternative implementation, refer to Figure 6 After step S30, steps S31 to S33 are also included: Step S31: When the target data synchronization interval exceeds the first preset data synchronization interval, update the target data synchronization interval to the first preset data synchronization interval. Step S32: When the target data synchronization interval is less than the second preset data synchronization interval, update the target data synchronization interval to the second preset data synchronization interval. Step S33: When the target synchronization interval is greater than or equal to the second preset data synchronization interval and less than or equal to the first preset data synchronization interval, perform the step of synchronizing the data to be synchronized to the target database at the target data synchronization interval. The second preset data synchronization interval is less than the first preset data synchronization interval.

[0065] Specifically, after adjusting the current data synchronization interval based on the current data synchronization interval to generate a target data synchronization interval, it is determined whether the generated target data synchronization interval falls within a preset data synchronization interval range. The minimum data synchronization interval corresponding to the preset data synchronization interval is the second preset data synchronization interval, and the maximum data synchronization interval is the first preset data synchronization interval, where the first preset data synchronization interval is greater than the second preset synchronization interval. When the target data synchronization interval falls within the preset data synchronization interval range, it is determined that the generated target data synchronization interval will not be too large or too small, and data synchronization is directly performed using this target data synchronization interval. When the target data synchronization interval does not fall within the preset data synchronization interval range, the comparison result between the target data synchronization interval and the first and second preset data synchronization intervals is determined. If the target data synchronization interval is smaller than the second preset data synchronization interval, it is determined that the data synchronization interval is too small, and data synchronization is performed using this target data synchronization interval. This can easily lead to a sharp increase in CPU utilization, task queue congestion, and database overload, ultimately causing system lag, slow response, or even service crashes. Therefore, when the target data synchronization interval is too small, a second preset data synchronization interval is used to replace the target data synchronization interval, thus performing data synchronization at the second preset interval. The second preset data synchronization interval is the minimum data synchronization interval that the system can tolerate during balanced operation. When the target data synchronization interval exceeds the first preset data synchronization interval, it is determined that the data synchronization interval is too large. If data synchronization is performed at this target data synchronization interval, it can easily lead to a continuous increase in data synchronization latency and long-term inconsistency between the source and target databases, failing to meet the near real-time synchronization requirements. Therefore, when the target data synchronization interval is too large, the first preset data synchronization interval is used to replace the target data synchronization interval, thus performing data synchronization at the first preset data synchronization interval. The first preset data synchronization interval is the maximum data synchronization interval that can be tolerated to ensure data consistency.

[0066] It is understandable that limiting the data synchronization interval restricts the reasonable range of synchronization frequency, preventing system instability caused by excessive PID control. By setting upper and lower limits for the synchronization interval, when the calculated target data synchronization interval is too small, it is limited to the minimum preset data synchronization interval to prevent excessively frequent synchronization from causing high CPU utilization, task queue backlog, and system resource exhaustion. When the target data synchronization interval is too large, it is limited to the maximum preset data synchronization interval to prevent excessive synchronization latency and data consistency. This ensures that the data synchronization interval is always within a safe and controllable range, maintaining a stable balance between real-time performance and load balancing, and avoiding oscillations, overloads, or failures.

[0067] It's important to note that during data synchronization, when the system experiences sudden high concurrency, continuous CPU overload, severe task queue backlog, and a sharp increase in synchronization latency, PID control exhibits a lag and cannot quickly suppress the continued deterioration of system load. If synchronization tasks continue to execute under these conditions, they will further consume system resources, leading to slower business response, service instability, or even complete system crashes. Therefore, by monitoring the system's health and triggering appropriate circuit breakers in a timely manner, system security is ensured, preventing system collapse under extreme load pressure.

[0068] Specifically, step S30 includes: A composite health index is generated based on the observed indicators; When the composite health index is greater than or equal to the first composite health index, the result of adding the data synchronization interval adjustment amount and the current data synchronization interval is taken as the target data synchronization interval; When the composite health index is less than the first composite health index, the result of adding the data synchronization interval adjustment amount and the current data synchronization interval is used as the initial data synchronization interval, and the initial data synchronization interval with a preset multiple is used as the target data synchronization interval. When the composite health index is less than the second composite health index, the step of adjusting the current data synchronization interval according to the data synchronization interval adjustment amount to generate the target data synchronization interval is paused, and the second composite health index is less than the first composite health index.

[0069] In one optional implementation, after collecting observation indicators at a preset collection frequency, a composite health index is generated based on the observation indicators. The observation indicators are then normalized to generate corresponding normalized values. The composite health index is then calculated based on the normalized values, as exemplified by the following formula:

[0070] Wherein, Health Score is a composite health index, a, b, and c are the health weight coefficients corresponding to the observed indexes, and Cpu, Delay, and Queue are the normalized data corresponding to data synchronization latency, CPU utilization, and task queue length, respectively.

[0071] Furthermore, after calculating the composite health index, the current system health level is determined based on the composite health index, and a tiered circuit breaker strategy is determined based on the current health level. The higher the composite health index, the healthier the system and the higher the health level. The lower the composite health index, the greater the system complexity and pressure, and it is urgent to reduce the system load pressure by reducing the data synchronization frequency.

[0072] Specifically, when the composite health index is greater than or equal to the first composite health index, the current tiered circuit breaker level is determined to be the first level. At this time, the system load pressure is determined to be low and the health status is good, so there is no need to execute the limit protection. The tiered circuit breaker strategy at this time is the trigger threshold warning strategy, which generates a warning reminder and outputs the warning reminder. At the same time, the result of adding the data synchronization interval adjustment amount and the current data synchronization interval is used as the target data synchronization interval, and data synchronization is performed at the target data synchronization interval.

[0073] In another optional implementation, when the composite health index is less than the first composite health index, the current graded circuit breaker level is determined to be the second level. At this time, it is determined that the system load pressure is large and the health status is poor, triggering a dynamic frequency reduction strategy. Then, the result of adding the data synchronization interval adjustment amount and the current data synchronization interval is used as the initial data synchronization interval, and the initial data synchronization interval with a preset multiple is used as the target data synchronization interval, thereby increasing the data synchronization interval and performing data synchronization with a smaller data synchronization efficiency to reduce the load pressure. In another optional implementation, when the composite health index is less than the second composite health index, and the second composite health index is less than the first composite health index, the current tiered circuit breaker level is determined to be the third level. At this time, it is determined that the system load pressure is high and the health status is deteriorating, triggering a forced pause strategy. That is, data synchronization is paused at this time, and the step of adjusting the current data synchronization interval according to the data synchronization interval adjustment amount to generate the target data synchronization interval is also paused, thereby realizing the forced pause of data synchronization and avoiding resource exhaustion.

[0074] It should be noted that when any level of circuit breaker mechanism is triggered, the system automatically records logs, including information such as the trigger time, composite health index, real-time values ​​of various observed indicators, circuit breaker level, and the executed tiered circuit breaker strategy. At the same time, it reports alarms to facilitate the rapid location, analysis and handling of anomalies by operation and maintenance personnel.

[0075] Understandably, a three-layer collaborative control mechanism is adopted to ensure stable system operation. Under normal conditions, the PID model performs fine adjustment of the data synchronization interval. When the system is in an abnormal state, the upper and lower limits of the data synchronization interval are set to prevent the adjustment from going out of control. At the same time, when it is determined that the system is under extreme load, a graded circuit breaker mechanism is triggered to achieve three levels of protection: early warning, frequency reduction, and suspension, to ensure that the system does not crash and services are not blocked.

[0076] In this embodiment, the current data synchronization delay is determined by integrating multiple dimensions of indicators to prevent problems such as system overload or low data synchronization efficiency caused by adjusting the data synchronization interval based on a single indicator. The current data synchronization delay is then used as input to a PID model, which determines the current error value based on the current data synchronization delay. The proportional, integral, and derivative components are then executed in parallel based on the current error value to obtain the data synchronization interval adjustment amount output by each component. The data synchronization interval adjustment amounts output by each component and their corresponding first weight values ​​are then weighted and fused to obtain the final data synchronization interval adjustment amount. The final data synchronization interval adjustment amount is generated based on the instantaneous error, cumulative error, and error change rate by the PID model. This ensures data synchronization efficiency while avoiding excessive adjustment or repeated and drastic fluctuations in the synchronization interval, and simultaneously ensures a balance between synchronization efficiency and system load balancing.

[0077] Third Embodiment Based on the above embodiments, referring to Figure 7 Before step S10, steps S50 to S80 are also included: Step S50: Detecting the existence of operation description information for the source database, converting the operation description information into operation description information in a preset format; Step S60: Obtain operation intent information according to the operation description information in a preset format; Step S70: Determine the data to be synchronized based on the operation intent information; Step S80: Generate a data synchronization task based on the data to be synchronized and store it in the task queue.

[0078] Understandably, before performing data synchronization, it is necessary to capture data change operations, determine the data to be synchronized in the source database based on these operations, and finally synchronize the data to be synchronized from the source database to the target database in real time. Common methods for capturing data change operations include adding triggers to the front end of the data synchronization system to detect these operations and obtain the data to be synchronized. However, adding triggers is a non-lightweight operation, resulting in high execution overhead for the data synchronization system and impacting its performance. Alternatively, the data to be synchronized can be obtained by parsing the operation logs of the source database. However, log parsing involves complex processing logic, requiring additional configuration of corresponding operation log parsing models. Furthermore, different types of databases require different operation log parsing models, leading to poor universality and high implementation costs. Another implementation method is application-layer dual-write, which requires embedding relevant business code for synchronization into the business code of both the source and target databases, resulting in high invasiveness and significant modification requirements.

[0079] Based on this, this application proposes a low-intrusive data change capture method. Specifically, by decoupling data change operation capture from data synchronization, the application layer (business processing layer) is shielded from the perception of data synchronization, which solves the problem of needing to add business code, resulting in high intrusiveness and large amount of modification to the business code. At the same time, data change operation capture can be achieved without parsing logs or adding triggers, reducing system overhead.

[0080] It is understandable that business applications typically use ORM frameworks to simplify database access logic when operating on databases. ORM frameworks reside within the business application, and their role is to perform data format conversion between program code and the database, allowing developers to manipulate data in an object-oriented manner without directly writing complex SQL statements. Common ORM frameworks include MyBatis and JPA, etc. Different frameworks differ in data operation methods, interface definitions, and execution flows, and are not universally compatible. This application embodiment obtains operation description information for the source database by monitoring the ORM framework, without needing to parse the database operation logs. It is understood that operation description information from different ORM frameworks requires configuring the corresponding ORM framework language for accurate recognition. If different business applications use different ORM frameworks, corresponding SQL interceptors and SQL parsers need to be configured separately for each ORM framework. Based on this, this application embodiment abstracts a unified, pluggable data change monitoring component that supports multiple different ORM frameworks, improving development efficiency.

[0081] In one optional implementation, the data change monitoring component includes an ORM intent recognition component, which comprises a unified SQL interception layer, a unified SQL parsing layer, and a multi-ORM framework adaptation plugin layer. In this case, the data synchronization system does not require modification of the business code; it intercepts data operations solely through native framework extension points, thereby accurately detecting and capturing data change events. Upon system startup, the multi-ORM framework adaptation plugin layer automatically scans the configuration file or Spring container to determine the current application's ORM framework, and then loads the plugin corresponding to the ORM framework used by the current data change operation, thus achieving the versatility of the data synchronization system. For example, if the current data operation uses MyBatis, the MyBatis adaptation plugin is loaded; if JPA / Hibernate is used, the JPA adaptation plugin is loaded, and each plugin completes the registration and initialization of its own interceptors.

[0082] Furthermore, when monitoring the ORM framework using appropriate adapter plugins, if data modification operations targeting the source database are detected within the ORM framework, a unified SQL interception layer is used to obtain the operation description information for this data modification operation. This unified SQL interception layer silently captures the operation description information without intruding on business logic. The operation description information includes, but is not limited to, the operation type (e.g., INSERT, UPDATE, or DELETE); the operation object (e.g., the name of the data table involved (or the table name corresponding to the entity class name); operation parameters (e.g., the specific data value or primary key ID after the change); and the transaction context (e.g., obtaining the transaction status of the current thread and binding the transaction ID through the Spring transaction synchronization manager).

[0083] Furthermore, after capturing the operation description information, it is sent to a unified SQL parsing layer. This unified SQL parsing layer is responsible for standardizing and uniformly parsing the heterogeneous operation description information captured by different ORM frameworks. Specifically, it converts the operation description information into operation description information in a preset format, and then generates operation intent information in a preset format based on the operation description information in the preset format. The determined operation intent information includes a standardized table name, operation type, primary key value, and modified data. The modified data is the data to be synchronized. For example, in the current ORM framework MyBatis, the native SQL string of MyBatis is parsed into an abstract syntax tree (AST), and the table name, condition fields, and values ​​are extracted. In the current ORM framework JPA / Hibernate, the entity state change of JPA / Hibernate is mapped to a standard data change entity. It can be understood that the preset format is a format that the business code can recognize.

[0084] Optionally, when data to be synchronized is detected, refer to Figure 8 Step S80 includes steps S81 to S84: Step S81: Obtain the execution status corresponding to the data to be synchronized; Step S82: When the execution state is the first execution state, a data synchronization task is generated based on the data to be synchronized and stored in the task queue; Step S83: When the execution state is the second execution state, discard the data to be synchronized, and return to the step of detecting the existence of operation description information for the source database and obtaining operation intent information based on the operation description information; Step S84: When the execution state is the third execution state, perform the step of generating a data synchronization task based on the data to be synchronized and storing it in the task queue.

[0085] Optionally, the data change monitoring component further includes a Spring transaction synchronization manager, which establishes a communication connection with the ORM intent recognition component. When data to be synchronized is detected, a data synchronization task is not directly generated. Instead, the Spring transaction synchronization manager performs execution status detection based on the transaction execution status corresponding to the data to be synchronized, and generates a data synchronization task based on the detection result. Specifically, when the execution status is the first execution status (transaction commit status), the current data change operation is confirmed to be effective, and the data to be synchronized is stored in the task queue for subsequent synchronization. When the execution status is the second execution status (transaction rollback status), the data to be synchronized is discarded, and no synchronization task is generated, ensuring data synchronization consistency. When the execution status is the third execution status (no transaction status), a synchronization task is directly generated based on the data to be synchronized and stored in the task queue. In this way, by detecting the execution status, the problem of erroneous synchronization due to data not actually changing is avoided, ensuring database data consistency.

[0086] Furthermore, after generating the data synchronization task, the data synchronization task is permanently stored in the task queue and subsequently pushed to the target database in near real-time to complete the data synchronization. In this way, there is no need to wait for the data synchronization to be completed before the corresponding task can be executed. Instead, task execution and business synchronization are processed in parallel, avoiding the blocking of business execution by data synchronization.

[0087] In this embodiment, the data synchronization system constructs a general ORM intent recognition component and integrates it with the Spring transaction synchronization manager to capture data change events in real time. Based on the execution status of each data change event, corresponding data synchronization tasks are generated. The ORM intent recognition component supports pluggable adaptation to multiple ORM frameworks, reducing development costs under different ORM framework environments. Simultaneously, by utilizing SQL interceptors to intercept data change events, data changes can be captured without triggers, database log parsing, or business code modification, reducing system overhead and improving data capture efficiency.

[0088] It should be noted that the above examples are only for understanding this application and do not constitute a limitation on the data synchronization method of this application. Any simple modifications based on this technical concept are within the protection scope of this application.

[0089] This application provides a data synchronization device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the data synchronization method in Embodiment 1 above.

[0090] The following is for reference. Figure 9 The diagram illustrates a structural schematic of a data synchronization device suitable for implementing embodiments of this application. The data synchronization device in the embodiments of this application may include, but is not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Description), PMPs (Portable Media Players), in-vehicle terminals (e.g., in-vehicle navigation terminals), and fixed terminals such as digital TVs and desktop computers. Figure 9 The data synchronization device shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of this application.

[0091] like Figure 9 As shown, the data synchronization device may include a processing unit 1001 (e.g., a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 1002 or a program loaded from a storage device 1003 into a random access memory (RAM) 1004. The RAM 1004 also stores various programs and data required for the operation of the data synchronization device. The processing unit 1001, ROM 1002, and RAM 1004 are interconnected via a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Typically, the following systems can be connected to the I / O interface 1006: input devices 1007 including, for example, a touchscreen, touchpad, keyboard, mouse, image sensor, microphone, accelerometer, gyroscope, etc.; output devices 1008 including, for example, a liquid crystal display (LCD), speaker, vibrator, etc.; storage devices 1003 including, for example, magnetic tape, hard disk, etc.; and communication devices 1009. Communication device 1009 allows the data synchronization device to communicate wirelessly or wiredly with other devices to exchange data. Although the figures show data synchronization devices with various systems, it should be understood that implementation or possession of all the systems shown is not required. More or fewer systems may be implemented alternatively.

[0092] Specifically, according to the embodiments disclosed in this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments disclosed in this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device, or installed from storage device 1003, or installed from ROM 1002. When the computer program is executed by processing device 1001, it performs the functions defined in the methods of the embodiments disclosed in this application.

[0093] The data synchronization device provided in this application, employing the data synchronization method described in the above embodiments, can solve the technical problem that a fixed data synchronization interval cannot balance the real-time performance and load balancing of data synchronization. Compared with the prior art, the beneficial effects of the data synchronization device provided in this application are the same as those of the data synchronization method described in the above embodiments, and other technical features of this data synchronization device are the same as those disclosed in the previous embodiment method, and will not be repeated here.

[0094] It should be understood that the various parts disclosed in this application can be implemented using hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials, or characteristics can be combined in any suitable manner in one or more embodiments or examples.

[0095] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0096] This application provides a computer-readable storage medium having computer-readable program instructions (i.e., a computer program) stored thereon, the computer-readable program instructions being used to execute the data synchronization method described in the above embodiments.

[0097] The computer-readable storage medium provided in this application may be, for example, a USB flash drive, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, system, or device. The program code contained on the computer-readable storage medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination thereof.

[0098] The aforementioned computer-readable storage medium may be included in the data synchronization device; or it may exist independently and not assembled into the data synchronization device.

[0099] The aforementioned computer-readable storage medium carries one or more programs that, when executed by a data synchronization device, enable the data synchronization device to write computer program code for performing the operations of this application in one or more programming languages ​​or a combination thereof. These programming languages ​​include object-oriented programming languages—such as Java, Smalltalk, and C++—and conventional procedural programming languages—such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0100] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0101] The modules described in the embodiments of this application can be implemented in software or hardware. The names of the modules do not necessarily limit the functionality of the unit itself.

[0102] The readable storage medium provided in this application is a computer-readable storage medium that stores computer-readable program instructions (i.e., a computer program) for executing the above-described data synchronization method. This solves the technical problem that a fixed data synchronization interval cannot balance the real-time performance and load balancing of data synchronization. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in this application are the same as those of the data synchronization method provided in the above embodiments, and will not be repeated here.

[0103] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the data synchronization method described above.

[0104] The computer program product provided in this application can solve the technical problem that a fixed data synchronization interval cannot balance the real-time performance and load balancing of data synchronization. Compared with the prior art, the beneficial effects of the computer program product provided in this application are the same as those of the data synchronization method provided in the above embodiments, and will not be repeated here.

[0105] The above description is only a part of the embodiments of this application and does not limit the patent scope of this application. All equivalent structural transformations made under the technical concept of this application and using the contents of the specification and drawings of this application, or direct / indirect applications in other related technical fields, are included in the patent protection scope of this application.

Claims

1. A data synchronization method, characterized in that, The steps of the data synchronization method include: Obtain the current error value between the current data synchronization delay determined based on the observation index and the standard data synchronization delay, and determine the cumulative error value and the change in error value based on the current error value; The adjustment amount of the data synchronization interval for the current data synchronization delay is calculated based on the current error value, the cumulative error value, and the change in the error value. The current data synchronization interval is adjusted according to the data synchronization interval adjustment amount to generate the target data synchronization interval; The data to be synchronized in the source database is transmitted to the target database at the target data synchronization interval.

2. The data synchronization method as described in claim 1, characterized in that, The step of calculating the data synchronization interval adjustment amount for the current data synchronization delay based on the current error value, the cumulative error value, and the change in the error value includes: Determine the first weight value corresponding to the current error value, the cumulative error value, and the change in the error value, respectively; The current error value, the cumulative error value, and the change in the error value are weighted and fused according to the first weight value, and the result is used as the data synchronization interval adjustment amount.

3. The data synchronization method as described in claim 1, characterized in that, The step of obtaining the current error value between the current data synchronization delay determined based on the observation index and the standard data synchronization delay includes: The data synchronization latency, CPU utilization, and task queue length are determined based on the observed indicators. The data synchronization latency, CPU utilization, and task queue length are normalized to determine the normalized values ​​corresponding to each observation indicator. The data synchronization delay, CPU utilization, and task queue length are respectively weighted and fused according to the second weight values ​​and the normalized values, and the calculated result is used as the current data synchronization delay. The current error value is determined based on the difference between the current data synchronization delay and the standard data synchronization delay.

4. The data synchronization method as described in claim 1, characterized in that, After the step of adjusting the current data synchronization interval according to the data synchronization interval adjustment amount to generate the target data synchronization interval, the method further includes: When the target data synchronization interval exceeds the first preset data synchronization interval, the target data synchronization interval is updated to the first preset data synchronization interval. When the target data synchronization interval is less than the second preset data synchronization interval, the target data synchronization interval is updated to the second preset data synchronization interval; When the target data synchronization interval is greater than or equal to the second preset data synchronization interval and less than or equal to the first preset data synchronization interval, the step of synchronizing the data to be synchronized to the target database at the target data synchronization interval is executed. The second preset data synchronization interval is less than the first preset data synchronization interval.

5. The data synchronization method as described in claim 1, characterized in that, Before the step of obtaining the current error value between the current data synchronization delay determined based on the observation index and the standard data synchronization delay, the method further includes: If operation description information targeting the source database is detected, the operation description information is converted into operation description information in a preset format. The operation intent information is obtained based on the operation description information in the preset format; The data to be synchronized is determined based on the aforementioned operational intent information; A data synchronization task is generated based on the data to be synchronized and stored in the task queue.

6. The data synchronization method as described in claim 5, characterized in that, The step of generating a data synchronization task based on the data to be synchronized and storing it in the task queue includes: Obtain the execution status corresponding to the data to be synchronized; When the execution state is the first execution state, the step of generating a data synchronization task based on the data to be synchronized and storing it in the task queue is performed.

7. The data synchronization method as described in claim 6, characterized in that, After the step of obtaining the execution status corresponding to the data to be synchronized, the method further includes: When the execution state is the second execution state, discard the data to be synchronized, and return to the step of detecting the existence of operation description information for the source database and converting the operation description information into operation description information in a preset format; When the execution state is the third execution state, the step of generating a data synchronization task based on the data to be synchronized and storing it in the task queue is performed.

8. The data synchronization method according to any one of claims 1-7, characterized in that, The step of adjusting the current data synchronization interval according to the data synchronization interval adjustment amount to generate the target data synchronization interval includes: A composite health index is generated based on the observed indicators; When the composite health index is greater than or equal to the first composite health index, the result of adding the data synchronization interval adjustment amount and the current data synchronization interval is taken as the target data synchronization interval; When the composite health index is less than the first composite health index, the result of adding the data synchronization interval adjustment amount and the current data synchronization interval is used as the initial data synchronization interval, and the initial data synchronization interval with a preset multiple is used as the target data synchronization interval. When the composite health index is less than the second composite health index, the step of adjusting the current data synchronization interval according to the data synchronization interval adjustment amount to generate the target data synchronization interval is paused, and the second composite health index is less than the first composite health index.

9. A data synchronization system, the data synchronization system comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the data synchronization method as described in any one of claims 1 to 8.

10. A storage medium, characterized in that, The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, it implements the steps of the data synchronization method as described in any one of claims 1 to 8.

Images