A data synchronization control method and system between heterogeneous databases and an electronic device

Abstract

The application discloses a kind of data synchronization control method, system and electronic equipment between heterogeneous databases, it is related to data synchronization field, obtains the synchronization running state parameter between source end database and target end database, and evaluates current system load level, according to this dynamic adjustment batch size of single data synchronization, obtain the task attribute characteristics of multiple concurrent synchronization tasks, based on task attribute characteristics Calculation of the scheduling weight of each concurrent synchronization task, and in combination with the real-time load state of target end database, dynamically allocate the execution resource of each concurrent synchronization task, based on the adjusted batch size and the allocated execution resource, incremental data in source end database is synchronized to target end database.The application adjusts batch size dynamically by sensing link load, balances high load congestion and low load resource waste;Based on the concurrent task of task attribute and target end load scheduling, priority is guaranteed to critical task, and the optimal balance of transmission efficiency and system stability is realized.

Description

Technical Field

[0001] This invention relates to the field of data synchronization technology, and more specifically, to a data synchronization control method, system, and electronic device for heterogeneous databases. Background Technology

[0002] With the rapid development of the Internet of Things (IoT), the Industrial Internet, and intelligent transportation systems, various terminal devices generate massive amounts of monitoring data and status information during locomotive operation monitoring and health management. This data is typically characterized by high-frequency writes, large scale, and diverse structures, thus requiring persistent storage using distributed columnar storage databases to meet high-throughput write demands. However, in practical business applications such as locomotive fault diagnosis, operational status statistical analysis, health assessment report generation, and PHM (Prognostics and Health Management) systems, efficient data access and analysis often rely on the powerful query capabilities and transaction support of relational databases. To break down the barriers between the data storage layer and the application analysis layer and maximize data value, the industry typically employs data synchronization technology to migrate or copy data from the source database to the target database to meet the data analysis needs of upper-layer business applications. Existing solutions usually acquire source data through periodic full scans or log-based incremental capture, then write it in batches to the target database after format conversion, thereby maintaining data consistency between the two ends.

[0003] While existing data synchronization technologies have enabled data transfer between heterogeneous databases to some extent, they still face persistent technical bottlenecks in complex and ever-changing real-world operating environments. Especially in scenarios with high concurrency, large data volumes, and dynamically fluctuating system loads, traditional synchronization strategies often struggle to balance transmission efficiency and system stability. When the data generation rate at the source changes drastically or the target database's processing capacity is limited, fixed synchronization batch sizes and static resource allocation patterns can easily lead to continuous backlog in the source cache queue, a sharp increase in synchronization latency, exhaustion of target database connection resources, and even service unavailability. Furthermore, in multi-task parallel environments, how to rationally schedule synchronization tasks of different priorities to avoid resource contention, and how to ensure data is not lost, duplicated, or lost while maintaining consistency after network fluctuations or abnormal program interruptions, remain common challenges that urgently need to be addressed in this field, directly impacting the real-time performance and reliability of upper-layer business applications. Summary of the Invention

[0004] In view of this, the present invention discloses a data synchronization control method, system and electronic device between heterogeneous databases, in order to solve the technical problem that traditional synchronization strategies are difficult to balance transmission efficiency and system stability in complex and ever-changing actual operating environments, resulting in increased synchronization delay, resource exhaustion and difficulty in ensuring data consistency.

[0005] A data synchronization control method between heterogeneous databases includes:

[0006] Obtain the synchronization status parameters between the source database and the target database;

[0007] The current system load level is assessed based on the synchronous operation status parameters, and the batch size of a single data synchronization is dynamically adjusted according to the current system load level.

[0008] The task attribute characteristics of multiple concurrent synchronous tasks are obtained, the scheduling weight of each concurrent synchronous task is calculated based on the task attribute characteristics, and the execution resources of each concurrent synchronous task are dynamically allocated in combination with the real-time load status of the target database.

[0009] Based on the adjusted batch size and allocated execution resources, incremental data in the source database is synchronized to the target database.

[0010] Optionally, the synchronization operation status parameters include the source-side cache queue length, the target-side average write time, the system processor resource utilization rate, and the current synchronization delay;

[0011] The assessment of the current system load level based on the synchronized operating status parameters includes:

[0012] The parameters in the synchronous operation status parameters are normalized, and a weighted fusion is performed based on the normalized parameters to obtain a load evaluation value that characterizes the current system load level.

[0013] Optionally, the formula for calculating the load evaluation value is:

[0014] ;

[0015] In the formula, for The load evaluation value at that time. The normalized source-side cache queue length, The average write time to the target end, The system processor resource utilization and The current synchronization delay, They represent Preset weight parameters, and satisfy Constraints;

[0016] The formula for adjusting the batch size is:

[0017] ;

[0018] In the formula, The batch size for the next time step. The batch size at the current moment. This is a preset adjustment factor;

[0019] The step of dynamically adjusting the batch size of a single data synchronization based on the current system load level includes:

[0020] When the load evaluation value is greater than or equal to the preset load threshold, the batch size is reduced; when the load evaluation value is less than the preset load threshold, the batch size is increased.

[0021] Optionally, the task attribute features include the amount of data to be synchronized, the task importance level, and the task time requirements;

[0022] The calculation of the scheduling weight of each concurrent synchronization task based on the task attribute characteristics includes:

[0023] Based on the amount of data to be synchronized, the importance level of the task, and the time requirement of the task, a scheduling score is calculated for each concurrent synchronization task, so as to determine the execution order of each concurrent synchronization task according to the scheduling score.

[0024] Optionally, the formula for calculating the scheduling score is:

[0025]

[0026] in, For the first The scheduling score of each concurrent synchronization task. For the first The amount of data to be synchronized for each of the concurrent synchronization tasks. This is a preset reference value for the maximum data volume. For the first The task importance levels of the concurrent synchronous tasks are described above. For the first The task time requirements for the aforementioned concurrent synchronous tasks Preset weighting coefficients corresponding to the amount of data to be synchronized, the importance level of the task, and the time requirement of the task, respectively;

[0027] The dynamic allocation of execution resources for each of the concurrent synchronization tasks includes:

[0028] The number of concurrent synchronization threads is dynamically adjusted based on the current number of database connections, write response time, and lock wait status of the target database, and the number of concurrent synchronization threads is allocated to each concurrent synchronization task according to the execution order.

[0029] Optionally, before synchronizing incremental data from the source database to the target database, the data synchronization control method further includes:

[0030] A synchronization point is constructed based on the current synchronization data location, data timestamp, and data verification information. The synchronization point is used to record the current synchronization data location.

[0031] Based on the synchronization point, perform an incremental scan on the source database to obtain newly added or changed data records as the incremental data;

[0032] After an abnormal interruption of the synchronization task, the synchronization task is resumed based on the recorded synchronization points.

[0033] Optionally, after synchronizing incremental data from the source database to the target database, the data synchronization control method further includes:

[0034] Perform a first-level fast verification, calculate the digest hash value of the corresponding data in the source database and the target database, and compare the hash difference between the two.

[0035] If the hash difference value is not zero, perform a second-level precise check and calculate the proportion of the number of difference fields to the total number of fields.

[0036] If the ratio is greater than or equal to a preset ratio threshold, compensation synchronization is triggered.

[0037] Optionally, synchronizing incremental data from the source database to the target database includes:

[0038] When writing the incremental data to the target database, the insertion operation is performed using a unique index;

[0039] When duplicate data is detected, an overwrite update operation is performed automatically.

[0040] A data synchronization control system for heterogeneous databases includes:

[0041] The parameter acquisition unit is used to acquire synchronization status parameters between the source database and the target database.

[0042] The adjustment unit is used to evaluate the current system load level based on the synchronous operation status parameters, and dynamically adjust the batch size of a single data synchronization according to the current system load level.

[0043] An execution resource determination unit is used to obtain the task attribute characteristics of multiple concurrent synchronous tasks, calculate the scheduling weight of each concurrent synchronous task based on the task attribute characteristics, and dynamically allocate the execution resources of each concurrent synchronous task in combination with the real-time load status of the target database.

[0044] The synchronization unit is used to synchronize incremental data in the source database to the target database based on the adjusted batch size and the allocated execution resources.

[0045] An electronic device, comprising: a memory and a processor;

[0046] The memory is used to store at least one instruction;

[0047] The processor is used to execute at least one instruction to implement the data synchronization control method between heterogeneous databases described above.

[0048] As can be seen from the above technical solution, this invention discloses a data synchronization control method, system, and electronic device for heterogeneous databases. It acquires synchronization operation status parameters between the source and target databases, assesses the current system load level based on these parameters, dynamically adjusts the batch size of a single data synchronization operation according to the current system load level, acquires the task attribute characteristics of multiple concurrent synchronization tasks, calculates the scheduling weight of each concurrent synchronization task based on these characteristics, and dynamically allocates execution resources for each concurrent synchronization task in conjunction with the real-time load status of the target database. Based on the adjusted batch size and allocated execution resources, incremental data from the source database is synchronized to the target database. This invention, by acquiring synchronization operation status parameters between the source and target databases, can perceive the link load in real time and dynamically adjust the batch size accordingly, avoiding the blocking risk caused by a fixed batch processing mode under high system load and the resource waste caused under low load. Meanwhile, by calculating scheduling weights based on task attribute characteristics and dynamically allocating execution resources in conjunction with the real-time load status of the target end, the system can intelligently schedule concurrent tasks according to business priorities and current system capacity. This ensures timely processing of critical tasks while preventing performance degradation of the target end database due to excessive resource contention. The above two adaptive control mechanisms work together to achieve the optimal balance between data transmission efficiency and system stability in complex and ever-changing heterogeneous database synchronization scenarios. Attached Figure Description

[0049] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the published drawings without creative effort.

[0050] Figure 1 This invention discloses a data synchronization control method between heterogeneous databases.

[0051] Figure 2 This is a schematic diagram of the structure of a data synchronization control device between heterogeneous databases disclosed in an embodiment of the present invention;

[0052] Figure 3 This is a schematic diagram of the structure of an electronic device disclosed in an embodiment of the present invention. Detailed Implementation

[0053] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0054] See Figure 1 The present invention discloses a flowchart of a data synchronization control method between heterogeneous databases, the method comprising:

[0055] Step S101: Obtain the synchronization status parameters between the source database and the target database.

[0056] In this embodiment, the source database is, for example, an HBase distributed database used for massive data storage, and the target database is, for example, a MySQL database used for business queries and relational storage.

[0057] Synchronization status parameters are a set of quantitative indicators that reflect the health of the current data synchronization link and resource pressure. They are mainly derived from the real-time monitoring interface or performance probes during system operation.

[0058] The synchronous running status parameters include the source cache queue length, the target average write time, the system processor resource utilization, and the current synchronization latency.

[0059] Synchronous running status parameters It is expressed as follows:

[0060] ;

[0061] In the formula, This parameter represents the length of the source cache queue, which refers to the number of pending data records in the source database, such as the HBase distributed columnar storage database, that are waiting to be read and synchronized to the target. This parameter quantifies the degree of data backlog and read pressure.

[0062] This represents the average write time on the target end. The average write time on the target end refers to the average response time required to write data to the target database, such as a MySQL relational database, within a preset time window. It is used to characterize the input / output (IO) performance and write bottlenecks of the target database.

[0063] This indicates the system processor resource utilization rate, which refers to the utilization rate of the central processing unit (CPU) of the server or computing node executing synchronous tasks, reflecting the busyness of computing resources.

[0064] This indicates the current synchronization delay, which refers to the time difference between when data is generated from the source and when it is successfully synchronized to the target. It is used to characterize the time lag between when data is generated and when synchronization is completed.

[0065] The synchronous operation status parameters construct a system status observation system from four dimensions: data backlog, write performance, computing resource consumption, and timeliness, which can accurately map the load pressure distribution during the synchronization process.

[0066] This embodiment reads performance counters by periodically calling the database monitoring API (Application Programming Interface) or by reporting heartbeat data packets through an agent program deployed on a key node of the synchronization link. The system can summarize the above parameters to fully perceive the current synchronization operation status and provide a data foundation for subsequent load assessment.

[0067] Step S102: Evaluate the current system load level based on the synchronous operation status parameters, and dynamically adjust the batch processing scale of a single data synchronization according to the current system load level.

[0068] System load level is a comprehensive assessment of the resource consumption of the current synchronization task. First, the acquired synchronization operation status parameters are normalized to eliminate the influence of dimensions. Then, based on a preset weighting strategy, the parameters are weighted and fused to obtain a load evaluation value that characterizes the current system busyness.

[0069] Batch size can refer to the number of data records or the size of data packets read from the source database and transmitted to the target database in a single synchronous operation.

[0070] This embodiment dynamically adjusts the batch size based on the load evaluation value, aiming to balance system throughput and stability: when the load evaluation value is high, the batch size is reduced to lower instantaneous I / O pressure and memory usage, preventing system congestion; when the load evaluation value is low, the batch size is increased to fully utilize idle resources and improve synchronous throughput efficiency. This adjustment process can be based, for example, on a preset feedback control algorithm, adjusting the batch size for the next moment proportionally according to the difference between the load evaluation value and a preset threshold; or it can employ a segmented adjustment strategy, directly switching to a preset fixed batch size when the load is in different ranges.

[0071] Step S103: Obtain the task attribute characteristics of multiple concurrent synchronization tasks, calculate the scheduling weight of each concurrent synchronization task based on the task attribute characteristics, and dynamically allocate the execution resources of each concurrent synchronization task in combination with the real-time load status of the target database.

[0072] Task attribute characteristics are a set of parameters describing the inherent characteristics and business requirements of concurrent synchronization tasks. Specifically, they may include the amount of data to be synchronized, the task's importance level, and the task's time requirements. The amount of data to be synchronized determines the task's processing time and resource consumption; the task's importance level reflects the priority definition at the business level; and the task's time requirements specify the deadline or timeliness standard for task completion. Based on these task attribute characteristics, for example, a weighted scoring model can be used, taking the amount of data to be synchronized, the task's importance level, and the task's time requirements as input variables. A linear weighting or non-linear mapping can be used to calculate the scheduling weight of each task; a higher weight indicates a higher priority.

[0073] Execution resources mainly include database connection count, thread resources, I / O bandwidth, and memory buffers. When dynamically allocating execution resources, a dual constraint is applied based on the task's scheduling weight and the real-time load status of the target database, such as the number of active connections, write response latency, and lock waiting status. For tasks with high importance and nearing their deadlines, more database connection threads and higher I / O bandwidth are allocated; for low-priority tasks with large amounts of data to be synchronized, the number of concurrent threads is limited, and they are scheduled for execution during system idle periods. Therefore, this invention can ensure that resources are prioritized for high-weight tasks without overloading the target database.

[0074] Step S104: Based on the adjusted batch processing size and the allocated execution resources, synchronize the incremental data in the source database to the target database.

[0075] Incremental data can refer to data records that have been added or whose content has changed in the source database after the last synchronization was completed.

[0076] Based on the batch processing scale determined in step S102, the synchronization task extracts incremental data from the source database (such as HBase) in batches. For heterogeneous database scenarios, due to the differences in data structures between the source and target ends (e.g., HBase uses columnar storage, while MySQL uses a relational table structure), data cleaning, format conversion, and field mapping are required before data is written.

[0077] In practical applications, there is a field mapping relationship between HBase and MySQL. The synchronization process accurately maps the column families and column names in HBase to the corresponding fields in the MySQL table to ensure the correctness of data semantics.

[0078] HBase and MySQL field mapping relationship as follows:

[0079] ;

[0080] In the formula, Represents the HBase source field. This represents the target field in MySQL. This indicates the field conversion rules.

[0081] After data transformation is complete, the processed data is written in batches to the target database (such as MySQL) using the execution resources allocated in step S103. During the writing process, for example, batch insert statements are used to reduce network interaction overhead, or transaction mechanisms are used to ensure the atomic writing of the same batch of data. By adjusting the batch size, the impact on the target database is avoided; by making reasonable use of the allocated execution resources, multiple concurrent tasks are ensured to compete for system resources in an orderly and efficient manner, thereby achieving efficient and reliable synchronization of data between heterogeneous databases.

[0082] In summary, this invention discloses a data synchronization control method between heterogeneous databases. It acquires synchronization operation status parameters between the source and target databases, assesses the current system load level based on these parameters, dynamically adjusts the batch size of a single data synchronization operation according to the current system load level, acquires task attribute characteristics of multiple concurrent synchronization tasks, calculates the scheduling weight of each concurrent synchronization task based on these characteristics, and dynamically allocates execution resources for each concurrent synchronization task in conjunction with the real-time load status of the target database. Based on the adjusted batch size and allocated execution resources, incremental data from the source database is synchronized to the target database. This invention, by acquiring synchronization operation status parameters between the source and target databases, can perceive the link load in real time and dynamically adjust the batch size accordingly, avoiding the blocking risk caused by a fixed batch processing mode under high system load and the resource waste caused under low load. Simultaneously, by calculating scheduling weights based on task attribute characteristics and dynamically allocating execution resources in conjunction with the real-time load status of the target database, the system can intelligently schedule concurrent tasks according to business priorities and current system capacity, ensuring timely processing of critical tasks and preventing performance degradation of the target database due to excessive resource contention. The two adaptive control mechanisms mentioned above work together to achieve the optimal balance between data transmission efficiency and system stability in complex and ever-changing heterogeneous database synchronization scenarios.

[0083] In one embodiment, the process of assessing the current system load level based on synchronous operating state parameters specifically includes:

[0084] The parameters in the synchronous operation status parameters are normalized, and a weighted fusion is performed based on the normalized parameters to obtain a load evaluation value that characterizes the current system load level.

[0085] Given that synchronous operation parameters have different physical dimensions and value ranges—for example, the source-side cache queue length may be as high as tens of thousands of entries, while the system processor resource utilization typically fluctuates between 0 and 100—direct numerical calculations would allow parameters with larger magnitudes to dominate the evaluation results, thus masking changes in smaller but critical parameters. Therefore, it is necessary to perform normalization on all parameters, mapping parameters with different dimensions to the same dimensionless numerical range. This eliminates the impact of magnitude differences on the evaluation results and ensures the comparability of each parameter in subsequent weighted calculations.

[0086] In practical implementation, the min-max standardization method can be used for normalization. For each collected parameter's original value, the difference between it and the preset minimum value is calculated, and then divided by the difference between the preset maximum value and the preset minimum value to obtain the normalized value. This calculation process can be formally represented as:

[0087] ;

[0088] In the formula, Indicates the first The original collected values ​​of the synchronous operation status parameters of the item. This represents the preset minimum value of the synchronous operation status parameter under normal operating conditions. This indicates the preset maximum value of the synchronous operation status parameter under normal operating conditions. This represents a dimensionless value after normalization, whose range is, for example, limited to the interval [0, 1].

[0089] and This can be determined based on the statistical distribution of historical operating data, or set by those skilled in the art based on experience. For example, regarding the average write time at the target end, It can be set to the fastest write time under ideal conditions. It can be set as the maximum allowed write time threshold. Through linear transformation, the parameters are converted into a standardized scoring form, providing a unified data foundation for subsequent fusion computing.

[0090] This embodiment, after obtaining the normalized values ​​of various parameters, further performs weighted fusion on these values ​​to generate a single indicator that comprehensively reflects the current system load level. The weighted fusion assigns corresponding weight coefficients to each normalized parameter based on the importance of different parameters to the overall system load, and then sums the weighted results. This process uses a mathematical model to reduce the dimensionality of multi-dimensional state information into a scalar value, simplifying the complexity of load assessment and facilitating subsequent dynamic decision-making based on this scalar value.

[0091] The formula for calculating the load evaluation value is as follows:

[0092] ;

[0093] In the formula, for The higher the load evaluation value at any given time, the higher the current load level of the system. The normalized source-side cache queue length, Average write time to the target end For system processor resource utilization and For the current synchronization delay, They represent Preset weight parameters, and satisfy The constraints and preset weight parameters can be set based on expert experience or obtained through training with historical data. For example, if the write performance of the target end is the main bottleneck, the weight can be appropriately increased. The value of makes the load evaluation value more sensitive to changes in write latency.

[0094] Through the weighted fusion calculation described above, multiple previously scattered and inconsistent operational status parameters are aggregated into a comprehensive index with clear physical meaning. This load evaluation value provides a quantitative basis for subsequent adjustments to the batch processing scale of single data synchronization, enabling the system to perform refined and adaptive control based on real-time load status.

[0095] The formula for adjusting the batch size is:

[0096] ;

[0097] In the formula, The batch size for the next time step. This represents the current batch size. This is a preset adjustment factor used to control the step size or intensity of the adjustment. Its value range can be set, for example, between (0,1), and the specific value can be calibrated according to the system's stability requirements. When the value is small, the adjustment process is relatively smooth, which can avoid system oscillations caused by drastic fluctuations in batch size; when When the value is large, the system responds faster to load changes and can quickly adapt to sudden traffic surges.

[0098] The adjustment formula establishes a negative feedback mechanism: when the load evaluation value... When it rises, The number of items is reduced, thus reducing the batch size at the next time step. Compared to the present The system shrinks when the load assessment value decreases and expands accordingly. Through this recursive calculation based on real-time feedback, the optimal throughput configuration for the current system state can be automatically found, avoiding resource waste caused by a fixed batch size under low load or exacerbating database congestion under high load.

[0099] In one embodiment, the process of dynamically adjusting the batch size of a single data synchronization based on the current system load level may specifically include:

[0100] When the load evaluation value is greater than or equal to the preset load threshold, the batch size is reduced; when the load evaluation value is less than the preset load threshold, the batch size is increased.

[0101] Preset load threshold It serves as a decision-making watershed for triggering batch size adjustments, used to determine whether the current system load is overloaded or lightly loaded. The preset load threshold can be determined, for example, based on system stress test results, i.e., by gradually increasing the concurrent load until the system throughput begins to decrease or the response time significantly increases, which is then used as the threshold; or it can be derived by working backward from the maximum permissible latency requirements in the service level agreement.

[0102] When the judgment Greater than or equal to the preset load threshold When this occurs, it indicates that system resources are strained or there is a risk of processing backlog. In this case, the strategy tends to reduce... This reduces the instantaneous consumption of memory, CPU, and database input / output resources by a single synchronization task, thereby alleviating system pressure; conversely, when Less than the preset load threshold When this occurs, it indicates that the system still has redundant processing capacity, and the strategy tends to increase... To fully utilize network bandwidth and idle database resources, the overall data synchronization throughput efficiency is improved. Based on this, the invention achieves bidirectional dynamic adjustment of batch processing size, ensuring the continuous and efficient operation of the heterogeneous database synchronization process under different load levels.

[0103] In this embodiment, the calculation of load evaluation value and the dynamic adjustment of batch size work together to form a closed-loop control mechanism based on real-time perception. By accurately quantifying the system load and directly mapping it to the adjustment parameters of batch size, this technical solution effectively solves the problem that traditional fixed batch strategies are difficult to adapt to dynamic traffic changes, enabling the data synchronization system to achieve a dynamic balance between resource constraints and performance maximization.

[0104] In one embodiment, task attribute characteristics include the amount of data to be synchronized, the task importance level, and the task time requirement.

[0105] Task attribute features are a comprehensive set of indicators used to quantify the resource requirements and service quality requirements of concurrent synchronous tasks across different dimensions. Compared to the single-dimensional processing method in existing technologies that only uses task submission time for First In First Out (FIFO) scheduling, task attribute features, through multi-dimensional feature collection, can more precisely characterize the business value and urgency of different synchronous tasks.

[0106] The amount of data to be synchronized represents the total number or volume of data records to be transferred in the current task. This is calculated by scanning the row key range of the source HBase table to obtain the number of rows. This data volume affects the task's requirements for network bandwidth and intermediate cache resources.

[0107] Task importance level is used to characterize the priority attribute of the business carried by the current task. The classification method includes dividing tasks into critical business tasks, important business tasks, and general business tasks. This level is specified by the upper-layer application system according to the purpose of the data.

[0108] Task time requirements characterize the constraints on task completion time, including the expected deadline or the maximum tolerable delay. These dimensions provide the data input basis for subsequent differentiated scheduling strategies.

[0109] In one alternative implementation, a task priority evaluation model is established to address the task competition problem when multiple locomotives and multiple data tables are synchronized simultaneously.

[0110] Definition of the first The synchronization task of the concurrent synchronization task is :

[0111] ;

[0112] in, For the first The amount of data to be synchronized for each concurrent synchronization task. For the first The task importance level of concurrent synchronous tasks. For the first The time requirements for concurrent synchronous tasks.

[0113] Substituting the above definition into the scheduling scoring formula, the scheduling weight of each task is calculated based on the feature parameters of the three dimensions to determine the execution order of each concurrent synchronous task.

[0114] In one embodiment, the process of calculating the scheduling weight of each concurrent synchronous task based on task attribute characteristics specifically includes:

[0115] Based on the amount of data to be synchronized, the importance level of the task, and the time requirement of the task, a scheduling score is calculated for each concurrent synchronization task, so as to determine the execution order of each concurrent synchronization task according to the scheduling score.

[0116] Based on the acquired task attribute characteristics, each concurrent synchronous task to be executed is quantitatively scored, thereby transforming abstract task attributes into comparable scheduling weight values. The calculation process of scheduling weights aims to balance the relationship between task throughput, business priority, and timeliness requirements, avoiding a decline in overall scheduling efficiency due to extreme values ​​of a single indicator.

[0117] The formula for calculating the scheduling score is as follows:

[0118]

[0119] in, For the first The scheduling score of each concurrent synchronization task. For the first The amount of data to be synchronized for each of the concurrent synchronization tasks. This is a preset maximum data volume reference value used to normalize the data volume to the [0, 1] interval. For the first The importance level of the concurrent synchronization tasks can be set, for example, a higher value indicates higher priority. For the first The time requirement for each concurrent synchronous task can be, for example, the remaining time until the deadline, expressed in seconds or minutes. , , These represent the preset weighting coefficients for the amount of data to be synchronized, the task importance level, and the task time requirement, respectively. For example, they can be set to... , , To prioritize the execution of high-priority tasks.

[0120] Among them, when the first Scheduling score for concurrent synchronous tasks Greater than the Scheduling score of the concurrent synchronization tasks When, prioritize scheduling the first A concurrent synchronous task.

[0121] Scheduling score It takes into account task size, business priority, and time urgency. Based on the calculated scheduling score, the execution order of each task is further determined, i.e., according to the scheduling score. Concurrent synchronization tasks are ranked from highest to lowest score, with higher-scoring tasks receiving priority allocation of computing and database write resources. In the implementation, the system iterates through all currently scheduled tasks and calculates the performance score for each task according to the formula described above. The system calculates the value and constructs a priority queue with a max-heap structure. The scheduler then retrieves tasks from the top of the heap for execution. This multi-feature fusion-based scoring mechanism enables the system to accurately identify and prioritize critical and urgent tasks when faced with a large number of concurrent synchronous requests, thereby ensuring the real-time performance of core business operations while optimizing the overall system's resource utilization efficiency.

[0122] In one embodiment, the process of dynamically allocating execution resources for each concurrent synchronous task may specifically include:

[0123] The number of concurrent synchronization threads is dynamically adjusted based on the current number of database connections, write response time, and lock wait status of the target database, and the number of concurrent synchronization threads is allocated to each concurrent synchronization task according to the execution order.

[0124] After determining the scheduling score and execution order of each task, instead of simply starting a fixed number of synchronization threads in sequence, the number of concurrent synchronization threads is dynamically scaled up or down by sensing the real-time load status of the target database. This aims to prevent the database performance from abruptly collapsing due to excessive concurrent threads, which could lead to exhaustion of the target database connection pool, severe write I / O contention, or frequent lock conflicts. At the same time, it avoids wasting system computing resources due to too few concurrent threads, resulting in low data synchronization throughput.

[0125] The real-time load status of the target database is primarily characterized by three core metrics: the current number of database connections, write response time, and lock wait status. The current number of database connections reflects the number of sessions the database is currently processing. If this number approaches or reaches the maximum connection limit configured in the database, it indicates that the database is under high load saturation. In this case, increasing the number of concurrent threads will very likely lead to connection request failures or timeouts. Write response time refers to the average time taken for synchronous threads to perform data write operations. This metric directly reflects the busyness of the database I / O subsystem. A significant increase in write response time usually means that disk I / O bandwidth has become a bottleneck, and further increasing concurrency will only exacerbate queue backlog. Lock wait status reflects the intensity of resource competition among concurrent transactions. If there are a large number of lock requests that are in a waiting state for a long time, it indicates that high concurrency has led to severe row lock or table lock conflicts, requiring a reduction in concurrency to mitigate the risk of deadlock.

[0126] After determining the number of concurrent threads the system can handle at the current moment, the limited thread resources are allocated to each concurrent synchronization task according to the execution order determined by the scheduling score calculated in the preceding steps. For example, for the task at the head of the queue, one or more concurrent synchronization threads are allocated or reserved for it first, allowing it to immediately start or continue executing data synchronization operations; for tasks in the subsequent queue, they are suspended or placed in a waiting queue until thread resources are released. By combining this allocation strategy, which combines task attribute-based scoring and ranking with total resource control based on the real-time state of the database, this application can achieve efficient scheduling of multiple concurrent synchronization tasks while ensuring the stability of the target database, ensuring that high-priority tasks are processed in a timely manner, and avoiding system resource overload.

[0127] This embodiment guides task sorting through a quantitative scoring formula and adaptively adjusts concurrency by combining real-time monitoring of multi-dimensional load indicators of the target database. This effectively resolves the contradiction between resource contention and database stability in multi-task concurrent scenarios and achieves a dual improvement in data synchronization efficiency and reliability between heterogeneous databases.

[0128] In one embodiment, before synchronizing incremental data from the source database to the target database, the data synchronization control method may further include:

[0129] (1) A synchronization point is constructed based on the current synchronization data location, data timestamp and data verification information. The synchronization point is used to record the current synchronization data location.

[0130] Synchronization points, as key data structures marking data transmission progress during heterogeneous database synchronization, are fundamental for incremental fetching and breakpoint resumption. During the synchronization task startup phase or after each round of data synchronization, the system constructs and persistently saves the current synchronization point. The construction process begins by reading the primary key identifier of the last successfully synchronized data record from the source database. For example, in HBase, this primary key identifier can be a RowKey formed by combining the locomotive number and the collection time. Next, the timestamp information corresponding to this data record is obtained, representing the precise time point of data generation or update. Simultaneously, the verification information for this data record is calculated. This verification information can be a digest value obtained by calculating the key field values ​​using hash algorithms such as MD5 or SHA-256, used for subsequent data consistency verification. Finally, the primary key identifier, timestamp, and verification information are encapsulated into a synchronization point object according to a predefined structure and stored in a persistent storage medium such as a local configuration file, metadata database, or Redis cache.

[0131] Through the above construction process, the synchronization point becomes a reliable basis for recording the current synchronization data position, enabling the system to accurately distinguish between processed data and data to be synchronized.

[0132] Synchronization sites The expression is as follows:

[0133] ;

[0134] In the formula, Indicates the current location of synchronized data. Represents data timestamps, This indicates data verification information.

[0135] The synchronization program performs an incremental scan from HBase based on the synchronization point information, acquiring only newly added or changed data records, thus avoiding duplicate data readings caused by traditional full synchronization. Simultaneously, by recording the point information after each successful synchronization, it enables abnormal recovery and resume functionality of the synchronization task.

[0136] It should be noted that before synchronizing incremental data from the source database to the target database, the read data undergoes a structured transformation for the HBase column-family storage structure. Specifically, this includes:

[0137] RowKey parsing, field mapping processing, and data standardization processing.

[0138] RowKey parsing is performed according to a predefined rule: RowKey = train_id + timestamp. The train number (train_id) and the data collection time (timestamp) are obtained by parsing the RowKey. Field mapping establishes the correspondence between HBase source fields and MySQL target fields.

[0139] Data standardization processing specifically includes: time format conversion, converting time fields in HBase to date and time formats supported by MySQL; numeric type conversion, converting string numeric values ​​in HBase to the corresponding numeric types in MySQL; null value handling, filling missing or null fields in HBase with default values ​​or marking them; and outlier filtering, removing or marking abnormal data that exceeds reasonable limits or does not conform to business logic.

[0140] The above processing transforms unstructured data in HBase into data objects that conform to the MySQL table structure.

[0141] (2) Perform an incremental scan on the source database based on the synchronization point to obtain newly added or changed data records as the incremental data.

[0142] When performing data synchronization, incremental scans are performed on the source database based on pre-built synchronization points to avoid the resource overhead of full scans.

[0143] The specific scanning logic first reads the previously saved synchronization point from persistent storage and parses out the primary key identifier and timestamp. Using this information as query conditions, a scan request is constructed, initiating a data read operation to the source database. For distributed columnar storage databases such as HBase, incremental scans can set the starting row key to the primary key identifier in the synchronization point and use a timestamp filter to select data records with timestamps greater than that point's timestamp.

[0144] During the scanning process, the system sequentially traverses data records that meet the criteria, extracting those records that have been added or changed since the last synchronization, and using them as incremental data for the current synchronization task. This directional scanning method based on synchronization points can efficiently and accurately obtain the subset of data that needs to be synchronized, significantly reducing the amount of data transmission and processing load.

[0145] (3) After the synchronization task is abnormally interrupted, the synchronization task is resumed based on the recorded synchronization points.

[0146] In real-world operating environments, data synchronization tasks may be unexpectedly interrupted due to network fluctuations, database service unavailability, or abnormal application exits. To avoid data loss or duplicate transmissions, breakpoint resume functionality is implemented through synchronization points.

[0147] When the system detects an abnormal interruption of the synchronization task and attempts to restart it, it first reads the latest synchronization point recorded in persistent storage. This point accurately records the position information of the last successfully processed data before the interruption. Subsequently, the system reinitializes the synchronization task based on this point, using it as the starting point for incremental scanning. The resumed synchronization task continues scanning and transmitting data directly from the point of interruption, without resynchronizing previously processed historical data or missing newly added data after the interruption. This ensures the continuity and integrity of data transmission, giving the synchronization system fault tolerance and self-recovery capabilities in the face of various anomalies, thereby guaranteeing the consistency and reliability of data between heterogeneous databases.

[0148] In one embodiment, after synchronizing incremental data from the source database to the target database, the data synchronization control method may further include:

[0149] Perform a first-level fast verification, calculate the digest hash value of the corresponding data in the source database and the target database, and compare the hash difference between the two.

[0150] If the hash difference value is not zero, perform a second-level precise check and calculate the proportion of the number of difference fields to the total number of fields.

[0151] If the ratio is greater than or equal to a preset ratio threshold, compensation synchronization is triggered.

[0152] After data synchronization is complete, to ensure data consistency between the source database (e.g., HBase) and the target database (e.g., MySQL), a first-level fast check is performed. This first-level fast check calculates the hash values ​​of corresponding data in the source and target databases and compares the hash differences between them. It aims to quickly identify records that may have data inconsistencies with minimal computational overhead.

[0153] Level 1 Fast Verification: Calculate the digest hash value:

[0154] ;

[0155] In the formula, This represents the summary hash value of the data record. Hash represents the hash operation function, RowKey represents the unique primary key identifier of the data record, and Value represents the core business field value of the data record.

[0156] Digest hash value It is a fixed-length fingerprint generated by hashing key fields, such as primary keys, timestamps, and core business field values, which can efficiently represent the uniqueness of data content.

[0157] Calculate the digest hash value before and after synchronization, and compare the differences between the two:

[0158] ;

[0159] In the formula, This represents the difference between the source digest hash value and the destination digest hash value. This represents the source digest hash value calculated from the source HBase database. This represents the target digest hash value calculated by the target MySQL database.

[0160] like If the value is 0, the data is considered consistent.

[0161] Specifically, the key information of the corresponding data records in the source and target databases is obtained, including the record's unique identifier (RowKey) and core attribute values, and then input into a hash function for calculation. Subsequently, the hash value calculated from the source database is logically compared with the hash value of the corresponding record in the target database. If they are completely identical, the data is considered consistent; if they are different, a discrepancy is determined.

[0162] When a non-zero hash difference value is detected, it indicates a potential inconsistency between data records, but the specific differing fields and their extent cannot be determined at this point. To further accurately assess the range of data deviation, a second-level precise check is required.

[0163] Second-level exact validation: For data that fails the first-level validation, a further field-level comparison is performed.

[0164] ;

[0165] In the formula, This represents the proportion of differences between the number of different fields and the total number of different fields. For the number of difference fields, This represents the total number of fields.

[0166] when When the value is greater than θ, compensation synchronization is triggered, where θ is a preset proportional threshold.

[0167] By combining rapid verification with precise verification, both efficiency and accuracy of consistency detection can be achieved in large-scale data synchronization scenarios.

[0168] In this embodiment, the ratio of the number of differing fields to the total number of fields is used as a quantitative indicator to measure the coverage of inconsistent fields in a single record relative to the entire record. This ratio is obtained by comparing the specific values ​​of the source and target data records field by field, counting the number of fields with inconsistent values, and dividing that number by the total number of fields in the record.

[0169] Based on the calculated discrepancy ratio, the severity of the data discrepancy needs to be further assessed to determine the subsequent processing strategy. The preset ratio threshold is the critical criterion for distinguishing between acceptable data deviation and data errors that need to be corrected. Its setting is usually determined based on the business's tolerance for data accuracy and the system resource cost.

[0170] If the difference ratio exceeds a preset threshold, it indicates that critical information in the record has been severely misaligned or lost. In this case, compensation synchronization must be triggered to correct the data on the target end. Compensation synchronization can refer to retrieving the complete data of the record from the source database and performing an overwrite update operation to ensure that the data in the target database is consistent with that of the source database. Through threshold-based hierarchical processing, this invention can effectively avoid frequent retransmission processes due to minor fluctuations or differences in non-critical fields, thereby ensuring data accuracy while minimizing the resource consumption and operating overhead of the synchronization system.

[0171] In summary, this embodiment combines primary fast verification with secondary precise verification to construct a complete verification chain from coarse-grained screening to fine-grained confirmation. The comparison of hash differences can quickly filter massive amounts of consistent data, while the calculation of the proportion of differing fields provides an objective standard for determining whether data needs repair. This synergy enables the system to achieve precise monitoring of data consistency with low computational cost during heterogeneous database synchronization, and automatically repairs severe data deviations by triggering a compensation synchronization mechanism, ensuring high reliability and high fidelity of the locomotive's big data flow between HBase and MySQL.

[0172] In one embodiment, the process of synchronizing incremental data from the source database to the target database may include:

[0173] When writing the incremental data to the target database, the insertion operation is performed using a unique index;

[0174] When duplicate data is detected, an overwrite update operation is performed automatically.

[0175] The unique index is built based on key fields of the source data. These key fields include a combination of information such as locomotive number, collection timestamp, and data version number that can uniquely identify a data record.

[0176] During the actual data writing process, incremental data records containing key fields are constructed into insert statements (such as SQL statements) that conform to the target database syntax, and a write request is sent to the database. At this time, the database query optimizer automatically uses the unique index to quickly retrieve and detect conflicts in the record to be written. This process is not a simple data append, but rather an efficient traversal of the index tree to determine whether the key of the record to be written already exists in the storage engine. If the unique index does not find a conflict, the database directly executes the insertion operation of the new record and persists the data to disk; if the unique index detects that the key already exists, the database returns the corresponding status signal according to the pre-configured conflict handling strategy, instead of directly throwing a primary key conflict exception.

[0177] By utilizing unique indexes to perform insert operations, atomicity determination of data duplication can be achieved at the underlying kernel level of the database, providing accurate execution branch basis for subsequent data processing logic, thereby ensuring the robustness of the synchronization process.

[0178] When a unique index detects that a record with the same key as the incremental data to be written already exists in the target database, an overwrite update mechanism is automatically triggered, rather than skipped or an error reported. This overwrite update operation utilizes specific database syntax features, such as the INSERT INTO table ON DUPLICATE KEY UPDATE statement in MySQL. After receiving the write request, if the database determines that the data is duplicated, it automatically switches the execution logic of the SQL statement from insert to update, that is, it replaces the corresponding field values ​​of the existing record in the target database with the field values ​​from the newly received incremental data.

[0179] This embodiment utilizes the synergistic effect of unique index constraints and idempotent update logic to quickly identify data status by leveraging the database's native index detection capabilities. It also combines an automatic overwrite strategy to flexibly handle duplicate write requests, thereby eliminating the risk of data conflicts during heterogeneous database synchronization and achieving efficient and accurate data migration.

[0180] Corresponding to the above method embodiments, the present invention also discloses a data synchronization control system between heterogeneous databases.

[0181] See Figure 2The present invention discloses a schematic diagram of a data synchronization control system between heterogeneous databases, the system comprising:

[0182] The parameter acquisition unit 201 is used to acquire the synchronization status parameters between the source database and the target database.

[0183] In this embodiment, the source database is, for example, an HBase distributed database used for massive data storage, and the target database is, for example, a MySQL database used for business queries and relational storage.

[0184] The adjustment unit 202 is used to evaluate the current system load level based on the synchronous operation status parameters, and dynamically adjust the batch processing scale of a single data synchronization according to the current system load level.

[0185] The execution resource determination unit 203 is used to obtain the task attribute characteristics of multiple concurrent synchronous tasks, calculate the scheduling weight of each concurrent synchronous task based on the task attribute characteristics, and dynamically allocate the execution resources of each concurrent synchronous task in combination with the real-time load status of the target database.

[0186] Synchronization unit 204 is used to synchronize incremental data in the source database to the target database based on the adjusted batch size and the allocated execution resources.

[0187] In summary, this invention discloses a data synchronization control device for heterogeneous databases. It acquires synchronization operation status parameters between the source and target databases, assesses the current system load level based on these parameters, dynamically adjusts the batch size of a single data synchronization operation according to the current system load level, acquires task attribute characteristics of multiple concurrent synchronization tasks, calculates the scheduling weight of each concurrent synchronization task based on these characteristics, and dynamically allocates execution resources for each concurrent synchronization task in conjunction with the real-time load status of the target database. Based on the adjusted batch size and allocated execution resources, incremental data from the source database is synchronized to the target database. This invention, by acquiring synchronization operation status parameters between the source and target databases, can perceive the link load in real time and dynamically adjust the batch size accordingly, avoiding the blocking risk caused by a fixed batch processing mode under high system load and the resource waste caused under low load. Simultaneously, by calculating scheduling weights based on task attribute characteristics and dynamically allocating execution resources in conjunction with the real-time load status of the target database, the system can intelligently schedule concurrent tasks according to business priorities and the current system capacity, ensuring timely processing of critical tasks and preventing performance degradation of the target database due to excessive resource contention. The two adaptive control mechanisms mentioned above work together to achieve the optimal balance between data transmission efficiency and system stability in complex and ever-changing heterogeneous database synchronization scenarios.

[0188] In one embodiment, the adjustment unit 202 can be specifically used for:

[0189] The parameters in the synchronous operation status parameters are normalized, and a weighted fusion is performed based on the normalized parameters to obtain a load evaluation value that characterizes the current system load level.

[0190] The formula for calculating the load evaluation value is as follows:

[0191] ;

[0192] In the formula, for The load evaluation value at that time. The normalized source-side cache queue length, The average write time to the target end, The system processor resource utilization and The current synchronization delay, They represent Preset weight parameters, and satisfy Constraints;

[0193] The formula for adjusting the batch size is:

[0194] ;

[0195] In the formula, The batch size for the next time step. The batch size at the current moment. This is a preset adjustment factor;

[0196] Correspondingly, the adjustment unit 202 can also be used for:

[0197] When the load evaluation value is greater than or equal to the preset load threshold, the batch size is reduced; when the load evaluation value is less than the preset load threshold, the batch size is increased.

[0198] In one embodiment, task attribute characteristics include the amount of data to be synchronized, the task importance level, and the task time requirement.

[0199] Correspondingly, the execution resource determination unit 203 can specifically be used for:

[0200] Based on the amount of data to be synchronized, the importance level of the task, and the time requirement of the task, a scheduling score is calculated for each concurrent synchronization task, so as to determine the execution order of each concurrent synchronization task according to the scheduling score.

[0201] The formula for calculating the scheduling score is:

[0202]

[0203] in, For the first The scheduling score of each concurrent synchronization task. For the first The amount of data to be synchronized for each of the concurrent synchronization tasks. This is a preset reference value for the maximum data volume. For the first The task importance levels of the concurrent synchronous tasks are described above. For the first The task time requirements for the aforementioned concurrent synchronous tasks These are preset weighting coefficients corresponding to the amount of data to be synchronized, the importance level of the task, and the time requirement of the task, respectively.

[0204] Correspondingly, the execution resource determination unit 203 can specifically be used for:

[0205] The number of concurrent synchronization threads is dynamically adjusted based on the current number of database connections, write response time, and lock wait status of the target database, and the number of concurrent synchronization threads is allocated to each concurrent synchronization task according to the execution order.

[0206] In one embodiment, the data synchronization control system may further include:

[0207] A synchronization point construction unit is used to construct a synchronization point based on the current synchronization data position, data timestamp, and data verification information before synchronizing incremental data from the source database to the target database. The synchronization point is used to record the current synchronization data position.

[0208] The scanning unit is used to perform an incremental scan on the source database based on the synchronization point to obtain newly added or changed data records as the incremental data;

[0209] The task recovery unit is used to recover the synchronization task based on the recorded synchronization points after the synchronization task is abnormally interrupted.

[0210] In one embodiment, the data synchronization control system may further include:

[0211] The first-level verification unit is used to perform a first-level fast verification after synchronizing incremental data from the source database to the target database, calculate the digest hash value of the corresponding data in the source database and the target database, and compare the hash difference between the two.

[0212] The secondary verification unit is used to perform a secondary precise verification if the hash difference value is not zero, and to calculate the proportion of the number of difference fields to the total number of fields.

[0213] A synchronization triggering unit is used to trigger compensation synchronization if the ratio is greater than or equal to a preset ratio threshold.

[0214] In one embodiment, the synchronization unit 204 can be specifically used for:

[0215] When writing the incremental data to the target database, the insertion operation is performed using a unique index;

[0216] When duplicate data is detected, an overwrite update operation is performed automatically.

[0217] It should be noted that for the specific working principles of each component in the device embodiment, please refer to the corresponding section of the method embodiment, which will not be repeated here.

[0218] Corresponding to the above embodiments, such as Figure 3 As shown, the present invention also provides a schematic diagram of the structure of an electronic device, which may include: a processor 1 and a memory 2;

[0219] The processor 1 and memory 2 communicate with each other via communication bus 3.

[0220] Processor 1, for executing at least one instruction;

[0221] Memory 2 is used to store at least one instruction;

[0222] Processor 1 may be a central processing unit (CPU), an application specific integrated circuit (ASIC), or one or more integrated circuits configured to implement embodiments of the present invention.

[0223] Memory 2 may include high-speed RAM memory, and may also include non-volatile memory, such as at least one disk storage device.

[0224] In this embodiment, processor 1 executes at least one instruction to implement the steps shown in the data synchronization control method for heterogeneous databases.

[0225] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0226] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.

[0227] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A data synchronization control method between heterogeneous databases, characterized in that, include: Obtain the synchronization status parameters between the source database and the target database; The current system load level is assessed based on the synchronous operation status parameters, and the batch size of a single data synchronization is dynamically adjusted according to the current system load level. Obtain the task attribute characteristics of multiple concurrent synchronous tasks, calculate the scheduling weight of each concurrent synchronous task based on the task attribute characteristics, and dynamically allocate the execution resources of each concurrent synchronous task in combination with the real-time load status of the target database. Based on the adjusted batch size and allocated execution resources, incremental data in the source database is synchronized to the target database.

2. The data synchronization control method according to claim 1, characterized in that, The synchronization operation status parameters include the source-side cache queue length, the target-side average write time, the system processor resource utilization rate, and the current synchronization latency. The assessment of the current system load level based on the synchronized operating status parameters includes: The parameters in the synchronous operation status parameters are normalized, and a weighted fusion is performed based on the normalized parameters to obtain a load evaluation value that characterizes the current system load level.

3. The data synchronization control method according to claim 2, characterized in that, The formula for calculating the load evaluation value is: ； In the formula, for The load evaluation value at that time, The normalized source-side cache queue length, The average write time to the target end, The system processor resource utilization and The current synchronization delay, They represent Preset weight parameters, and satisfy Constraints; The formula for adjusting the batch size is: ； In the formula, The batch size for the next time step. The batch size at the current moment. This is a preset adjustment factor; The step of dynamically adjusting the batch size of a single data synchronization based on the current system load level includes: When the load evaluation value is greater than or equal to the preset load threshold, the batch size is reduced. When the load evaluation value is less than the preset load threshold, the batch processing size is increased.

4. The data synchronization control method according to claim 1, characterized in that, The task attribute characteristics include the amount of data to be synchronized, the task importance level, and the task time requirements; The calculation of the scheduling weight of each concurrent synchronization task based on the task attribute characteristics includes: Based on the amount of data to be synchronized, the importance level of the task, and the time requirement of the task, a scheduling score is calculated for each concurrent synchronization task, so as to determine the execution order of each concurrent synchronization task according to the scheduling score.

5. The data synchronization control method according to claim 4, characterized in that, The formula for calculating the scheduling score is as follows: in, For the first The scheduling score of the concurrent synchronization tasks. For the first The amount of data to be synchronized for each of the concurrent synchronization tasks. This is a preset reference value for the maximum data volume. For the first The task importance levels of the concurrent synchronous tasks are described above. For the first The task time requirements for the aforementioned concurrent synchronous tasks Preset weighting coefficients corresponding to the amount of data to be synchronized, the importance level of the task, and the time requirement of the task, respectively; The dynamic allocation of execution resources for each of the concurrent synchronization tasks includes: The number of concurrent synchronization threads is dynamically adjusted based on the current number of database connections, write response time, and lock wait status of the target database, and the number of concurrent synchronization threads is allocated to each concurrent synchronization task according to the execution order.

6. The data synchronization control method according to claim 1, characterized in that, Before synchronizing incremental data from the source database to the target database, the data synchronization control method further includes: A synchronization point is constructed based on the current synchronization data location, data timestamp, and data verification information. The synchronization point is used to record the current synchronization data location. Based on the synchronization point, perform an incremental scan on the source database to obtain newly added or changed data records as the incremental data; After an abnormal interruption of the synchronization task, the synchronization task is resumed based on the recorded synchronization points.

7. The data synchronization control method according to claim 6, characterized in that, After synchronizing incremental data from the source database to the target database, the data synchronization control method further includes: Perform a first-level fast verification, calculate the digest hash value of the corresponding data in the source database and the target database, and compare the hash difference between the two. If the hash difference value is not zero, perform a second-level precise check and calculate the proportion of the number of difference fields to the total number of fields. If the ratio is greater than or equal to a preset ratio threshold, compensation synchronization is triggered.

8. The data synchronization control method according to claim 1, characterized in that, The step of synchronizing incremental data from the source database to the target database includes: When writing the incremental data to the target database, the insertion operation is performed using a unique index; When duplicate data is detected, an overwrite update operation is performed automatically.

9. A data synchronization control system for heterogeneous databases, characterized in that, include: The parameter acquisition unit is used to acquire synchronization status parameters between the source database and the target database. The adjustment unit is used to evaluate the current system load level based on the synchronous operation status parameters, and dynamically adjust the batch size of a single data synchronization according to the current system load level. An execution resource determination unit is used to obtain the task attribute characteristics of multiple concurrent synchronous tasks, calculate the scheduling weight of each concurrent synchronous task based on the task attribute characteristics, and dynamically allocate the execution resources of each concurrent synchronous task in combination with the real-time load status of the target database. The synchronization unit is used to synchronize incremental data in the source database to the target database based on the adjusted batch size and the allocated execution resources.

10. An electronic device, characterized in that, The electronic device includes: a memory and a processor; The memory is used to store at least one instruction; The processor is used to execute the at least one instruction to implement the data synchronization control method between heterogeneous databases as described in any one of claims 1 to 8.

Images