Data synchronization method and device, electronic equipment, storage medium and program product

Abstract

The invention provides a data synchronization method and device, electronic equipment, a storage medium and a program product, and relates to the technical field of data processing. The data synchronization method comprises the steps of performing a pressure test on a to-be-executed data synchronization task to obtain a pressure test result; generating a parallel synchronization parameter of the data synchronization task based on a result of the pressure test and an execution parameter of the data synchronization task; generating a synchronization task queue based on the parallel synchronization parameters and the priorities and dependencies of the data synchronization tasks so as to start the synchronization task queue to execute data synchronization operation; and controlling the speed of the data synchronization task based on the execution working condition of the synchronization task queue and / or the newly added synchronization task. Through the technical scheme of the invention, the synchronization operation can adapt to continuously increasing data synchronization requirements and changing service environments, so that the timeliness of data synchronization can be improved.

Description

Background Art

[0002] Data aggregation refers to the process of synchronizing data to aggregate business data scattered across various business systems into a data lake / data warehouse, thereby breaking down data silos and facilitating enterprises to fully utilize global data. The timeliness of data extraction determines the specific time when business data is applied to designated services. Therefore, how to improve the timeliness of data synchronization has become a technical issue that needs to be urgently addressed.

[0003] It should be noted that the information disclosed in the above background technology section is only used to enhance the understanding of the background of the present disclosure, and therefore may include information that does not constitute the prior art known to ordinary technicians in the field. Summary of the invention

[0004] The purpose of the present disclosure is to provide a data synchronization method, a data synchronization device, an electronic device, a computer-readable storage medium and a computer program product, which can at least to some extent improve the problem that the timeliness of data synchronization in related technologies needs to be improved.

[0005] Other features and advantages of the present disclosure will become apparent from the following detailed description, or may be learned in part by the practice of the present disclosure.

[0006] According to one aspect of the present disclosure, a data synchronization method is provided, including: performing a stress test on a data synchronization task to be executed to obtain a result of the stress test; generating a parallel synchronization parameter of the data synchronization task based on the result of the stress test and the execution parameter of the data synchronization task; generating a synchronization task queue based on the parallel synchronization parameter, the priority and dependency of the data synchronization task to start the synchronization task queue to perform a data synchronization operation; and controlling the speed of the data synchronization task based on the execution condition of the synchronization task queue and / or a newly added synchronization task.

[0007] In some embodiments of the present disclosure, a data synchronization task to be executed is stress tested to obtain a result of the stress test, including: configuring a test scenario based on at least one of the data volume, concurrency volume, and time requirements of the data synchronization task; starting a stress testing tool in the test scenario, and stress testing the data synchronization task to be executed based on a concurrency mode to obtain performance indicator results and resource utilization results as the result of the stress test, wherein the resource utilization results include computing resources and network resources required for the data synchronization task, and the performance indicator results include source-end performance indicators and target-end performance indicators of the data synchronization task.

[0008] In some embodiments of the present disclosure, parallel synchronization parameters of the data synchronization task are generated based on the results of the stress test and the execution parameters of the data synchronization task, including: determining the number of the data synchronization tasks that can be executed in parallel based on the required computing resources and the computing resources provided by the service device, and the required network resources and the network resources provided by the network; determining the data extraction rate and the write rate based on the required network resources, the provided network resources, the source-end performance indicators, and the target-end performance indicators, so as to generate the parallel synchronization parameters based on the number of the data synchronization tasks, the data extraction rate, and the write rate.

[0009] In some embodiments of the present disclosure, a synchronization task queue is generated based on the parallel synchronization parameters, the priority and dependency of the data synchronization tasks, including: the parallel synchronization parameters include the number of the data synchronization tasks executed in parallel, and the number of the synchronization task queues is determined based on the number of the data synchronization tasks; the head task in each of the synchronization task queues is determined based on the start time, delay time and task priority of the data synchronization task; the middle task and the tail task in each of the synchronization task queues are determined based on the dependency with the head task, the start time, the delay time and the task priority, wherein a first weight is configured for the delay time, and a second weight is configured for the task priority, the delay time is positively correlated with the first weight, and the task priority is positively correlated with the second weight.

[0010] In some embodiments of the present disclosure, the synchronization task queue is started to perform a data synchronization operation, including: if the synchronization task queue is started on time, the data synchronization operation of the synchronization task queue is performed; in the process of performing the data synchronization operation of the synchronization task queue, if it is detected that the service device releases the computing resources and the network releases the network resources, the start time of the unstarted task is modified to start in advance based on the priority, and the unstarted task depends on the completed data synchronization task; if a first synchronization task queue with a task delay is detected, it is monitored that there is a first synchronization task in the first synchronization task queue, the priority of the first synchronization task is higher than the second synchronization task in the second synchronization task queue, and the start time of the second synchronization task is earlier than the first synchronization task, then the first synchronization task is started before the second synchronization task.

[0011] In some embodiments of the present disclosure, starting the first synchronization task before the second synchronization task also includes: configuring the insertion extraction rate of the first synchronization task based on the initial extraction rate of the second synchronization task and the delay time of the second synchronization task.

[0012] In some embodiments of the present disclosure, the speed of the data synchronization task is controlled based on the execution condition of the synchronization task queue and / or the newly added synchronization task, including: the execution condition of the synchronization task queue includes the execution time, and when a delayed data synchronization task whose execution time exceeds the delay time is detected in the synchronization task queue, the priority of the delayed data synchronization task is increased; and the extraction rate of the delayed data synchronization task is adjusted based on the increased priority.

[0013] In some embodiments of the present disclosure, the data synchronization task is speed controlled based on the execution condition of the synchronization task queue and / or a newly added synchronization task, including: the execution condition of the synchronization task queue includes the source-side performance indicator and the target-side performance indicator, and the source-side performance indicator and the target-side performance indicator are monitored; based on the source-side performance indicator, a performance degradation of the source database is detected, and the extraction rate of the synchronization task queue is proportionally reduced according to the degree of performance degradation, wherein the source-side performance indicator includes a query response time and / or a number of database connections; based on the target-side performance indicator, it is detected that the performance of the target database exceeds a set indicator, and the write rate of the data synchronization task is dynamically adjusted, wherein the target-side performance indicator includes at least one of write throughput, storage utilization rate and index update time.

[0014] In some embodiments of the present disclosure, the speed of the data synchronization task is controlled based on the execution condition of the synchronization task queue and / or the newly added synchronization task, and also includes: in response to the newly added synchronization task, detecting the priority and start time of the newly added synchronization task, and the adjustable synchronization task in the synchronization task queue, the priority of the adjustable synchronization task is lower than the priority of the newly added synchronization task, and the start time of the adjustable synchronization task is later than the start time of the newly added synchronization task; adjusting the execution parameters of the adjustable synchronization task based on a resource yield mechanism to allocate the required computing resources and the network resources to the newly added synchronization task.

[0015] In some embodiments of the present disclosure, starting the synchronization task queue to perform data synchronization operations also includes: monitoring the occupancy conditions of computing resources and network resources of multiple synchronization task queues to obtain resource occupancy trends; and collaboratively controlling the speed of the synchronization task queues based on the resource occupancy trends.

[0016] According to another aspect of the present disclosure, a data synchronization device is provided, including: a testing module, used to perform a stress test on a data synchronization task to be executed and obtain a result of the stress test; a first generating module, used to generate a parallel synchronization parameter of the data synchronization task based on the result of the stress test and the execution parameters of the data synchronization task; a second generating module, used to generate a synchronization task queue based on the parallel synchronization parameters, the priority and dependency of the data synchronization task, so as to start the synchronization task queue to perform a data synchronization operation; a synchronization module, used to control the speed of the data synchronization task based on the execution condition of the synchronization task queue and / or a newly added synchronization task.

[0017] According to another aspect of the present disclosure, an electronic device is provided, comprising: a processor; and a memory for storing executable instructions of the processor; wherein the processor is configured to execute any one of the above-mentioned data synchronization methods by executing the executable instructions.

[0018] According to another aspect of the present disclosure, a computer-readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, any one of the above-mentioned data synchronization methods is implemented.

[0019] According to another aspect of the present disclosure, a computer program product is provided, on which a computer program is stored, and when the computer program is executed by a processor, any one of the above-mentioned data synchronization methods is implemented.

[0020] The data synchronization solution provided by the embodiments of the present disclosure can understand the environment in which the data synchronization task is located through stress testing. Based on the understanding of the environment and in combination with the execution parameters of the data synchronization task, a parallel synchronization solution can be determined. Furthermore, a synchronization task queue is generated considering the priority and dependency of the tasks, and resources are allocated according to the stress test results. This can ensure that important tasks are given priority to obtain sufficient resources and improve resource utilization efficiency. During the execution of the data synchronization task, it can be flexibly adjusted and optimized according to the specific execution conditions, so that the synchronization operation can adapt to the growing data synchronization needs and the changing business environment, thereby facilitating the timeliness of data synchronization.

[0021] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] The accompanying drawings herein are incorporated into the specification and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the specification are used to explain the principles of the present disclosure. Obviously, the accompanying drawings described below are only some embodiments of the present disclosure, and for ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative work.

[0023] Figure 1 A schematic diagram showing a data synchronization system structure in an embodiment of the present disclosure;

[0024] Figure 2 A flow chart of a data synchronization method according to an embodiment of the present disclosure is shown;

[0025] Figure 3 A flowchart showing another data synchronization method in an embodiment of the present disclosure is shown;

[0026] Figure 4 A flowchart of another data synchronization method according to an embodiment of the present disclosure is shown;

[0027] Figure 5 A flowchart showing another data synchronization method in an embodiment of the present disclosure is shown;

[0028] Figure 6 A schematic diagram showing a data synchronization device according to an embodiment of the present disclosure is shown;

[0029] Figure 7 A schematic diagram showing another data synchronization device according to an embodiment of the present disclosure;

[0030] Figure 8 A schematic diagram of an electronic device in an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0031] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in a variety of forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided so that the disclosure will be more comprehensive and complete and to fully convey the concepts of the example embodiments to those skilled in the art. The described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.

[0032] In addition, the accompanying drawings are only schematic illustrations of the present disclosure and are not necessarily drawn to scale. The same reference numerals in the figures represent the same or similar parts, and their repeated description will be omitted. Some of the block diagrams shown in the accompanying drawings are functional entities and do not necessarily correspond to physically or logically independent entities. These functional entities can be implemented in software form, or implemented in one or more hardware modules or integrated circuits, or implemented in different networks and / or processor devices and / or microcontroller devices.

[0033] Data synchronization can be understood as data replication and migration across data sources (common databases, file systems, interfaces, etc.). Data replication refers to the process of accurately copying data from one data source to another, while data migration is a more complex process that not only involves the movement of data, but may also include changes in data format, storage method, database architecture, etc. It is usually performed for reasons such as system upgrades, replacement of storage devices, or data source integration.

[0034] In the field of big data, data collection is an important part of enterprise digitalization. Through data synchronization operations, data scattered in various business systems are collected into the data lake / data warehouse. Data synchronization is the first link in data development. The timeliness of data extraction determines the specific time of enterprise data services. Therefore, how to ensure the reliability and timeliness of data synchronization has become an important part of the data development process.

[0035] In some embodiments, a management tool for a single synchronization task is created to focus on task-level management operations, including: continuously adjusting extraction parameters, concurrency, etc. according to the amount of data and the time of task extraction to find the optimal extraction rate. On the one hand, during the execution of this solution, the extraction rate of a single task depends on manual tuning. When the amount of data changes, manual intervention is still required, and parameters cannot be adjusted in time, resulting in reduced extraction efficiency. On the other hand, this solution lacks management of global tasks and tasks, and the management of a single data synchronization task mainly relies on manual setting of task priority, concurrency, etc. to control the synchronization rate of the task.

[0036] The data synchronization solution disclosed in the present invention aims to automatically optimize and adjust the priority, synchronization efficiency, number of concurrency, etc. of data synchronization tasks, realize intelligent management of data synchronization tasks, ensure the synchronization efficiency of data synchronization links, and accelerate enterprise data synchronization.

[0037] Figure 1 1 is a schematic diagram of a computer system provided by an exemplary embodiment of the present application. The system includes: a plurality of terminals 120 and a server cluster 140.

[0038] The terminal 120 may be a mobile terminal such as a mobile phone, a game console, a tablet computer, an e-book reader, smart glasses, an MP4 (Moving Picture Experts Group Audio Layer IV) player, a smart home device, an AR (Augmented Reality) device, a VR (Virtual Reality) device, or the terminal 120 may also be a personal computer (PC), such as a laptop computer and a desktop computer.

[0039] The terminal 120 may be installed with an application for providing data synchronization, and the server cluster 140 may provide a source database and a target database respectively.

[0040] The terminal 120 is connected to the server cluster 140 via a communication network. Optionally, the communication network is a wired network or a wireless network.

[0041] The server cluster 140 is a server, or is composed of several servers, or is a virtualization platform, or is a cloud computing service center. The server cluster 140 is used to provide background services for applications that provide data synchronization. Optionally, the server cluster 140 undertakes the main computing work, and the terminal 120 undertakes the secondary computing work; or, the server cluster 140 undertakes the secondary computing work, and the terminal 120 undertakes the main computing work; or, the terminal 120 and the server cluster 140 adopt a distributed computing architecture for collaborative computing.

[0042] In some optional embodiments, the server cluster 140 is used to store data synchronization program information.

[0043] Optionally, the logistics user terminals of the application programs installed in different terminals 120 are the same, or the logistics user terminals of the application programs installed on the two terminals 120 are logistics user terminals of the same type of application programs on different control system platforms. Based on the different terminal platforms, the specific forms of the logistics user terminals of the application programs may also be different, for example, the logistics user terminals of the application programs may be mobile phone logistics user terminals, PC logistics user terminals, or global wide area network (World Wide Web, Web) logistics user terminals, etc.

[0044] Those skilled in the art will appreciate that the number of the terminals 120 may be more or less. For example, there may be only one terminal, or there may be dozens or hundreds of terminals, or a greater number. The embodiment of the present application does not limit the number and device type of the terminals.

[0045] Optionally, the system may also include a management device ( Figure 1 (not shown), the management device is connected to the server cluster 140 via a communication network. Optionally, the communication network is a wired network or a wireless network.

[0046] Optionally, the above-mentioned wireless network or wired network uses standard communication technology and / or protocol. The network is usually the Internet, but it can also be any network, including but not limited to a local area network (LAN), a metropolitan area network (MAN), a wide area network (WAN), a mobile, wired or wireless network, a dedicated network or any combination of a virtual private network). In some embodiments, the data exchanged through the network is represented by technologies and / or formats including Hyper Text Mark-up Language (HTML), Extensible Markup Language (XML), etc. In addition, conventional encryption technologies such as Secure Socket Layer (SSL), Transport Layer Security (TLS), Virtual Private Network (VPN), Internet Protocol Security (IPsec) can also be used to encrypt all or some links. In other embodiments, customized and / or dedicated data communication technologies can also be used to replace or supplement the above-mentioned data communication technologies.

[0047] Exemplarily, in the present disclosure, the data synchronization system includes a task center, a scheduling center, and a configuration center.

[0048] The data synchronization method in this example implementation will be described in more detail below with reference to the accompanying drawings and embodiments.

[0049] like Figure 2 As shown, a data synchronization method according to an embodiment of the present disclosure includes:

[0050] Step S202: performing a stress test on the data synchronization task to be executed, and obtaining a result of the stress test.

[0051] In some embodiments, a test environment can be prepared to accurately reflect the performance and resource usage in the actual production environment to ensure the reliability of the stress test.

[0052] In some embodiments, the results of the stress test can be understood as environmental parameters of the environment in which the data synchronization task is located, which may include the computing resources required for the data synchronization task, such as CPU utilization and memory usage, network resources, such as bandwidth utilization and network latency, and performance indicators of the source and target ends, such as data extraction rate and write rate.

[0053] In some embodiments, stress testing tools can be used or test scripts can be written to simulate different load conditions, such as increasing the number of concurrent data synchronization tasks, increasing the amount of data, etc., so as to monitor and record the data of various indicators in real time during the test.

[0054] Step S204: Generate parallel synchronization parameters of the data synchronization task based on the result of the stress test and the execution parameters of the data synchronization task.

[0055] In some embodiments, the execution parameters of the data synchronization task include a start time, a delay time, and an execution priority.

[0056] In some embodiments, the results and execution parameters of the stress test affect the generation of parallel synchronization parameters. For example, high-priority tasks may require more resources and faster startup times. If the stress test shows that the system's performance degrades under high load, the number of tasks executed in parallel cannot be too large. If the system resources are sufficient, the degree of parallelism can be appropriately increased to improve the efficiency of data synchronization.

[0057] In addition, if the performance of the source end is low, the extraction rate should be reduced to prevent excessive pressure on the source end. If the write speed of the target end is limited, the write rate also needs to be reduced accordingly.

[0058] In some embodiments, parallel synchronization parameters of data synchronization tasks are generated based on the results of stress testing and the execution parameters of data synchronization tasks, and also include: reasonably allocating network bandwidth to different data synchronization tasks according to the network resource usage in the stress testing results. For example, tasks with higher priority can be allocated more network bandwidth to ensure the speed and stability of their data transmission.

[0059] Step S206: Generate a synchronization task queue based on the parallel synchronization parameters, the priority and dependency of the data synchronization tasks, and start the synchronization task queue to perform the data synchronization operation.

[0060] In some embodiments, the number of tasks executed in parallel, that is, the number of queues executed in parallel, can be determined based on parallel synchronization parameters. In each synchronous task queue, tasks with high priority are placed in the queue first, and tasks that depend on other tasks can only be executed after the tasks they depend on are completed.

[0061] In some embodiments, the synchronization task queue is started to start the data synchronization operation, the execution of the task is monitored, and errors and abnormalities in the task execution process are handled in a timely manner.

[0062] Step S208: Control the speed of the data synchronization task based on the execution status of the synchronization task queue and / or the newly added synchronization task.

[0063] In some embodiments, the execution status of the synchronization task queue is monitored in real time, including the execution progress of the task, resource usage, and error rate. If it is found that the task execution is slow, the resource utilization is too high, or the error rate increases, the extraction rate and writing rate of the data synchronization task can be dynamically adjusted to reduce the system load.

[0064] In some embodiments, when a new synchronization task is added, resource conditions and task priorities are re-evaluated, and parallel synchronization parameters and task queues are adjusted according to the new situation to ensure that the new task can be reasonably integrated into the existing data synchronization process.

[0065] In this embodiment, through stress testing, the environment in which the data synchronization task is located can be understood. Based on the understanding of the environment and combined with the execution parameters of the data synchronization task, the parallel synchronization plan can be determined. Furthermore, the synchronization task queue is generated by considering the priority and dependency of the task, and resources are allocated according to the stress test results. This can ensure that important tasks obtain sufficient resources first and improve resource utilization efficiency. During the execution of the data synchronization task, it can be flexibly adjusted and optimized according to the specific execution conditions, so that the synchronization operation can adapt to the growing data synchronization needs and the changing business environment, which is conducive to improving the timeliness of data synchronization.

[0066] In some embodiments of the present disclosure, a stress test is performed on a data synchronization task to be executed, and a result of the stress test is obtained, including:

[0067] Configure a test scenario based on at least one of the data volume, concurrency, and time requirements of the data synchronization task.

[0068] In some embodiments, if the data synchronization task is to migrate a large amount of historical data from one database to another, it is necessary to configure the test scenario according to the specific scale of the data. If the data synchronization task is performed in a multi-user system, it is necessary to estimate the number of concurrent users and configure the corresponding test scenario based on the time requirements of the data synchronization task.

[0069] Start the stress test tool in the test scenario, and perform stress testing on the data synchronization task to be executed based on the concurrent mode, and obtain performance indicator results and resource utilization results as the result of the stress test. The resource utilization results include the computing resources and network resources required for the data synchronization task, and the performance indicator results include the source-side performance indicators and target-side performance indicators of the data synchronization task.

[0070] In some embodiments, a suitable stress testing tool is selected according to the characteristics of the data synchronization task and the testing requirements, such as a network stress testing tool such as Apache JMeter. In addition, a stress testing tool provided by the database may also be selected.

[0071] In some embodiments, a concurrency mode is set in the stress testing tool to simulate the situation where multiple tasks are running in parallel. The stress testing tool is started in the test scenario. The stress testing tool will send multiple data synchronization requests at the same time according to the set concurrency mode, monitor the output of the testing tool in real time, and record various performance indicators and resource utilization results. The performance indicators include source-side performance indicators of the data synchronization task (such as the query response time of the source database, data extraction speed, etc.) and target-side performance indicators (such as the write speed and response time of the target database). The resource utilization results include computing resources (such as CPU usage, memory occupancy, etc.) and network resources (such as network bandwidth usage, network latency, etc.) required for the data synchronization task.

[0072] In this embodiment, the resource utilization results obtained by the stress test can help determine the computing resources and network resources required for the data synchronization task. According to the results of the stress test, the resource allocation strategy can be reasonably adjusted to ensure that important resources are allocated to key tasks and improve resource utilization efficiency. For data synchronization tasks with time requirements, stress testing can be used to determine whether the synchronization operation can be completed within the specified time. If the time requirement cannot be met, optimization measures can be taken to ensure the realization of business needs.

[0073] In some embodiments of the present disclosure, generating a parallel synchronization parameter of a data synchronization task based on a result of a stress test and an execution parameter of the data synchronization task includes:

[0074] Based on the required computing resources and the computing resources provided by the service device, as well as the required network resources and the network resources provided by the network, determine the number of data synchronization tasks that can be executed in parallel; based on the required network resources, the provided network resources, the source-side performance indicators, and the target-side performance indicators, determine the data extraction rate and the write rate to generate parallel synchronization parameters based on the number of data synchronization tasks, the data extraction rate, and the write rate.

[0075] In some embodiments, source-side performance indicators include data extraction speed, response time, and concurrent processing capability, and target-side performance indicators include data writing speed, storage capacity, and response time.

[0076] In this embodiment, for the required computing resources, the nature and scale of the data synchronization tasks are analyzed; for the required network resources, factors such as the frequency of data synchronization, the amount of data, and the distance between the source and target ends are considered; for the computing resources provided by the service device, the hardware configuration of the service device is checked to determine the computing resources that can be provided; for network resources, the bandwidth, latency and other parameters of the network are determined by checking the specifications and performance of the network equipment; based on the required computing resources and the computing resources provided by the service device, as well as the required network resources and the network resources provided by the network, the number of data synchronization tasks that can be executed in parallel is determined; by reasonably setting the number of parallel tasks and the extraction write rate, combined with the performance indicators of the source and target ends and the limitations of network resources, it is helpful to ensure the stable execution of data synchronization tasks.

[0077] like Figure 3 As shown, in some embodiments of the present disclosure, a synchronization task queue is generated based on parallel synchronization parameters, priorities and dependencies of data synchronization tasks, including:

[0078] Step S302: The parallel synchronization parameters include the number of data synchronization tasks executed in parallel, and the number of synchronization task queues is determined based on the number of data synchronization tasks.

[0079] Step S304: determining the head task in each synchronization task queue based on the start time, delay time and task priority of the data synchronization task.

[0080] In some embodiments, the delay time indicates the estimated execution time of the data synchronization task, and the task with the earliest start time, the longest delay time and the highest priority can be selected as the head task in each synchronization task queue.

[0081] Step S306, determine the in-line tasks and the tail tasks in each synchronous task queue based on the dependency with the head task, the start time, the delay time and the task priority, wherein a first weight is configured for the delay time and a second weight is configured for the task priority, the delay time is positively correlated with the first weight, and the task priority is positively correlated with the second weight.

[0082] In some embodiments, in addition to the start time, delay time and task priority, the dependency between tasks also needs to be considered, and the synchronization tasks with dependencies are assigned to the same queue. In addition, a first weight is configured for the delay time, and a second weight is configured for the task priority. The delay time is positively correlated with the first weight, indicating that for any data synchronization task, the longer the delay time, the greater the weight, and the task priority is positively correlated with the second weight, that is, the higher the task priority, the greater the weight. The comprehensive score is determined based on the start time score, the delay time score (calculated according to the first weight), the task priority score (calculated according to the second weight) and the dependency. Tasks with higher scores will be placed at the front of the queue, and tasks with lower scores will be placed at the back.

[0083] In this embodiment, by reasonably dividing the synchronization task queues, dividing the tasks into different queues, and determining the task order according to the attributes and dependencies of the tasks, resources can be allocated and tasks can be scheduled for each queue, and the execution efficiency of data synchronization tasks can be effectively improved, making task management and monitoring more convenient. In addition, the number of synchronization task queues and task allocation can be dynamically adjusted according to changes in system resources to ensure the high efficiency of executing data synchronization tasks.

[0084] like Figure 4 As shown, in some embodiments of the present disclosure, including:

[0085] Step S402, generating a synchronization task queue.

[0086] Start the synchronization task queue to perform data synchronization operations, including:

[0087] Step S404: If the synchronization task queue is started on time, the data synchronization operation of the synchronization task queue is executed.

[0088] Among them, in the first working condition, when the task is started normally, it can be queued and executed normally according to the task start time and dependency relationship.

[0089] Step S406, during the data synchronization operation of the synchronization task queue, if it is detected that the service device releases computing resources and the network releases network resources, the start time of the unstarted task is modified to start in advance based on the priority, and the unstarted task depends on the completed data synchronization task.

[0090] Among them, in the second working condition, when the task is started normally, the computing resources and network bandwidth are more sufficient than during the stress test, so the tasks with higher priority and whose dependencies have been completed but are expected to start later can be started in advance.

[0091] Step S408, when detecting a first synchronization task queue with task delay, if it is detected that there is a first synchronization task in the first synchronization task queue, the priority of the first synchronization task is higher than the second synchronization task in the second synchronization task queue, and the start time of the second synchronization task is earlier than the first synchronization task, then the first synchronization task is started before the second synchronization task.

[0092] Among them, under the third working condition, there is a situation where the task is delayed in starting. At this time, for delayed start tasks a and b in different queues, if the priority of task a in queue A is higher than that of task b in queue B, but the start time of task b is earlier than that of task a, task a can be started in advance.

[0093] In addition, whether tasks can continue to be submitted is controlled based on the idle status of resource group resources and network bandwidth, thereby controlling the concurrency of synchronization tasks.

[0094] In this embodiment, the start time of the task can be dynamically adjusted according to the usage of resources to achieve optimal allocation of resources. When the service device releases computing resources and the network releases network resources, these resources are promptly allocated to high-priority unstarted tasks, which can improve resource utilization. In addition, processing task delays and priority adjustments can ensure that critical tasks can be executed in a timely manner, reducing system instability factors caused by task delays.

[0095] In some embodiments of the present disclosure, starting the first synchronization task before the second synchronization task further includes:

[0096] Step S410: configuring the insertion extraction rate of the first synchronization task based on the initial extraction rate of the second synchronization task and the delay time of the second synchronization task.

[0097] In some embodiments, the insertion extraction rate of the first synchronization task may be adjusted according to the ratio of the delay time of the second synchronization task to the total task time (the time from the start to the expected completion time).

[0098] In some embodiments, weights may be assigned to the priority of the first synchronization task and the delay time of the second synchronization task respectively. If the priority of the first synchronization task is relatively high, the corresponding weight is set to be relatively large, and then the weight of the delay time of the second synchronization task is determined. For example, assuming that the weight of the priority of the first synchronization task is 0.7 and the weight of the delay time of the second synchronization task is 0.3, the proportional relationship between the insertion extraction rate of the first synchronization task and the initial extraction rate of the second synchronization task is determined by calculating the comprehensive value (comprehensive value = the score corresponding to the priority of the first synchronization task × 0.7 + (1-the proportion of the delay time of the second synchronization task) × 0.3).

[0099] In some embodiments, the insertion and extraction rate of the first synchronization task can also be dynamically adjusted according to the current resource status of the system (such as the idleness of computing resources, the remaining amount of network bandwidth, etc.) and the progress of the second synchronization task.

[0100] In this embodiment, by configuring the insertion extraction rate of the first synchronization task, it is possible to minimize the impact on the second synchronization task while ensuring that the high-priority first synchronization task can be inserted and executed as soon as possible, so as to achieve a balance between overall performance and task execution efficiency in the case of task queue interruption.

[0101] In some embodiments of the present disclosure, the speed of the data synchronization task is controlled based on the execution status of the synchronization task queue and / or the newly added synchronization task, including:

[0102] The execution condition of the synchronization task queue includes the execution time. When it is detected that there is a delayed data synchronization task in the synchronization task queue whose execution time exceeds the delay time, the priority of the delayed data synchronization task is increased; and the extraction rate of the delayed data synchronization task is adjusted based on the increased priority.

[0103] In some embodiments, a monitoring mechanism is established to periodically query the task status and receive feedback information during the task execution process, or an event trigger mechanism is used to realize real-time monitoring of the task execution status. When it is monitored that the execution time of a data synchronization task in the synchronization task queue exceeds its preset delay time, the task is marked as a delayed data synchronization task. Once the delayed data synchronization task is determined, the priority adjustment mechanism is automatically triggered. The mechanism decides whether to increase the priority of the task based on the delay of the task and the overall operation status of the system. If it is decided to increase the priority of the delayed data synchronization task, the priority identifier of the task in the task queue is modified, and the resource requirements of the delayed data synchronization task are re-evaluated based on the increased priority, and the adjustment range of the data extraction rate is determined based on the resource requirements and the current system resource status, so as to adjust the extraction rate by modifying the task execution parameters or calling the interface of the data synchronization tool.

[0104] In this embodiment, the priority of the task uses the original configured priority by default. If the execution time of the task exceeds the task delay time, the priority of the task will be temporarily increased to ensure that the task that has been delayed for too long can be executed in time. By timely detecting delayed tasks and raising their priorities and adjusting the extraction rate, the execution speed of delayed tasks can be accelerated, thereby optimizing synchronization efficiency and resource utilization.

[0105] like Figure 5 As shown, in some embodiments of the present disclosure, the speed of the data synchronization task is controlled based on the execution condition of the synchronization task queue and / or the newly added synchronization task, including:

[0106] Step S502: the execution conditions of the synchronization task queue include source-end performance indicators and target-end performance indicators, and the source-end performance indicators and the target-end performance indicators are monitored.

[0107] Among them, the source-side performance indicators mainly include query response time and the number of database connections. The query response time refers to the time interval from issuing a query request to returning the result, which reflects the response speed and processing capacity of the source database. The number of database connections indicates the number of connections currently established with the source database. Too many connections may cause the performance of the source database to deteriorate.

[0108] The target-side performance indicators include write throughput, storage utilization, and index update time. Write throughput refers to the amount of data that can be written to the target database per unit time. Storage utilization indicates the proportion of storage space used by the target database to the total storage space. Index update time reflects the speed at which the target database updates its index.

[0109] In some embodiments, source-side performance indicators and target-side performance indicators are continuously monitored based on a monitoring mechanism.

[0110] Step S504, based on the source-side performance indicators, it is detected that the source database performance has degraded, and the extraction rate of the synchronization task queue is proportionally reduced according to the degree of performance degradation, wherein the source-side performance indicators include query response time and / or number of database connections.

[0111] In some embodiments, if the query response time is doubled, the extraction rate can be reduced by 50%. The specific reduction ratio can be adjusted according to the actual situation and can be determined by pre-set rules or algorithms. Reducing the extraction rate can reduce the pressure on the source database and prevent further deterioration of the performance of the source database. If an ETL (Extract Transform Load) tool is used for data synchronization, the frequency of data extraction, the amount of data extracted each time and other parameters can be modified in the tool's configuration file to achieve the purpose of reducing the extraction rate.

[0112] Step S506, based on the target-side performance indicator, it is detected that the target database performance exceeds the set indicator, and the write rate of the data synchronization task is dynamically adjusted, wherein the target-side performance indicator includes at least one of write throughput, storage utilization rate and index update time.

[0113] In some embodiments, if the write throughput is close to the maximum write capacity, the write rate can be appropriately reduced to avoid data backlog and write failure. If the storage utilization rate is too high, the data synchronization task can be suspended and writing can be continued after the storage resources are released. If the index update time is extended, the index structure can be optimized or the frequency of index updates can be adjusted to improve write performance.

[0114] In this embodiment, by monitoring the performance indicators of the source and target ends in real time and dynamically adjusting the extraction rate and the writing rate according to the performance status, data synchronization failure or system performance degradation caused by database performance problems can be prevented.

[0115] In some embodiments of the present disclosure, the speed of the data synchronization task is controlled based on the execution status of the synchronization task queue and / or the newly added synchronization task, and further includes:

[0116] In response to a newly added synchronization task, the priority and start time of the newly added synchronization task, as well as the adjustable synchronization tasks in the synchronization task queue are detected. The priority of the adjustable synchronization task is lower than the priority of the newly added synchronization task, and the start time of the adjustable synchronization task is later than the start time of the newly added synchronization task; based on the resource release mechanism, the execution parameters of the adjustable synchronization task are adjusted to allocate the required computing resources and network resources to the newly added synchronization task.

[0117] In some embodiments, the newly added synchronization task is evaluated to determine its priority and start time, wherein the priority can be determined based on factors such as the importance and urgency of the task, and the start time refers to the time point when the task is scheduled to start execution.

[0118] In some embodiments, by traversing the existing synchronization task queue, adjustable synchronization tasks that meet specific conditions are found, the priority of these tasks is lower than the priority of the newly added synchronization task, and the start time is later than the start time of the newly added synchronization task.

[0119] In some embodiments, the computing resources (such as CPU, memory, etc.) and network resources (such as bandwidth) required for the newly added synchronization task are determined, and based on the resource release mechanism, a strategy for adjusting the execution parameters of the adjustable synchronization task is formulated to modify the execution parameters of the adjustable synchronization task based on the strategy. This can be achieved by modifying the task configuration file, calling the task management interface, or using a resource management tool.

[0120] In this embodiment, by timely detecting new synchronization tasks and identifying adjustable synchronization tasks, when a new synchronization task with a high priority appears, by adjusting the adjustable synchronization tasks with a low priority and a later start time, the execution order of tasks and resource allocation can be dynamically adjusted according to actual conditions. The resource release mechanism used can enable the system to allocate computing resources and network resources more reasonably.

[0121] In some embodiments of the present disclosure, starting a synchronization task queue to perform a data synchronization operation also includes:

[0122] Monitor the occupancy of computing resources and network resources by multiple synchronous task queues to obtain resource occupancy trends; and perform collaborative speed control on synchronous task queues based on resource occupancy trends.

[0123] In some embodiments, if the resource usage trend shows that the resource demand of a task queue continues to grow and may cause a system resource bottleneck, it is necessary to consider controlling the speed of the task queue. For example, for a task queue whose data extraction rate is too fast and causes excessive CPU usage, a strategy can be formulated to reduce its extraction rate by a certain percentage, while considering the collaborative relationship between task queues. If reducing the speed of a task queue will have an impact on other task queues that depend on it, a comprehensive trade-off is required. For example, for task queues A and B that have a data dependency relationship, when reducing the speed of A, it is necessary to ensure that B has enough buffered data to continue execution, or adjust B's execution parameters at the same time.

[0124] In some embodiments, for a data synchronization task, parameters may be set in a software tool (such as an ETL tool) for task execution to adjust the rate of data extraction and writing.

[0125] In some embodiments, since the resource requirements and system environment of the data synchronization task may change continuously, the speed control strategy has the ability to be dynamically adjusted, and machine learning technology can be used to more accurately predict resource usage trends, thereby achieving more intelligent dynamic adjustment.

[0126] In this embodiment, when multiple synchronization tasks are executed in parallel, from the perspective of the overall data synchronization task arrangement, due to the limited network and synchronization computing resources, once one or a pair of synchronization tasks are started, they will always occupy the network and synchronization computing resources. By configuring a collaborative speed control strategy, effective management can be achieved when encountering scenarios such as wanting to speed up important tasks and controlling the arrangement of data synchronization tasks.

[0127] like Figure 6 As shown, a data synchronization device 600 according to an embodiment of the present disclosure includes a synchronization task scheduling unit 602 , a task speed control unit 604 , and a stress test and synchronization monitoring unit 606 .

[0128] Among them, for the data synchronization tasks to be executed, task concurrency control and task start control are performed based on the task scheduling unit 602. Specifically, through the stress testing function in the stress testing and synchronization monitoring unit 606, performance indicator results and resource utilization results are obtained. The resource utilization results include the computing queue resource data and network bandwidth data required for the data synchronization task. The performance indicator results include the source-end performance indicators and target-end performance indicators of the data synchronization task, that is, the database performance data. Combined with the task execution data, parallel synchronization parameters for task concurrency control can be generated. Furthermore, based on the parallel synchronization parameters, the priority and dependency of the data synchronization task, a synchronization task queue is generated, including queue A, queue B and queue N, etc., to start the synchronization task queue to perform data synchronization operations to achieve task start control.

[0129] During the data synchronization operation, based on the stress test and synchronization monitoring function in the synchronization monitoring unit 606, the computing queue resource data, network bandwidth data, task execution data and database performance data continue to be monitored, and these data are fed back to the task speed control unit 604, so that the task speed control unit 604 can control the task speed through task parallelism control and task priority control.

[0130] In some embodiments, an intelligent data synchronization device will control synchronization tasks (including startup, speed control, delayed startup, etc.) at regular intervals to ensure effective execution of synchronization tasks and achieve reasonable arrangement of global tasks. At the same time, for tasks that need to be accelerated, at startup, tasks with low priority and short execution time will give up computing resources and network bandwidth to ensure that accelerated tasks have sufficient resources and bandwidth to accelerate task execution.

[0131] It should be noted that the above figures are only schematic illustrations of the processes included in the method according to the exemplary embodiments of the present disclosure, and are not intended to be limiting. It is easy to understand that the processes shown in the above figures do not indicate or limit the time sequence of these processes. In addition, it is also easy to understand that these processes can be performed synchronously or asynchronously, for example, in multiple modules.

[0132] Those skilled in the art will appreciate that various aspects of the present disclosure may be implemented as systems, methods or program products. Therefore, various aspects of the present disclosure may be specifically implemented in the following forms, namely: complete hardware implementation, complete software implementation (including firmware, microcode, etc.), or a combination of hardware and software, which may be collectively referred to herein as "circuits", "modules" or "systems".

[0133] Refer to the following Figure 7 The data synchronization device 700 according to this embodiment of the present disclosure is described. Figure 7 The data synchronization device 700 shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0134] The data synchronization device 700 is in the form of a hardware module. The components of the data synchronization device 700 may include but are not limited to: a test module 702, which is used to perform a stress test on the data synchronization task to be executed and obtain the result of the stress test; a first generation module 704, which is used to generate a parallel synchronization parameter of the data synchronization task based on the result of the stress test and the execution parameter of the data synchronization task; a second generation module 706, which is used to generate a synchronization task queue based on the parallel synchronization parameter, the priority and dependency of the data synchronization task, so as to start the synchronization task queue to perform the data synchronization operation; a synchronization module 708, which is used to control the speed of the data synchronization task based on the execution condition of the synchronization task queue and / or the newly added synchronization task.

[0135] Refer to the following Figure 8 The electronic device 800 according to this embodiment of the present disclosure is described. Figure 8 The electronic device 800 shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0136] like Figure 8 As shown, the electronic device 800 is in the form of a general computing device. The components of the electronic device 800 may include but are not limited to: at least one processing unit 810, at least one storage unit 820, and a bus 830 connecting different system components (including the storage unit 820 and the processing unit 810).

[0137] The storage unit stores program codes, which can be executed by the processing unit 810, so that the processing unit 810 performs the steps according to various exemplary embodiments of the present disclosure described in the above “Exemplary Method” section of this specification. For example, the processing unit 810 can perform the following steps: Figure 2 Steps S202 and S204 shown in , and other steps defined in the data synchronization method of the present disclosure.

[0138] The storage unit 820 may include a readable medium in the form of a volatile storage unit, such as a random access memory unit (RAM) 8201 and / or a cache memory unit 8202 , and may further include a read-only memory unit (ROM) 8203 .

[0139] The storage unit 820 may also include a program / utility 8204 having a set (at least one) of program modules 8205, such program modules 8205 including but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment.

[0140] Bus 830 may represent one or more of several types of bus structures, including a memory unit bus or memory unit controller, a peripheral bus, an accelerated graphics port, a processing unit, or a local bus using any of a variety of bus architectures.

[0141] The electronic device 800 may also communicate with one or more external devices 860 (e.g., keyboards, pointing devices, Bluetooth devices, etc.), may also communicate with one or more devices that enable a user to interact with the electronic device, and / or may communicate with any device that enables the electronic device 800 to communicate with one or more other computing devices (e.g., routers, modems, etc.). Such communication may be performed via an input / output (I / O) interface 850. Furthermore, the electronic device 800 may also communicate with one or more networks (e.g., local area networks (LANs), wide area networks (WANs), and / or public networks, such as the Internet) via a network adapter 850. As shown, the network adapter 850 communicates with other modules of the electronic device 800 via a bus 830. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in conjunction with the electronic device, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems, etc.

[0142] Through the description of the above implementation, it is easy for those skilled in the art to understand that the example implementation described here can be implemented by software, or by software combined with necessary hardware. Therefore, the technical solution according to the implementation of the present disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, including several instructions to enable a computing device (which can be a personal computer, a server, a terminal device, or a network device, etc.) to execute the method according to the implementation of the present disclosure.

[0143] In an exemplary embodiment of the present disclosure, a computer-readable storage medium is also provided, on which a program product capable of implementing the above method of the present specification is stored. In some possible implementations, various aspects of the present disclosure may also be implemented in the form of a program product, which includes a program code, and when the program product is run on a terminal device, the program code is used to enable the terminal device to execute the steps according to various exemplary implementations of the present disclosure described in the above "Exemplary Method" section of the present specification.

[0144] According to the program product for implementing the above method in the embodiment of the present disclosure, it can adopt a portable compact disk read-only memory (CD-ROM) and include program code, and can be run on a terminal device, such as a personal computer. However, the program product of the present disclosure is not limited thereto. In this document, a readable storage medium can be any tangible medium containing or storing a program, which can be used by or in combination with an instruction execution system, an apparatus or a device.

[0145] Computer readable signal media may include data signals propagated in baseband or as part of a carrier wave, in which readable program code is carried. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Readable signal media may also be any readable medium other than a readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0146] The program code embodied on the readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wired, optical cable, RF, etc., or any suitable combination of the foregoing.

[0147] Program code for performing the operations of the present disclosure may be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, C++, etc., and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a separate software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving a remote computing device, the remote computing device may be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computing device (e.g., through the Internet using an Internet service provider).

[0148] It should be noted that, although several modules or units of the device for action execution are mentioned in the above detailed description, this division is not mandatory. In fact, according to the embodiments of the present disclosure, the features and functions of two or more modules or units described above can be embodied in one module or unit. On the contrary, the features and functions of one module or unit described above can be further divided into multiple modules or units to be embodied.

[0149] In addition, although the steps of the method in the present disclosure are described in a specific order in the drawings, this does not require or imply that the steps must be performed in this specific order, or that all the steps shown must be performed to achieve the desired results. Additionally or alternatively, some steps may be omitted, multiple steps may be combined into one step, and / or one step may be decomposed into multiple steps, etc.

[0150] Through the description of the above implementation, it is easy for those skilled in the art to understand that the example implementation described here can be implemented by software, or by software combined with necessary hardware. Therefore, the technical solution according to the implementation of the present disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, including several instructions to enable a computing device (which can be a personal computer, a server, a mobile terminal, or a network device, etc.) to execute the method according to the implementation of the present disclosure.

[0151] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the disclosure herein. This application is intended to cover any variations, uses or adaptations of the present disclosure, which follow the general principles of the present disclosure and include common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The specification and examples are intended to be exemplary only, and the true scope and spirit of the present disclosure are indicated by the appended claims.

Claims

1. A data synchronization method, characterized in that: include: Perform stress testing on the data synchronization task to be executed and obtain the stress testing result; generating a parallel synchronization parameter of the data synchronization task based on the result of the stress test and the execution parameter of the data synchronization task; Generate a synchronization task queue based on the parallel synchronization parameters, the priority and dependency of the data synchronization tasks, and start the synchronization task queue to perform data synchronization operations; The data synchronization task is speed-controlled based on the execution status of the synchronization task queue and / or the newly added synchronization task.

2. The data synchronization method according to claim 1, characterized in that: Perform stress testing on the data synchronization task to be executed and obtain the stress test results, including: Configure a test scenario based on at least one of the data volume, concurrency volume, and time requirements of the data synchronization task; In the test scenario, a stress test tool is started, and a stress test is performed on the data synchronization task to be executed based on a concurrent mode, and a performance indicator result and a resource utilization result are obtained as the result of the stress test. The resource utilization result includes the computing resources and network resources required for the data synchronization task, and the performance indicator result includes the source-side performance indicator and the target-side performance indicator of the data synchronization task.

3. The data synchronization method according to claim 2, characterized in that: Generating a parallel synchronization parameter of the data synchronization task based on the result of the stress test and the execution parameter of the data synchronization task includes: Determine the number of the data synchronization tasks that can be executed in parallel based on the required computing resources and the computing resources provided by the service device, and the required network resources and the network resources provided by the network; The data extraction rate and the writing rate are determined based on the required network resources, the provided network resources, the source-side performance indicators and the target-side performance indicators, so as to generate the parallel synchronization parameters based on the number of the data synchronization tasks, the data extraction rate and the writing rate.

4. The data synchronization method according to claim 1, characterized in that: Generating a synchronization task queue based on the parallel synchronization parameter, the priority and dependency of the data synchronization task, including: The parallel synchronization parameter includes the number of the data synchronization tasks executed in parallel, and the number of the synchronization task queues is determined based on the number of the data synchronization tasks; Determine the head task in each synchronization task queue based on the start time, delay time and task priority of the data synchronization task; The in-team tasks and the tail-team tasks in each synchronous task queue are determined based on the dependency with the head-of-team task, the start time, the delay time and the task priority, wherein a first weight is configured for the delay time and a second weight is configured for the task priority, the delay time is positively correlated with the first weight, and the task priority is positively correlated with the second weight.

5. The data synchronization method according to claim 3, characterized in that: Starting the synchronization task queue to perform data synchronization operations includes: If the synchronization task queue is started on time, the data synchronization operation of the synchronization task queue is executed; During the data synchronization operation of the synchronization task queue, if it is detected that the service device releases the computing resources and the network releases the network resources, the start time of the unstarted task is modified to start in advance based on the priority, and the unstarted task depends on the completed data synchronization task; If a first synchronization task queue with task delay is detected, and it is monitored that there is a first synchronization task in the first synchronization task queue, the priority of the first synchronization task is higher than the second synchronization task in the second synchronization task queue, and the start time of the second synchronization task is earlier than the first synchronization task, then the first synchronization task is started before the second synchronization task.

6. The data synchronization method according to claim 5, characterized in that: Starting the first synchronization task before the second synchronization task also includes: The insertion extraction rate of the first synchronization task is configured based on the initial extraction rate of the second synchronization task and the delay time of the second synchronization task.

7. The data synchronization method according to claim 1, characterized in that: The speed of the data synchronization task is controlled based on the execution condition of the synchronization task queue and / or the newly added synchronization task, including: The execution condition of the synchronization task queue includes the execution time, and when it is detected that there is a delayed data synchronization task in the synchronization task queue whose execution time exceeds the delay time, the priority of the delayed data synchronization task is increased; The extraction rate of the delayed data synchronization task is adjusted based on the increased priority.

8. The data synchronization method according to claim 2, characterized in that: The speed of the data synchronization task is controlled based on the execution condition of the synchronization task queue and / or the newly added synchronization task, including: The execution condition of the synchronous task queue includes the source end performance indicator and the target end performance indicator, and the source end performance indicator and the target end performance indicator are monitored; Detecting a performance degradation of the source database based on the source-side performance indicator, and reducing the extraction rate of the synchronization task queue in proportion to the degree of performance degradation, wherein the source-side performance indicator includes a query response time and / or a number of database connections; Based on the target-side performance indicator, it is detected that the target database performance exceeds the set indicator, and the write rate of the data synchronization task is dynamically adjusted, wherein the target-side performance indicator includes at least one of write throughput, storage utilization rate and index update time.

9. The data synchronization method according to claim 2, characterized in that: The data synchronization task is controlled in speed based on the execution condition of the synchronization task queue and / or the newly added synchronization task, further comprising: In response to the newly added synchronization task, detecting the priority and start time of the newly added synchronization task, and the adjustable synchronization tasks in the synchronization task queue, wherein the priority of the adjustable synchronization task is lower than the priority of the newly added synchronization task, and the start time of the adjustable synchronization task is later than the start time of the newly added synchronization task; The execution parameter of the adjustable synchronization task is adjusted based on a resource yield mechanism to allocate the required computing resources and network resources to the newly added synchronization task.

10. The data synchronization method according to claim 2, characterized in that: Starting the synchronization task queue to perform data synchronization operations also includes: Monitor the occupancy of computing resources and network resources by the plurality of synchronous task queues to obtain resource occupancy trends; The synchronous task queue is cooperatively speed-controlled based on the resource occupancy trend.

11. A data synchronization device, characterized in that: include: The test module is used to perform stress testing on the data synchronization task to be executed and obtain the result of the stress testing; A first generating module, configured to generate a parallel synchronization parameter of the data synchronization task based on a result of the stress test and an execution parameter of the data synchronization task; A second generating module is used to generate a synchronization task queue based on the parallel synchronization parameter, the priority and dependency of the data synchronization task, so as to start the synchronization task queue to perform a data synchronization operation; A synchronization module is used to control the speed of the data synchronization task based on the execution status of the synchronization task queue and / or the newly added synchronization task.

12. An electronic device, characterized in that: include: processor; as well as A memory, configured to store executable instructions of the processor; The processor is configured to execute the data synchronization method according to any one of claims 1 to 10 by executing the executable instructions.

13. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the data synchronization method according to any one of claims 1 to 10 is implemented.

14. A computer program product having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the data synchronization method according to any one of claims 1 to 10 is implemented.

Images