Data synchronization implementation method, device, system and medium for distributed parallel computing
Through distributed parallel computing technology, the hospital data synchronization tasks are dynamically allocated and dispatched, the problem of unbalanced resource management in multi-hospital data synchronization is solved, data update efficiency and accuracy are improved, and task execution deviations and resource waste are reduced.
Patent Information
- Application Number
- CN202410954014.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-17
- Publication Date
- 2024-11-08
- Estimated Expiration
- 2044-07-17
AI Technical Summary
In the multi-hospital data synchronization scenario, the data format inconsistent caused by differences in internal systems and software of the hospital, resulting in uneven resource management and scheduling of the data management platform, affecting the efficiency of data updates, and there are task congestion, idle resources, duplication of tasks and loss of tasks and tasks. etc.
Using distributed parallel computing method, dynamically allocate tasks to execution nodes by identifying the number and characteristics of synchronous tasks, and generate task time rounds to ensure that tasks are scheduled according to predetermined time intervals and scales, avoid task chaos and conflicts, and improve resource allocation balance reduces task repetition and loss.
It improves the execution efficiency and accuracy of data synchronization tasks, avoids task congestion and idle resources, reduces execution deviations, and improves data update efficiency and system resource utilization.
Smart Images

Figure CN118502922B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the technical field of data synchronization, and in particular to a method, device, system and medium for implementing data synchronization of distributed parallel computing. Background Art
[0002] Since different hospitals may use different medical systems and software, the formats of medical data or daily work data generated by different hospitals may be different. In order to carry out unified information management of different hospitals, a unified data management platform is generally established for different hospitals. By performing unified and standardized operations on the data, centralized management of the data generated by different hospitals can be achieved.
[0003] However, since the hospital is a dynamic environment, the data generated by the hospital needs to be updated in a timely manner to ensure the accuracy and effectiveness of the data. The relevant technology generally adopts a timed update method to synchronize the stored data. However, due to the large number of patients, departments, instruments, etc. within the hospital, the amount of data stored in the data management platform is generally large. When there is a large amount of data that needs to be updated and synchronized, the data management platform may affect the data update efficiency due to its own unbalanced resource management and scheduling. Summary of the invention
[0004] In order to reasonably allocate and utilize computing resources and thus improve data update efficiency, the present application provides a method, device, system and medium for implementing data synchronization in distributed parallel computing.
[0005] In a first aspect, the present application provides a method for implementing data synchronization of distributed parallel computing, which adopts the following technical solution:
[0006] A method for implementing data synchronization of distributed parallel computing, comprising:
[0007] Identify the synchronization quantity of multiple synchronization tasks and the task characteristics of each synchronization task. Each synchronization task corresponds to a synchronization script to be updated.
[0008] Based on the synchronization quantity and each task feature, determine a task allocation list for each execution node, each task allocation list including a plurality of synchronization tasks;
[0009] Identify the task amount of each synchronization task, and determine the task time wheel information corresponding to each execution node based on the synchronization quantity corresponding to each task allocation list and the task amount of each synchronization task, wherein the task time wheel information includes a time interval and a scale number;
[0010] Obtain the execution time of each synchronization task in each task allocation list, write the task allocation list of each execution node into the corresponding task time wheel according to each execution time and the corresponding task time wheel information, and obtain the scheduling time wheel corresponding to each execution node;
[0011] Control each execution node to execute synchronization tasks according to the corresponding scheduling time round.
[0012] By adopting the above technical solution, by identifying the number and characteristics of each synchronization task and allocating tasks based on this information, it is possible to ensure that tasks with similar or identical characteristics are allocated to corresponding execution nodes. By setting different execution contents for different execution nodes, the efficiency and accuracy of each execution node in executing synchronization tasks can be improved. Based on the number of synchronizations in each task allocation list and the number of tasks of each synchronization task, task time wheel information can be dynamically generated for each execution node. On the basis of the task time wheel, a scheduling time wheel corresponding to each execution node is generated according to the execution time of each synchronization task. Based on the scheduling time wheel of each execution node, each execution node is controlled to perform synchronization tasks, which can improve the resource allocation balance of each execution node and avoid task congestion or resource idleness during the execution of synchronization tasks. In addition, by controlling each execution node to perform tasks according to the scheduling time wheel of each execution node, it is possible to reduce the occurrence of task duplication, task loss, etc. during the execution of tasks, thereby improving the efficiency and accuracy of the task execution process.
[0013] In a possible implementation, the task allocation list of each execution node is written into the corresponding task time round according to each execution moment and the corresponding task time round information to obtain the scheduling time round corresponding to each execution node, including:
[0014] Identify the time interval and number of ticks in each task time wheel information;
[0015] Determine the task slot of each synchronization task in the corresponding task time wheel based on the time interval, the number of ticks and each execution time corresponding to each task time wheel information;
[0016] Write each synchronous task pair into the corresponding task slot to obtain the scheduling time wheel corresponding to each execution node.
[0017] By adopting the above technical solution, the synchronization system can achieve accurate time management by identifying the time interval and number of ticks in each task time wheel information, which means that each synchronization task will be scheduled according to the predetermined time interval and number of ticks, avoiding confusion and conflict in task execution. In addition, this allocation method also takes into account the characteristics, time requirements and capabilities of the execution nodes of each synchronization task, so as to improve the execution efficiency of each synchronization task.
[0018] In a possible implementation, determining the task slot of each synchronization task in the corresponding task time wheel based on the time interval, the number of ticks and each execution time corresponding to each task time wheel information includes:
[0019] Identify the start time of each task time wheel, and determine the slot index of each synchronization task according to the time interval, number of ticks, start time, and each execution time corresponding to each task time wheel information and the preset task slot calculation formula;
[0020] According to the slot index of each synchronization task, the task slot of each synchronization task in the corresponding task time wheel is determined, wherein the preset task slot calculation formula includes:
[0021] x = [(B_hour - A_hour) * 60 + (B_minute - A_minute)] mod (number of ticks * time interval);
[0022] Among them, A_hour is used to represent the hour corresponding to the start time, and A_minute is used to represent the minute corresponding to the start time;
[0023] B_hour is used to represent the hour corresponding to the execution time, and B_ minute is used to represent the minute corresponding to the execution time;
[0024] x is used to represent the slot index of the task slot corresponding to the execution time.
[0025] By adopting the above technical solution, since the number of task slots is not infinite, the task slot corresponding to each synchronization task is determined by taking the remainder method, which facilitates distinguishing multiple synchronization tasks. By setting a slot index for each preset task slot, and then determining the corresponding task slot based on the calculation result and the slot index, it is convenient to improve the accuracy of determining the task slot.
[0026] In a possible implementation, writing each synchronization task pair into a corresponding task slot includes:
[0027] Create a one-way linked list corresponding to each task slot;
[0028] Determine the linked list position of each synchronization task in the corresponding one-way linked list according to the execution time of each synchronization task;
[0029] According to the linked list position corresponding to each synchronization task, each synchronization task is written into the one-way linked list of the corresponding task slot.
[0030] By adopting the above technical solution, the synchronous tasks corresponding to the same task slot are managed by adding a one-way linked list, which facilitates improving data standardization, thereby reducing the probability of execution deviation in the scheduling process.
[0031] In a possible implementation, controlling each execution node to execute a synchronization task according to a corresponding scheduling time round includes:
[0032] Identify the scheduling period of each scheduling time round, and determine the task slot to be executed corresponding to the current time from each scheduling time round;
[0033] Obtain a one-way linked list to be executed corresponding to the task slot to be executed, and identify the execution time corresponding to each synchronization task data in the one-way linked list to be executed;
[0034] According to each execution time and the scheduling period corresponding to the current time, the target synchronization task corresponding to each execution node at the current time is determined, and the corresponding target synchronization task is executed. The target synchronization task is a synchronization task whose execution time is within the scheduling period corresponding to the current time.
[0035] By adopting the above technical solution and adding a scheduling cycle comparison link in the scheduling process, it is easy to distinguish the synchronization tasks corresponding to different scheduling cycles in the same one-way linked list, thereby further reducing the probability of execution deviation in the scheduling process.
[0036] In a possible implementation, the method of determining the synchronization task includes:
[0037] Obtain multiple synchronization scripts, and identify script features and update dates corresponding to each synchronization script;
[0038] Determine the synchronization strategy corresponding to each synchronization script according to each script feature and strategy mapping relationship, wherein the strategy mapping relationship is the corresponding relationship between the script feature and the synchronization strategy;
[0039] According to the update date and synchronization strategy of each synchronization script, determine the synchronization area of each synchronization script in the preset time window;
[0040] The synchronization script whose synchronization area is the update area is determined as the synchronization script to be updated.
[0041] By adopting the above technical solution, due to the differences in different source data systems, not all data can be updated or synchronized without restrictions. By adding a time window mechanism, it is possible to determine whether the synchronization script is within the update period, so as to screen the synchronization scripts to be updated. By eliminating the synchronization scripts that do not meet the corresponding update period, it helps to reduce the computing pressure of the synchronization system, thereby facilitating the improvement of the synchronization efficiency of the synchronization system.
[0042] In one possible implementation, the method further includes:
[0043] When receiving a synchronization instruction, identifying a synchronization feature type contained in the synchronization instruction, wherein the synchronization feature type includes second-level scheduling and balanced scheduling;
[0044] When the synchronization feature type is second-level scheduling, the synchronization task corresponding to the feature synchronization instruction is assigned to the first target execution node, and the first target execution node is controlled to execute the corresponding synchronization task;
[0045] When the synchronization feature type is balanced scheduling, the execution rate and execution safety level of each execution node are identified, and based on the synchronization task corresponding to the synchronization instruction of interest, the execution rate and execution safety level of each execution node, the second target execution node corresponding to each synchronization task is determined, and each second target execution node is controlled to execute the corresponding synchronization task.
[0046] By adopting the above technical solution, a variety of execution methods are provided according to different execution requirements, that is, a variety of task scheduling methods are provided, which can improve the flexibility of the task scheduling process. The appropriate scheduling method is selected according to the actual content that needs to be scheduled, rather than using a fixed mode for task scheduling, which facilitates improving the adaptability between synchronization tasks and scheduling methods.
[0047] In a second aspect, the present application provides a data synchronization implementation device for distributed parallel computing, which adopts the following technical solution:
[0048] A data synchronization implementation device for distributed parallel computing, comprising:
[0049] An identification module, used to identify the synchronization quantity of multiple synchronization tasks and the task characteristics of each synchronization task, each synchronization task corresponds to a synchronization script to be updated;
[0050] A distribution list determination module is used to determine a task distribution list for each execution node based on the synchronization quantity and each task feature, each task distribution list including a plurality of synchronization tasks;
[0051] A task time wheel determination module is used to identify the task amount of each synchronization task, and determine the task time wheel information corresponding to each execution node based on the synchronization quantity corresponding to each task allocation list and the task amount of each synchronization task, wherein the task time wheel information includes a time interval and a scale number;
[0052] The scheduling time wheel determination module is used to obtain the execution time of each synchronization task in each task allocation list, write the task allocation list of each execution node into the corresponding task time wheel according to each execution time and the corresponding task time wheel information, and obtain the scheduling time wheel corresponding to each execution node;
[0053] The execution task module is used to control each execution node to execute the synchronization task according to the corresponding scheduling time wheel.
[0054] By adopting the above technical solution, by identifying the number and characteristics of each synchronization task and allocating tasks based on this information, it is possible to ensure that tasks with similar or identical characteristics are allocated to corresponding execution nodes. By setting different execution contents for different execution nodes, the efficiency and accuracy of each execution node in executing synchronization tasks can be improved. Based on the number of synchronizations in each task allocation list and the number of tasks of each synchronization task, task time wheel information can be dynamically generated for each execution node. On the basis of the task time wheel, a scheduling time wheel corresponding to each execution node is generated according to the execution time of each synchronization task. Based on the scheduling time wheel of each execution node, each execution node is controlled to perform synchronization tasks, which can improve the resource allocation balance of each execution node and avoid task congestion or resource idleness during the execution of synchronization tasks. In addition, by controlling each execution node to perform tasks according to the scheduling time wheel of each execution node, it is possible to reduce the occurrence of task duplication, task loss, etc. during the execution of tasks, thereby improving the efficiency and accuracy of the task execution process.
[0055] In a third aspect, the present application provides a synchronization system, which adopts the following technical solution:
[0056] A synchronization system, comprising:
[0057] at least one processor;
[0058] Memory;
[0059] At least one application, wherein the at least one application is stored in a memory and configured to be executed by at least one processor, and the at least one application is configured to: execute the above-mentioned distributed parallel computing data synchronization implementation method.
[0060] In a fourth aspect, the present application provides a computer-readable storage medium, which adopts the following technical solution:
[0061] A computer-readable storage medium includes: a computer program that can be loaded by a processor and execute the above-mentioned distributed parallel computing data synchronization implementation method.
[0062] In summary, the present application includes at least one of the following beneficial technical effects:
[0063] By identifying the number and characteristics of each synchronization task and assigning tasks based on this information, it is possible to ensure that tasks with similar or identical characteristics are assigned to corresponding execution nodes. By setting different execution contents for different execution nodes, the efficiency and accuracy of each execution node in executing synchronization tasks can be improved. Based on the number of synchronizations in each task assignment list and the number of tasks of each synchronization task, task time wheel information can be dynamically generated for each execution node. On the basis of the task time wheel, a scheduling time wheel corresponding to each execution node is generated according to the execution time of each synchronization task. Based on the scheduling time wheel of each execution node, each execution node is controlled to perform synchronization tasks, which can improve the resource allocation balance of each execution node and avoid task congestion or resource idleness during the execution of synchronization tasks. In addition, controlling each execution node to perform tasks according to the scheduling time wheel of each execution node can reduce the occurrence of task duplication, task loss, etc. during the execution of tasks, thereby improving the efficiency and accuracy of the task execution process.
[0064] Since the number of task slots is not infinite, the task slot corresponding to each synchronization task is determined by taking the remainder method, which facilitates distinguishing multiple synchronization tasks. By setting a slot index for each preset task slot, and then determining the corresponding task slot based on the calculation result and the slot index, it is convenient to improve the accuracy of determining the task slot. BRIEF DESCRIPTION OF THE DRAWINGS
[0065] Figure 1 It is a flow chart of a method for implementing data synchronization of distributed parallel computing in an embodiment of the present application;
[0066] Figure 2 is a schematic diagram of a time window in an embodiment of the present application;
[0067] Figure 3 is a schematic diagram of a time wheel in an embodiment of the present application;
[0068] Figure 4 It is a structural diagram of a data synchronization implementation device for distributed parallel computing in an embodiment of the present application;
[0069] Figure 5 It is a structural diagram of a synchronization system in an embodiment of the present application. DETAILED DESCRIPTION
[0070] The following is combined with Figure 1-5 This application is described in further detail.
[0071] After reading this specification, those skilled in the art may make non-creative modifications to this embodiment as needed, but such modifications are protected by patent law as long as they are within the scope of the claims of this application.
[0072] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.
[0073] Specifically, an embodiment of the present application provides a method for implementing data synchronization of distributed parallel computing, which is executed by a synchronization system. The synchronization system can be an independent physical server, a server cluster or a distributed synchronization system composed of multiple physical servers, or a cloud server that provides cloud computing services, which is not specifically limited in the embodiments of the present application.
[0074] refer to Figure 1 , Figure 1 1 is a flow chart of a method for implementing data synchronization of distributed parallel computing in an embodiment of the present application. The method includes steps S110-S150, wherein:
[0075] Step S110: Identify the synchronization quantities of multiple synchronization tasks and the task characteristics of each synchronization task, each synchronization task corresponds to a synchronization script to be updated.
[0076] Specifically, the synchronization task is the task corresponding to the synchronization script to be updated. The synchronization script to be updated corresponds to the synchronization task one by one. The update of the corresponding synchronization script can be completed by executing the synchronization task. The synchronization script is formed when the detailed data generated in the hospital is synchronized to the metadata. Since the hospital contains many business sources, each business source corresponds to a different data generation object. The data generation object can be an instrument, a doctor, a nurse, and a patient, etc. Different data generation objects correspond to different detailed data. For the convenience of management, different detailed data are generally stored in separate tables according to the data generation object, that is, the detailed data contained in the same data table originates from the same data generation object. In addition, each data table contains not only detailed data, but also metadata corresponding to the detailed data. The metadata corresponding to the detailed data includes data name, type, format, and range, etc. The detailed data is synchronized to the metadata, that is, the detailed data is filled in according to the content contained in the metadata. In addition, since there are many data generation objects and each data generation object corresponds to a large number of detailed data, the detailed data generated by the entire hospital and the corresponding metadata can form hundreds of data tables, and each data table can be regarded as a synchronization script.
[0077] The synchronization script corresponding to the synchronization task is a synchronization script that meets the conditions. Since different hospitals have different corresponding data update strategies, different business sources in the same hospital also have different corresponding update strategies. For example, business source 1 stipulates that data after 3 months is not allowed to be modified, and business source 2 stipulates that data after 4 months is not allowed to be modified. When determining the synchronization task, it is necessary to judge whether the synchronization script meets the update conditions, and not all synchronization scripts are regarded as synchronization scripts to be updated. The method of determining the synchronization task can specifically include:
[0078] Obtain multiple synchronization scripts, and identify script features and update dates corresponding to each synchronization script;
[0079] According to the mapping relationship between each script feature and the strategy, the synchronization strategy corresponding to each synchronization script is determined. The strategy mapping relationship is the correspondence between the script feature and the synchronization strategy; according to the update date and synchronization strategy of each synchronization script, the synchronization area of each synchronization script in the preset time window is determined; the synchronization script whose synchronization area is the update area is determined as the synchronization script to be updated.
[0080] Specifically, different hospitals or different business sources have corresponding update cycles. For example, business source 1 needs to be synchronized once every 30 days; business source 2 needs to be synchronized once every 45 days; business source 3 needs to be synchronized once every 60 days. When the synchronization date of each business source is reached, all synchronization scripts corresponding to the business source need to be screened to determine the synchronization script to be updated. The synchronization script contains corresponding script features and update dates, wherein the script features are used to characterize the content of the current synchronization script. The specific form of the script features is not specifically limited in the embodiment of the present application, and can be set by relevant technical personnel, as long as multiple synchronization scripts can be distinguished by the script features. The update date is the date when the synchronization script is first formed. The update date can be used to judge whether the synchronization script is within the synchronization period. Only when it is within the synchronization period can the synchronization script be updated. Different script features correspond to different synchronization strategies. According to the strategy mapping relationship and the script features, the synchronization strategy of each synchronization script can be determined. The synchronization strategy is used to characterize the synchronization period of the synchronization script. The specific content of the strategy mapping relationship is not specifically limited in the embodiment of the present application, and can be set by relevant technical personnel.
[0081] Each synchronization script corresponds to a preset time window, which contains a monthly coordinate axis. When the update date of the synchronization script is in the corresponding area of the time window, the synchronization script is determined as the synchronization script to be updated. For example, the medical record synchronization system of XX Province stipulates that data after 4 months is not allowed to be modified. When synchronizing detailed data to metadata in the data warehouse, the synchronization strategy of the synchronization script is as follows: Figure 2 As shown, if the current date is early November 2023, the data from June 2023 to October 2023 can be updated synchronously, that is, the period corresponding to June 2023 to October 2023 is determined as the time window, and the synchronization script with the update date in this area can be determined as the synchronization script to be updated, that is, the detailed data corresponding to the synchronization script with the update date in this area can be updated. However, since the data of October 2023 is the most recently added content, the area corresponding to the time window can be divided. For example, the detailed data corresponding to June 2023 to September 2023 can be updated synchronously, and the detailed data corresponding to October 2023 can be added synchronously. The synchronous update is to compare the detailed data corresponding to two or more business sources, and update the data items with differences or changes to ensure that the corresponding detailed data in all business sources are consistent, and the synchronous addition is to add the latest detailed data from one business source to another business source to ensure that the corresponding detailed data in all business sources are the latest and complete. By eliminating synchronization scripts that do not meet the corresponding update deadlines, it can also help reduce the computing pressure of the synchronization system, thereby facilitating the improvement of the synchronization efficiency of the synchronization system.
[0082] The synchronization number of a synchronization task is the total number of multiple synchronization tasks. The task characteristics of the synchronization task are used to characterize the requirements during the task execution process, such as fast execution, accurate execution, etc. The specific content is not specifically limited in the embodiments of the present application.
[0083] Step S120: Based on the synchronization quantity and the characteristics of each task, determine the task allocation list of each execution node, each task allocation list includes multiple synchronization tasks.
[0084] Specifically, the current synchronization system includes multiple servers, each of which has different uses. The target server is used to process synchronization tasks, and the target server includes multiple worker terminals. After receiving the synchronization system instruction, the target server can allocate the corresponding synchronization task to the worker terminal that meets its task characteristics, and can control the worker terminal to execute the allocated synchronization task after receiving the synchronization system instruction. After the allocation is completed, an allocation list can be generated for each woker terminal according to the allocation result, that is, for each execution node, and the allocation list corresponding to each worker terminal contains the synchronization tasks that the execution node needs to execute.
[0085] Step S130: Identify the task amount of each synchronization task, and determine the task time wheel information corresponding to each execution node based on the synchronization quantity corresponding to each task allocation list and the task amount of each synchronization task, the task time wheel information including the time interval and the number of scales.
[0086] Specifically, the amount of a synchronization task is related to the execution time of the synchronization task. Sorting multiple synchronization tasks according to the execution time of each synchronization task and controlling the corresponding worker end to execute in order can avoid execution confusion during the task execution process. In this application, a time wheel is used to sort each synchronization task. In addition, a linked list can also be used for sorting. The specific sorting method is not specifically limited in the embodiments of this application. The algorithm model of the time wheel is as follows: Figure 3 As shown, each time wheel has N slots, and the time interval between two slots is fixed. At each time interval, the pointer in the time wheel moves forward one grid. When the pointer points to the task slot, it starts to process the synchronization task in the current task slot. The number of scales is the number of task slots in the time wheel.
[0087] The time wheel is a scheduling model that efficiently utilizes thread resources for batch scheduling. It can bind a large number of synchronization tasks to the same scheduling model, and use this scheduling model to manage, trigger and run all synchronization tasks. It can efficiently manage various delayed tasks, periodic tasks and notification tasks.
[0088] Step S140: Obtain the execution time of each synchronization task in each task allocation list, write the task allocation list of each execution node into the corresponding task time wheel according to each execution time and the corresponding task time wheel information, and obtain the scheduling time wheel corresponding to each execution node.
[0089] Specifically, different synchronization tasks may have different corresponding execution times due to different contents. For example, there are three scheduled synchronization tasks, which are executed after 1 second, 6 seconds, and 13 seconds respectively. The execution time corresponding to the synchronization task can be marked by the relevant staff in the synchronization script to be updated corresponding to the synchronization task by adding tags. The task time wheel corresponding to each execution node is empty before the synchronization task is executed. After the synchronization tasks in the task allocation list are written, a scheduling time wheel is formed. When writing the synchronization tasks, the time slot corresponding to each synchronization task can be determined according to the execution time of each synchronization task. According to each execution time and the corresponding task time wheel information, the task allocation list of each execution node is written into the corresponding task time wheel to obtain the scheduling time wheel corresponding to each execution node, which specifically includes:
[0090] Identify the time interval and number of ticks in each task time wheel information; determine the task slot of each synchronization task in the corresponding task time wheel based on the time interval, number of ticks and each execution time corresponding to each task time wheel information; write each synchronization task pair into the corresponding task slot to obtain the scheduling time wheel corresponding to each execution node.
[0091] Specifically, the number of task slots contained in the task time wheel can be determined according to the task time wheel information, and each task slot can be numbered. For example, the first task slot can be numbered 0, and the second task slot can be numbered 1. The task slot corresponding to each synchronization task is determined by the time interval, the number of scales and the number of task slots, and then the synchronization task is written into the corresponding task slot. After all the synchronization tasks are written into the corresponding task slots, the scheduling time wheel corresponding to each execution node can be obtained. Since the execution time corresponding to each task slot in the scheduling time wheel is different, for example, a scheduling time wheel contains 5 task slots, and the time interval between each task slot is 2 seconds, that is, after 2 seconds, the pointer in the scheduling time wheel will move from the corresponding area of the current task slot to the corresponding area of the next task slot, that is, the corresponding execution node needs to execute the synchronization task corresponding to the next task slot. Assuming that there are 3 scheduled synchronization tasks, synchronization task 1 needs to be executed after 1 second, synchronization task 2 needs to be executed after 4 seconds, and synchronization task 3 needs to be executed after 6 seconds. At this time, synchronization task 1 can be written into the first task slot; synchronization task 2 can be written into the second task slot; synchronization task 3 can be written into the third task slot. When executing synchronization tasks according to the scheduling time wheel, the corresponding synchronization tasks can be executed on time at the corresponding time. This means that each synchronization task will be scheduled according to the predetermined time interval and number of scales, avoiding confusion and conflict in task execution. In addition, this allocation method also takes into account the characteristics, time requirements and capabilities of each synchronization task, which is convenient for improving the execution efficiency of each synchronization task.
[0092] Furthermore, in order to improve the accuracy of determining the task slot, the task slot of each synchronization task in the corresponding task time wheel is determined based on the time interval, the number of ticks and each execution time corresponding to each task time wheel information, including:
[0093] Identify the start time of each task time wheel, and determine the slot subscript of each synchronization task according to the time interval, number of scales, start time, each execution time and the preset task slot calculation formula corresponding to each task time wheel information; determine the task slot of each synchronization task in the corresponding task time wheel according to the slot subscript of each synchronization task, wherein the preset task slot calculation formula includes: x=[(B_hour - A_hour) × 60 + (B_minute - A_minute)] mod(number of scales*time interval); wherein A_hour is used to represent the hour corresponding to the start time, and A_ minute is used to represent the minute corresponding to the start time; B_hour is used to represent the hour corresponding to the execution time, and B_ minute is used to represent the minute corresponding to the execution time; x is used to represent the slot subscript of the task slot corresponding to the execution time.
[0094] Specifically, the start time of the task time wheel is the time when the task time wheel starts to rotate, and the execution time of the synchronization task is the time when the synchronization task needs to be executed. For example, the execution time of synchronization task a is 20:06, which means that synchronization task a needs to be executed by the corresponding execution node at 20:06. (B_hour - A_hour) * 60 + (B_minute - A_minute) is used to represent the duration between the execution time and the start time. For example, the start time is 20:00, and the execution time of the synchronization task is 20:06. At this time, (B_hour - A_hour) * 60 + (B_minute - A_minute) = (20-20) * 60 + (6-0) = 6 minutes, that is, the duration between the execution time and the start time is 6 minutes. The remainder of the product of the duration between the execution time and the start time and the number of ticks and the time interval is taken, and the calculation result is determined as the slot index. For example, when the execution time of synchronization task a is 20:06, the corresponding start time of the task time wheel is 20:00, the number of ticks is 5, and the time interval is 2min, x=[(20-20)*60+(6-0)] mod (5*2)=3 mod 5=3. At this time, synchronization task a can be written into the task slot with slot index 3. Since the design of the task time wheel is annular, when there are multiple synchronization tasks, all synchronization tasks can also be written by the above calculation formula. The task slot corresponding to each synchronization task is determined by taking the remainder, which is convenient for distinguishing multiple synchronization tasks. By setting a slot index for each preset task slot, and then determining the corresponding task slot based on the calculation result and the slot index, it is convenient to improve the accuracy of determining the task slot.
[0095] When the remainders obtained after calculation of different synchronization tasks are consistent, each synchronization task pair is written into the corresponding task slot, including:
[0096] Create a one-way linked list corresponding to each task slot; determine the linked list position of each synchronization task in the corresponding one-way linked list according to the execution time of each synchronization task; and write each synchronization task into the one-way linked list of the corresponding task slot according to the linked list position corresponding to each synchronization task.
[0097] Specifically, when the execution time of synchronization task b1 is 20:00, the execution time of synchronization task b2 is 20:02, the execution time of synchronization task b3 is 20:06, the execution time of synchronization task b4 is 20:08, and the execution time of synchronization task b5 is 20:10, the start time of the corresponding task time wheel is 20:00, the number of scales is 5, and the time interval is 2min. At this time, after time slot calculation, it can be determined that these synchronization tasks will be inserted into the task slots with slot subscripts 0, 1, 2, 3, 4, and 0 respectively. It can be seen that the task slots corresponding to synchronization task b1 and synchronization task b5 are the same. At this time, in order to distinguish different synchronization tasks, the synchronization tasks can be stored by constructing a one-way data linked list.
[0098] Each task slot corresponds to a one-way data linked list. When the pointer of the task time wheel points to the current task slot, the execution node is controlled to execute the synchronization task in the one-way linked list corresponding to the current task slot. Since different synchronization tasks correspond to different execution times, when the same task slot corresponds to multiple synchronization tasks, the multiple synchronization tasks can be sorted and written according to the execution time of each synchronization task, which is convenient for improving data standardization, thereby reducing the possibility of execution deviation in the scheduling process. The length of the one-way linked list is not specifically limited in the embodiment of the present application, and can be added and deleted according to the actual writing situation. In order to avoid repeated execution, the synchronization system can directly clear the synchronization task in the one-way data linked list after the synchronization task is executed.
[0099] Step S150: Control each execution node to execute the synchronization task according to the corresponding scheduling time round.
[0100] Specifically, each execution node reads the synchronization task in the one-way linked list in the next task slot according to the scheduling time wheel cycle and executes it until the scheduling time wheel is empty.
[0101] In order to supervise the execution process, the execution node will generate a write-back log during the execution process and feed the write-back log back to the synchronization system. When the synchronization system detects that there is a synchronization task that has not been executed successfully, it will mark the synchronization task as an exception and send the abnormal synchronization task again to the next round of synchronization task allocation until the synchronization task is successfully executed, thereby improving the reliability of the synchronization task execution process.
[0102] By identifying the number and characteristics of each synchronization task and assigning tasks based on this information, it is possible to ensure that tasks with similar or identical characteristics are assigned to corresponding execution nodes. By setting different execution contents for different execution nodes, the efficiency and accuracy of each execution node in executing synchronization tasks can be improved. Based on the number of synchronizations in each task assignment list and the number of tasks of each synchronization task, task time wheel information can be dynamically generated for each execution node. On the basis of the task time wheel, a scheduling time wheel corresponding to each execution node is generated according to the execution time of each synchronization task. Based on the scheduling time wheel of each execution node, each execution node is controlled to perform synchronization tasks, which can improve the resource allocation balance of each execution node and avoid task congestion or resource idleness during the execution of synchronization tasks. In addition, controlling each execution node to perform tasks according to the scheduling time wheel of each execution node can reduce the occurrence of task duplication, task loss, etc. during the execution of tasks, thereby improving the efficiency and accuracy of the task execution process.
[0103] Furthermore, in order to reduce the probability of execution deviation during the scheduling process, each execution node is controlled to execute the synchronization task according to the corresponding scheduling time round, which may specifically include:
[0104] Identify the scheduling cycle of each scheduling time wheel, and determine the task slot to be executed corresponding to the current moment from each scheduling time wheel; obtain the one-way linked list to be executed corresponding to the task slot to be executed, and identify the execution time corresponding to each synchronization task data in the one-way linked list to be executed; according to the scheduling cycle corresponding to each execution moment and the current moment, determine the target synchronization task corresponding to each execution node at the current moment, and execute the corresponding target synchronization task, which is a synchronization task whose execution time is within the scheduling cycle corresponding to the current moment.
[0105] Specifically, the scheduling period is the time required for the pointer in the scheduling time wheel to rotate one circle. For example, when the time interval of the scheduling time wheel is 1s and the number of scales is 12, the corresponding scheduling period is 12s. Since the time intervals and the number of scales corresponding to different scheduling time wheels may be different, the scheduling periods corresponding to different scheduling time wheels may be different. The task slot to be executed corresponding to the current moment is the task slot pointed to by the pointer in the scheduling time wheel at the current moment, that is, the execution node at the current moment needs to point to the synchronization task in the one-way linked list corresponding to the task slot. The one-way linked list may contain multiple synchronization tasks. When executing a synchronization task, not all synchronization tasks in the one-way linked list are executed. Instead, they need to be screened from multiple synchronization tasks according to the execution time of each synchronization task. When the same one-way linked list contains multiple synchronization tasks, different The synchronization tasks may belong to the same scheduling cycle. For example, when the time interval of the scheduling time wheel is 2 minutes, the start time is 20:00, and the number of scales is 5, the execution time of the synchronization task c1 is 20:01, and the execution time of the synchronization task c2 is 20:02. At this time, the synchronization tasks c1 and c2 will be written into the one-way linked list corresponding to the task slot with the slot subscript 0; different synchronization tasks may also belong to different scheduling cycles. For example, when the time interval of the scheduling time wheel is 2 minutes, the start time is 20:00, the execution time of the synchronization task c1 is 20:01, the execution time of the synchronization task c2 is 20:02, and the execution time of the synchronization task c3 is 20:10. The synchronization tasks c1, c2 and c3 will all be written into the one-way linked list corresponding to the task slot with the slot subscript 0.
[0106] The target synchronization task is a synchronization task in the one-way linked list whose execution time belongs to the scheduling period corresponding to the current time. The number of target synchronization tasks can be one or more. The specific number is not specifically limited in the embodiment of the present application. For example, in the above example, when the pointer points to the task slot with the slot subscript 0, synchronization task c1 and synchronization task c2 belong to the target synchronization task, that is, only synchronization task 1 and synchronization task 2 will be executed in the current scheduling period, and synchronization task 3 will not be executed. When executing synchronization task 1 and synchronization task 2, the execution operation must also be strictly performed according to the execution time of each synchronization task. After synchronization task 1 and synchronization task 3 are executed, synchronization task 1 and synchronization task 2 will no longer be stored in the one-way linked list corresponding to the task slot with the slot subscript 0. By adding a scheduling period comparison link in the scheduling process, it is easy to distinguish the synchronization tasks corresponding to different scheduling periods in the same one-way linked list, thereby further reducing the probability of execution deviation in the scheduling process.
[0107] Furthermore, in order to improve the adaptability between the synchronization task and the scheduling mode, the method provided in the embodiment of the present application also includes:
[0108] When a synchronization instruction of interest is received, the synchronization feature type contained in the synchronization instruction of interest is identified, and the synchronization feature types include second-level scheduling and balanced scheduling; when the synchronization feature type is second-level scheduling, the synchronization task corresponding to the feature synchronization instruction is assigned to the first target execution node, and the first target execution node is controlled to execute the corresponding synchronization task; when the synchronization feature type is balanced scheduling, the execution rate and execution safety level of each execution node are identified, and based on the synchronization task corresponding to the synchronization instruction of interest, the execution rate of each execution node and the execution safety level, the second target execution node corresponding to each synchronization task is determined, and each second target execution node is controlled to execute the corresponding synchronization task.
[0109] Specifically, in addition to the use of the time wheel mechanism for task scheduling mentioned in the above embodiments, the embodiments of the present application also include second-level scheduling and balanced scheduling, wherein the second-level scheduling is characterized by an extremely high operating frequency, which is generally executed every several seconds to tens of seconds. Since the second-level scheduling requires an extremely fast response rate, a large amount of computing resources and storage resources may be required when executing the second-level scheduling, which may lead to high costs and low resource utilization. Therefore, not all execution nodes can execute second-level scheduling. In order to balance the computing resources on the server side, an execution node can be selected from the execution nodes on the server side to execute second-level scheduling. When relevant instructions are detected, the synchronization system can directly assign the synchronization task corresponding to the relevant instructions to the first target execution node without going through the processes of seer-side allocation, writing to a one-way linked list, and waiting for scheduling.
[0110] The characteristic of balanced scheduling is that it can dynamically adjust the execution order and resource allocation of synchronization tasks according to the actual load of the synchronization system and the changes in synchronization tasks. This can better adapt to changes in the synchronization system environment and synchronization task requirements, and improve the flexibility and scalability of the synchronization system. In addition, multiple synchronization tasks can be evenly distributed to multiple execution nodes, one execution node corresponds to one synchronization task, and different tasks are assigned to different host nodes, which facilitates the simultaneous execution of multiple synchronization tasks, thereby improving the task execution rate.
[0111] The premise for implementing second-level scheduling and balanced scheduling is that the number of synchronization tasks is small. When the number of synchronization tasks is large, due to the limited number of first target execution nodes, even if the synchronization tasks are directly assigned to the first target execution node through the synchronization system, due to the large number of synchronization tasks, the synchronization tasks may need to queue up for execution at the first target execution node. At this time, second-level scheduling cannot be achieved; since the premise for implementing balanced scheduling is that one execution node corresponds to one synchronization task, when the number of synchronization tasks is large, one execution node may need to correspond to multiple synchronization tasks. At this time, balanced scheduling may not be achieved. Therefore, second-level scheduling or balanced scheduling can only be achieved when the number of synchronization tasks is small.
[0112] However, multiple execution methods are provided according to different execution requirements, that is, multiple task scheduling methods are provided, which can improve the flexibility of the task scheduling process and select the appropriate scheduling method according to the actual content that needs to be scheduled, rather than using a fixed mode for task scheduling, so as to improve the adaptability between the synchronization task and the scheduling method.
[0113] The above embodiment introduces a method for implementing data synchronization of distributed parallel computing from the perspective of method flow. The following embodiment introduces a device for implementing data synchronization of distributed parallel computing from the perspective of virtual modules or virtual units. For details, please refer to the following embodiment.
[0114] The present application embodiment provides a data synchronization implementation device for distributed parallel computing, such as Figure 4 As shown, the device may specifically include an identification module 410, a distribution list determination module 420, a task time round determination module 430, a scheduling time round determination module 440, and a task execution module 450, wherein:
[0115] An identification module 410 is used to identify the synchronization quantity of multiple synchronization tasks and the task characteristics of each synchronization task, each synchronization task corresponds to a synchronization script to be updated;
[0116] A determination allocation list module 420 is used to determine a task allocation list for each execution node based on the synchronization quantity and each task feature, each task allocation list including a plurality of synchronization tasks;
[0117] The task time wheel determination module 430 is used to identify the task amount of each synchronization task, and determine the task time wheel information corresponding to each execution node based on the synchronization quantity corresponding to each task allocation list and the task amount of each synchronization task, wherein the task time wheel information includes the time interval and the number of scales;
[0118] The scheduling time wheel determination module 440 is used to obtain the execution time of each synchronization task in each task allocation list, write the task allocation list of each execution node into the corresponding task time wheel according to each execution time and the corresponding task time wheel information, and obtain the scheduling time wheel corresponding to each execution node;
[0119] The task execution module 450 is used to control each execution node to execute the synchronization task according to the corresponding scheduling time round.
[0120] In a possible implementation, the scheduling time wheel determination module 440 writes the task allocation list of each execution node into the corresponding task time wheel according to each execution moment and the corresponding task time wheel information, and obtains the scheduling time wheel corresponding to each execution node, specifically for:
[0121] Identify the time interval and number of ticks in each task time wheel information;
[0122] Determine the task slot of each synchronization task in the corresponding task time wheel based on the time interval, the number of ticks and each execution time corresponding to each task time wheel information;
[0123] Write each synchronous task pair into the corresponding task slot to obtain the scheduling time wheel corresponding to each execution node.
[0124] In one possible implementation, the scheduling time wheel determination module 440 is specifically used to determine the task slot of each synchronization task in the corresponding task time wheel based on the time interval, the number of ticks and each execution time corresponding to each task time wheel information:
[0125] Identify the start time of each task time wheel, and determine the slot index of each synchronization task according to the time interval, number of ticks, start time, and each execution time corresponding to each task time wheel information and the preset task slot calculation formula;
[0126] According to the slot index of each synchronization task, the task slot of each synchronization task in the corresponding task time wheel is determined, wherein the preset task slot calculation formula includes:
[0127] x = [(B_hour - A_hour) * 60 + (B_minute - A_minute)] mod (number of ticks * time interval);
[0128] Among them, A_hour is used to represent the hour corresponding to the start time, and A_minute is used to represent the minute corresponding to the start time;
[0129] B_hour is used to represent the hour corresponding to the execution time, and B_ minute is used to represent the minute corresponding to the execution time;
[0130] x is used to represent the slot index of the task slot corresponding to the execution time.
[0131] In a possible implementation, when the scheduling time wheel module 440 writes each synchronization task pair into the corresponding task slot, it is specifically used to:
[0132] Create a one-way linked list corresponding to each task slot;
[0133] Determine the linked list position of each synchronization task in the corresponding one-way linked list according to the execution time of each synchronization task;
[0134] According to the linked list position corresponding to each synchronization task, each synchronization task is written into the one-way linked list of the corresponding task slot.
[0135] In a possible implementation, when controlling each execution node to execute a synchronization task according to a corresponding scheduling time round, the execution task module 450 is specifically used to:
[0136] Identify the scheduling period of each scheduling time round, and determine the task slot to be executed corresponding to the current time from each scheduling time round;
[0137] Obtain the pending one-way linked list corresponding to the pending task slot, and identify the execution time corresponding to each synchronization task data in the pending one-way linked list;
[0138] According to each execution time and the scheduling period corresponding to the current time, the target synchronization task corresponding to each execution node at the current time is determined, and the corresponding target synchronization task is executed. The target synchronization task is a synchronization task whose execution time is within the scheduling period corresponding to the current time.
[0139] In a possible implementation, the device further includes:
[0140] The script identification module 410 is used to obtain multiple synchronization scripts and identify the script features and update date corresponding to each synchronization script;
[0141] A synchronization strategy determination module is used to determine the synchronization strategy corresponding to each synchronization script according to each script feature and the strategy mapping relationship, where the strategy mapping relationship is the corresponding relationship between the script feature and the synchronization strategy;
[0142] A synchronization area determination module is used to determine the synchronization area of each synchronization script in a preset time window according to the update date and synchronization strategy of each synchronization script;
[0143] The module for determining synchronization scripts to be updated is used to determine the synchronization scripts whose synchronization area is the update area as the synchronization scripts to be updated.
[0144] In a possible implementation, the device further includes:
[0145] A feature type identification module is used to identify the synchronization feature type contained in the synchronization instruction when receiving the synchronization instruction, and the synchronization feature type includes second-level scheduling and balanced scheduling;
[0146] The first type execution module is used to assign the synchronization task corresponding to the feature synchronization instruction to the first target execution node when the synchronization feature type is second-level scheduling, and control the first target execution node to execute the corresponding synchronization task;
[0147] The second type execution module is used to identify the execution rate and execution safety level of each execution node when the synchronization feature type is balanced scheduling, and based on the synchronization task corresponding to the synchronization instruction, the execution rate and execution safety level of each execution node, determine the second target execution node corresponding to each synchronization task, and control each second target execution node to execute the corresponding synchronization task.
[0148] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the distributed parallel computing data synchronization implementation device described above can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.
[0149] The embodiment of the present application provides a synchronization system, such as Figure 5 As shown, Figure 5 The synchronization system 500 shown includes: a processor 501 and a memory 503. The processor 501 and the memory 503 are connected, such as through a bus 502. Optionally, the synchronization system 500 may also include a transceiver 504. It should be noted that in actual applications, the transceiver 504 is not limited to one, and the structure of the synchronization system 500 does not constitute a limitation on the embodiments of the present application.
[0150] Processor 501 may be a CPU (Central Processing Unit), a general-purpose processor, a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array) or other programmable logic devices, transistor logic devices, hardware components or any combination thereof. It may implement or execute various exemplary logic blocks, modules and circuits described in conjunction with the disclosure of this application. Processor 501 may also be a combination that implements computing functions, such as a combination of one or more microprocessors, a combination of a DSP and a microprocessor, etc.
[0151] The bus 502 may include a path to transmit information between the above components. The bus 502 may be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus. The bus 502 may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 5 The fact that only one line is used in the diagram does not mean that there is only one bus or only one type of bus.
[0152] The memory 503 may be a ROM (Read Only Memory) or other types of static storage devices that can store static information and instructions, a RAM (Random Access Memory) or other types of dynamic storage devices that can store information and instructions, or an EEPROM (Electrically Erasable Programmable Read Only Memory), a CD-ROM (Compact Disc Read Only Memory) or other optical disk storage, optical disk storage (including compressed optical disk, laser disk, optical disk, digital versatile disk, Blu-ray disk, etc.), a magnetic disk storage medium or other magnetic storage device, or any other medium that can be used to carry or store the desired program code in the form of instructions or data structures and can be accessed by a computer, but is not limited thereto.
[0153] The memory 503 is used to store the application code for executing the solution of the present application, and the execution is controlled by the processor 501. The processor 501 is used to execute the application code stored in the memory 503 to implement the contents shown in the above method embodiment. Figure 5 The synchronization system shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.
[0154] An embodiment of the present application provides a computer-readable storage medium, on which a computer program is stored. When the computer-readable storage medium is run on a computer, the computer can execute the corresponding content in the aforementioned method embodiment.
[0155] It should be understood that, although the steps in the flowchart of the accompanying drawings are displayed in sequence as indicated by the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least a part of the steps in the flowchart of the accompanying drawings may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be executed in turn or alternately with other steps or at least a part of the sub-steps or stages of other steps.
[0156] The above description is only a partial implementation method of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.
Claims
1. A method for implementing data synchronization in distributed parallel computing. It is characterized in that include: Identify the synchronization quantity of multiple synchronization tasks and the task characteristics of each synchronization task. Each synchronization task corresponds to a synchronization script to be updated. Based on the synchronization quantity and each task feature, determine a task allocation list for each execution node, each task allocation list including a plurality of synchronization tasks; Identify the task amount of each synchronization task, and determine the task time wheel information corresponding to each execution node based on the synchronization quantity corresponding to each task allocation list and the task amount of each synchronization task, wherein the task time wheel information includes a time interval and a scale number; Obtain the execution time of each synchronization task in each task allocation list, write the task allocation list of each execution node into the corresponding task time wheel according to each execution time and the corresponding task time wheel information, and obtain the scheduling time wheel corresponding to each execution node; Control each execution node to execute synchronization tasks according to the corresponding scheduling time wheel; Wherein, the task allocation list of each execution node is written into the corresponding task time wheel according to each execution moment and the corresponding task time wheel information to obtain the scheduling time wheel corresponding to each execution node, including: Identify the time interval and number of ticks in each task time wheel information; Determine the task slot of each synchronization task in the corresponding task time wheel based on the time interval, the number of ticks and each execution time corresponding to each task time wheel information; Write each synchronous task pair into the corresponding task slot to obtain the scheduling time wheel corresponding to each execution node; The step of determining the task slot of each synchronization task in the corresponding task time wheel based on the time interval, the number of ticks and each execution time corresponding to each task time wheel information includes: Identify the start time of each task time wheel, and determine the slot index of each synchronization task according to the time interval, number of ticks, start time, and each execution time corresponding to each task time wheel information and the preset task slot calculation formula; According to the slot index of each synchronization task, the task slot of each synchronization task in the corresponding task time wheel is determined, wherein the preset task slot calculation formula includes: ; Among them, A_hour is used to represent the hour corresponding to the start time, and A_minute is used to represent the minute corresponding to the start time; B_hour is used to represent the hour corresponding to the execution time, and B_ minute is used to represent the minute corresponding to the execution time; x is used to represent the slot index of the task slot corresponding to the execution time; Writing each synchronization task pair into a corresponding task slot includes: Create a one-way linked list corresponding to each task slot; Determine the linked list position of each synchronization task in the corresponding one-way linked list according to the execution time of each synchronization task; According to the linked list position corresponding to each synchronization task, each synchronization task is written into the one-way linked list of the corresponding task slot; The controlling each execution node to execute the synchronization task according to the corresponding scheduling time round includes: Identify the scheduling period of each scheduling time round, and determine the task slot to be executed corresponding to the current time from each scheduling time round; Obtain a one-way linked list to be executed corresponding to the task slot to be executed, and identify the execution time corresponding to each synchronization task data in the one-way linked list to be executed; According to each execution time and the scheduling period corresponding to the current time, determine the target synchronization task corresponding to each execution node at the current time, and execute the corresponding target synchronization task, wherein the target synchronization task is a synchronization task whose execution time is within the scheduling period corresponding to the current time; The method further includes: recording the process of executing the synchronization task to generate a write-back log, and when the write-back log contains a synchronization task that has not been successfully executed, marking the synchronization task that has not been successfully executed as an exception, and sending the synchronization task with the exception mark to the next round of synchronization task allocation until the synchronization task with the exception mark is successfully executed; The method of determining the synchronization task includes: Obtain multiple synchronization scripts, and identify script features and update dates corresponding to each synchronization script; Determine the synchronization strategy corresponding to each synchronization script according to each script feature and strategy mapping relationship, wherein the strategy mapping relationship is the corresponding relationship between the script feature and the synchronization strategy; According to the update date and synchronization strategy of each synchronization script, determine the synchronization area of each synchronization script in the preset time window; Determine the synchronization script whose synchronization area is the update area as the synchronization script to be updated; Also includes: When receiving a synchronization instruction, identifying a synchronization feature type contained in the synchronization instruction, wherein the synchronization feature type includes second-level scheduling and balanced scheduling; When the synchronization feature type is second-level scheduling, the synchronization task corresponding to the feature synchronization instruction is assigned to the first target execution node, and the first target execution node is controlled to execute the corresponding synchronization task; When the synchronization feature type is balanced scheduling, the execution rate and execution safety level of each execution node are identified, and based on the synchronization task corresponding to the synchronization instruction of interest, the execution rate and execution safety level of each execution node, the second target execution node corresponding to each synchronization task is determined, and each second target execution node is controlled to execute the corresponding synchronization task.
2. A data synchronization implementation device for distributed parallel computing, It is characterized in that include: An identification module, used to identify the synchronization quantity of multiple synchronization tasks and the task characteristics of each synchronization task, each synchronization task corresponds to a synchronization script to be updated; A distribution list determination module is used to determine a task distribution list for each execution node based on the synchronization quantity and each task feature, each task distribution list including a plurality of synchronization tasks; A task time wheel determination module is used to identify the task amount of each synchronization task, and determine the task time wheel information corresponding to each execution node based on the synchronization quantity corresponding to each task allocation list and the task amount of each synchronization task, wherein the task time wheel information includes a time interval and a scale number; The scheduling time wheel determination module is used to obtain the execution time of each synchronization task in each task allocation list, write the task allocation list of each execution node into the corresponding task time wheel according to each execution time and the corresponding task time wheel information, and obtain the scheduling time wheel corresponding to each execution node; The execution task module is used to control each execution node to execute synchronization tasks according to the corresponding scheduling time wheel; Among them, the scheduling time wheel determination module writes the task allocation list of each execution node into the corresponding task time wheel according to each execution moment and the corresponding task time wheel information, and obtains the scheduling time wheel corresponding to each execution node, which is specifically used for: Identify the time interval and number of ticks in each task time wheel information; Determine the task slot of each synchronization task in the corresponding task time wheel based on the time interval, the number of ticks and each execution time corresponding to each task time wheel information; Write each synchronous task pair into the corresponding task slot to obtain the scheduling time wheel corresponding to each execution node; The scheduling time wheel module determines the task slot of each synchronization task in the corresponding task time wheel based on the time interval, the number of ticks and each execution time corresponding to each task time wheel information, and is specifically used to: Identify the start time of each task time wheel, and determine the slot index of each synchronization task according to the time interval, number of ticks, start time, and each execution time corresponding to each task time wheel information and the preset task slot calculation formula; According to the slot index of each synchronization task, the task slot of each synchronization task in the corresponding task time wheel is determined, wherein the preset task slot calculation formula includes: ; Among them, A_hour is used to represent the hour corresponding to the start time, and A_minute is used to represent the minute corresponding to the start time; B_hour is used to represent the hour corresponding to the execution time, and B_ minute is used to represent the minute corresponding to the execution time; x is used to represent the slot index of the task slot corresponding to the execution time; Wherein, when determining that the scheduling time wheel module writes each synchronization task pair into the corresponding task slot, it is specifically used to: Create a one-way linked list corresponding to each task slot; Determine the linked list position of each synchronization task in the corresponding one-way linked list according to the execution time of each synchronization task; According to the linked list position corresponding to each synchronization task, each synchronization task is written into the one-way linked list of the corresponding task slot; Among them, the execution task module is specifically used to control each execution node to execute the synchronization task according to the corresponding scheduling time wheel: Identify the scheduling period of each scheduling time round, and determine the task slot to be executed corresponding to the current time from each scheduling time round; Obtain a one-way linked list to be executed corresponding to the task slot to be executed, and identify the execution time corresponding to each synchronization task data in the one-way linked list to be executed; According to each execution time and the scheduling period corresponding to the current time, determine the target synchronization task corresponding to each execution node at the current time, and execute the corresponding target synchronization task, wherein the target synchronization task is a synchronization task whose execution time is within the scheduling period corresponding to the current time; The device is further used to: record the process of executing the synchronization task to generate a write-back log, and when the write-back log contains a synchronization task that has not been successfully executed, mark the synchronization task that has not been successfully executed as an exception, and send the synchronization task with the exception mark to the next round of synchronization task allocation until the synchronization task with the exception mark is successfully executed; Wherein, when determining the mode of the synchronization task, the script identification module is used to obtain multiple synchronization scripts and identify the script features and update date corresponding to each synchronization script; A synchronization strategy determination module is used to determine the synchronization strategy corresponding to each synchronization script according to each script feature and strategy mapping relationship, wherein the strategy mapping relationship is a corresponding relationship between the script feature and the synchronization strategy; A synchronization area determination module is used to determine the synchronization area of each synchronization script in a preset time window according to the update date and synchronization strategy of each synchronization script; A module for determining synchronization scripts to be updated, used for determining synchronization scripts whose synchronization areas are update areas as synchronization scripts to be updated; The device also includes: A feature type identification module, used for identifying the synchronization feature type contained in the synchronization instruction when receiving the synchronization instruction, wherein the synchronization feature type includes second-level scheduling and balanced scheduling; The first type execution module is used to assign the synchronization task corresponding to the feature synchronization instruction to the first target execution node when the synchronization feature type is second-level scheduling, and control the first target execution node to execute the corresponding synchronization task; The second type execution module is used to identify the execution rate and execution safety level of each execution node when the synchronization feature type is balanced scheduling, and based on the synchronization task corresponding to the synchronization instruction of interest, the execution rate and execution safety level of each execution node, determine the second target execution node corresponding to each synchronization task, and control each second target execution node to execute the corresponding synchronization task.
3. A synchronization system, It is characterized in that The synchronization system includes: at least one processor; Memory; at least one application, wherein the at least one application is stored in the memory and configured to be executed by at least one processor, and the at least one application is configured to: execute a data synchronization implementation method for distributed parallel computing as described in any one of claims 1.
4. A computer-readable storage medium, It is characterized in that include: A computer program is stored which can be loaded by a processor and execute a data synchronization implementation method for distributed parallel computing as claimed in any one of claims 1.
