Multi-engine based data integration platform, data synchronization method, device and medium

Abstract

The present disclosure relates to a multi-engine-based data integration platform, a data synchronization method, equipment and a medium. The platform comprises: a task configuration module configured to receive configuration information of a data integration task and obtain a target synchronization engine selected by a user; a task script construction module configured to generate a target task script corresponding to the target synchronization engine according to the configuration information and the target synchronization engine; a task scheduling module configured to determine an available service node according to a current node load list and distribute the data integration task to the available service node; and a task running module configured to provide a service node, each service node corresponding to a plurality of synchronization engines; when the available service node receives the data integration task, the target synchronization engine in the available service node is started according to a target start command corresponding to the target synchronization engine and the target task script to execute the data integration task. The data integration platform designed in the present scheme supports task configuration and data synchronization capabilities based on a plurality of synchronization engines.

Description

Technical Field

[0001] This disclosure relates to the field of big data processing technology, and in particular to a multi-engine-based data integration platform, data synchronization method, device and medium. Background Technology

[0002] With the continuous development of information technology and the increasing emphasis on the value of data, more and more enterprises and institutions have realized that the "data silos" phenomenon, caused by decentralized business processes, lack of interoperability, and lack of data sharing, poses significant challenges to data analysis, utilization, and mining. The first step in solving the "data silos" problem is to aggregate the data from various separate business systems or departments, establishing a unified data collection, processing, and sharing center. One of the data processing methods used in this process is data integration technology.

[0003] Data integration is the logical or physical aggregation of data from different sources, formats, and characteristics, thereby providing comprehensive data sharing for enterprises and institutions. The core of data integration technology is the data synchronization engine, a tool or framework used for data migration between various heterogeneous data sources. However, different data synchronization engines have different implementations, significant differences in characteristics, and different applicable scopes. Some are only used for specific scenarios, limiting the data integration capabilities of platforms that only support a single data synchronization engine. Summary of the Invention

[0004] To solve the above-mentioned technical problems, or at least partially solve them, at least one embodiment of this disclosure provides a multi-engine-based data integration platform, data synchronization method, device, and medium.

[0005] Firstly, this disclosure provides a multi-engine-based data integration platform, including: a task configuration module, a task script construction module, a task scheduling module, and a task execution module; wherein,

[0006] The task configuration module is used to receive configuration information for the data integration task and to obtain the target synchronization engine selected by the user.

[0007] The task script construction module is used to generate a target task script corresponding to the target synchronization engine based on the configuration information and the target synchronization engine.

[0008] The task scheduling module is used to determine available service nodes based on the current node load list and to send the data integration task to the available service nodes.

[0009] The task execution module is used to provide service nodes, each of which corresponds to multiple synchronization engines. The program packages corresponding to the multiple synchronization engines are stored in a designated directory in the container of the service node using containerized deployment. When the available service node receives the data integration task, it starts the target synchronization engine in the available service node according to the target start command and the target task script, so as to start the data synchronization process corresponding to the target synchronization engine in the container to execute the data integration task.

[0010] Secondly, this disclosure provides a data synchronization method applied to a multi-engine-based data integration platform as described in the first aspect, the method comprising:

[0011] The task configuration module receives configuration information for the data integration task and obtains the target synchronization engine selected by the user.

[0012] Based on the configuration information and the target synchronization engine, the target task script corresponding to the target synchronization engine is generated by the task script construction module.

[0013] The task scheduling module determines available service nodes based on the current node load list and distributes the data integration task to the available service nodes.

[0014] Through the task execution module, the available service node starts the target synchronization engine in the available service node according to the target startup command and the target task script corresponding to the target synchronization engine, so as to start the data synchronization process corresponding to the target synchronization engine in the container of the available service node to execute the data integration task. The task execution module provides service nodes, each service node corresponds to multiple synchronization engines, and the program packages corresponding to the multiple synchronization engines are stored in a designated directory in the container of the service node using containerized deployment.

[0015] Thirdly, this disclosure provides an electronic device, including: a processor and a memory; the processor executes a data synchronization method as described in any of the second aspects by invoking a program or instructions stored in the memory.

[0016] Fourthly, this disclosure provides a computer-readable storage medium storing a program or instructions that cause a computer to perform any of the data synchronization methods provided in the second aspect.

[0017] Fifthly, this disclosure provides a computer program product, including a computer program, wherein the computer program, when executed by a processor, implements any of the data synchronization methods provided in the second aspect.

[0018] The technical solution provided in this disclosure has at least the following advantages compared with the prior art:

[0019] In this embodiment, the task configuration module receives configuration information for the data integration task and obtains the target synchronization engine selected by the user. Then, the task script construction module generates a target task script corresponding to the target synchronization engine based on the configuration information and the target synchronization engine. The task scheduling module determines available service nodes based on the current node load list and distributes the data integration task to the available service nodes. The task execution module provides services to the service nodes, with each service node corresponding to multiple synchronization engines. A containerized deployment is used to store the program packages corresponding to each synchronization engine in a designated directory within the container of the service node. When an available service node receives the data integration task, it starts the target synchronization engine in the available service node according to the target startup command and target task script corresponding to the target synchronization engine, thereby starting the data synchronization process corresponding to the target synchronization engine in the container to execute the data integration task. This solution's data integration platform supports task configuration and data synchronization capabilities based on multiple synchronization engines. Users can select a suitable synchronization engine to execute the data synchronization task, enriching the application scenarios and improving the applicability of the data integration platform, demonstrating strong data integration capabilities. Attached Figure Description

[0020] The accompanying drawings, which are incorporated in and form a part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure.

[0021] To more clearly illustrate the technical solutions in the embodiments of this disclosure or the prior art, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0022] Figure 1 This is a schematic diagram of the structure of a multi-engine-based data integration platform provided in an embodiment of the present disclosure;

[0023] Figure 2 A schematic diagram illustrating the workflow of a task scheduling module according to an exemplary embodiment of this disclosure is shown.

[0024] Figure 3 A schematic diagram illustrating the data integration task allocation result of an exemplary embodiment of this disclosure is shown;

[0025] Figure 4 A schematic diagram illustrating node load updates performed by the task scheduling module in an exemplary embodiment of this disclosure is shown.

[0026] Figure 5 A schematic flowchart illustrating a data synchronization method provided in an embodiment of this disclosure;

[0027] Figure 6 This is a schematic flowchart of a data synchronization method provided in another embodiment of the present disclosure. Detailed Implementation

[0028] To better understand the above-described objectives, features, and advantages of this disclosure, the present disclosure will be further described in detail below with reference to the accompanying drawings and embodiments. It is understood that the described embodiments are only some, not all, of the embodiments of this disclosure. The specific embodiments described herein are merely for explaining this disclosure and not for limiting it. Unless otherwise specified, the embodiments of this disclosure and the features within them can be combined with each other. All other embodiments obtained by those skilled in the art based on the described embodiments of this disclosure are within the scope of protection of this disclosure.

[0029] Numerous specific details are set forth in the following description in order to provide a full understanding of this disclosure, but this disclosure may also be implemented in other ways different from those described herein; obviously, the embodiments in the specification are only some, and not all, of the embodiments of this disclosure.

[0030] Existing data integration platforms typically only support a single data synchronization engine, which has the following problems: (1) Existing data integration platforms often only support a single engine at the bottom layer, and users cannot choose freely, resulting in low flexibility; (2) The application scope of a single engine is limited, and it is only used in specific scenarios, which affects the data integration capabilities of the platform; (3) Under the single engine architecture, it is difficult to expand to a new engine, and the cost of modifying the existing architecture is high; (4) There is no engine support at the task granularity, and it is impossible to use different synchronization engines for different types of integration tasks.

[0031] To address the aforementioned issues, this disclosure provides a multi-engine-based data integration platform. It abandons the design of supporting only a single synchronization engine, offering multi-engine support for both the data integration platform and data synchronization. This supports multi-engine-based data integration task configuration and data synchronization capabilities. During the data integration task configuration phase, users are provided with the option to select multiple synchronization engines. Through the task configuration module, users can choose the appropriate data synchronization engine for different data integration tasks. The selected synchronization engine is associated with the task level, allowing different types of integration tasks to use different synchronization engines. This fully leverages the functional advantages of multiple engines, broadens the application scenarios and applicability, and significantly improves the data integration capabilities of the data integration platform. This solution adopts microservices and containerized deployment, making node expansion simple. Service nodes can be easily built without additional configuration, and the same architecture is compatible with multiple synchronization engines simultaneously. The transformation cost is low, making it suitable for widespread adoption.

[0032] Figure 1 This is a schematic diagram of the structure of a multi-engine-based data integration platform provided in an embodiment of this disclosure, as shown below. Figure 1 As shown, the multi-engine-based data integration platform 10 includes: a task configuration module 110, a task script construction module 120, a task scheduling module 130, and a task execution module 140.

[0033] In this embodiment of the disclosure, the task configuration module 110 provides configuration functions for data integration tasks, which are used to receive configuration information of data integration tasks and obtain the target synchronization engine selected by the user.

[0034] The configuration information for the data integration task may include, but is not limited to, source-side basic information, target-side basic information, field mapping information, and scheduling strategy information. Source-side basic information may include, but is not limited to, task name, task directory, task description, source data source, and source table; target-side basic information may include, but is not limited to, target data source, data integration method, and primary key conflict policy; field mapping information may include, but is not limited to, the field mapping relationship between the source table and the target table; scheduling strategy information may include, but is not limited to, scheduling type, scheduling date, scheduling start time, synchronization strategy, whether rate limiting is enabled, whether to retry on failure, the limit for triggering automatic task termination due to dirty data, and timeout.

[0035] In this embodiment of the disclosure, in order to achieve multi-engine adaptation at the task level, in addition to configuring the relevant basic information as described above, the solution of this disclosure also requires the selection of engine type when configuring the integrated task.

[0036] For example, DataX and FlinkX, two currently mainstream synchronization engines, can be chosen as the underlying synchronization engines for the data integration platform. The difference between DataX and FlinkX is:

[0037] (1) DataX is a single-machine multi-threaded application that does not depend on other services and components and does not support distributed operation; FlinkX depends on the Flink environment and requires submitting Flink jobs to the Flink cluster, and supports distributed operation.

[0038] (2) DataX only supports offline batch processing; FlinkX supports both batch processing and streaming processing.

[0039] (3) DataX supports Groovy functions, which can implement user-defined transformation functions; FlinkX only supports user-defined functions (UDFs).

[0040] (4) DataX is relatively stable; FlinkX appeared later and is currently in a rapid iteration process, so its stability is poor.

[0041] (5) After testing, for small data integration tasks, there is little difference between DataX and FlinkX, but for large data integration tasks, FlinkX has a clear performance advantage.

[0042] In this embodiment of the disclosure, the user can select a suitable synchronization engine as the target synchronization engine according to the use scenario of the data integration task. After the task configuration module associates the data integration task with the corresponding target synchronization engine, the configuration of the task configuration module is completed.

[0043] Furthermore, after configuring the relevant data integration basic information through the task configuration module 110, this basic information needs to be constructed into a task script that the target synchronization engine can run. The task script generation can be implemented in the task script construction module 120. Specifically, the task script construction module 120 generates the target task script corresponding to the target synchronization engine based on the configuration information configured in the task configuration module and the target synchronization engine.

[0044] It is understandable that for the same task, the task scripts required when executed by different synchronization engines are different; and for the same synchronization engine, the task scripts corresponding to different tasks are also different.

[0045] Taking the data integration platform supporting both DataX and FlinkX synchronization engines as an example, both engines support JSON format task scripts. Therefore, the task script construction module can generate the corresponding JSON format task script based on the target synchronization engine and configuration information configured in the task configuration module. Since the required JSON format task scripts for DataX and FlinkX differ, the script can be constructed according to the selected engine type (i.e., DataX or FlinkX). Assuming the data synchronization task is a mysql2mysql task, the script examples for DataX and FlinkX are shown below:

[0046] (1) DataX script example:

[0047]

[0048]

[0049]

[0050]

[0051] (2) FlinkX script example:

[0052]

[0053]

[0054]

[0055]

[0056]

[0057]

[0058] Therefore, in this embodiment of the disclosure, corresponding script templates can be pre-set for different tasks and synchronization engines. Then, based on the target synchronization engine selected by the user in the task configuration module 110 and the input configuration information, a suitable target script template is determined. Based on the target script template, the configuration information of the data integration task is constructed into the corresponding target task script.

[0059] In this embodiment, the task execution module 140 provides service nodes, which can be one or more. Each service node corresponds to multiple synchronization engines, and containerized deployment is used to store the program packages corresponding to the multiple synchronization engines in a designated directory within the container of the service node. By adopting containerized deployment, the task execution module 140 can accommodate multiple synchronization engines, enabling multiple synchronization engines to be associated under one service node, which helps improve the data integration capabilities of the data integration platform.

[0060] For example, suppose the data integration platform supports two synchronization engines: DataX and FlinkX. Both engines are started in local mode, meaning one service node corresponds to one DataX engine and one FlinkX engine. The service nodes can be deployed using Docker, with the DataX and FlinkX engine packages mounted as directories in a specified directory within the service node's Docker container.

[0061] For each service node provided by the task execution module 140, the task scheduling module 130 provides resource management and scheduling functions for the service nodes, including node load calculation, task distribution to be run, node load update, and node load correction.

[0062] In this embodiment, the task scheduling module 130 can determine the available service nodes in the service nodes according to the current node load list, and distribute the data integration task to the available service nodes. When the available service node receives the data integration task, it starts the target synchronization engine in the available service node according to the target startup command and target task script corresponding to the target synchronization engine, so as to start the data synchronization process corresponding to the target synchronization engine in the container of the available service node to execute the data integration task.

[0063] For example, assuming the target synchronization engine is the DataX engine, the corresponding target startup command is:

[0064] The filename is located at `python3 / home / datax / bin / datax.py / home / datax / pzjson / json`. This filename is the name of the target task script.

[0065] For example, assuming the target synchronization engine is the FlinkX engine, the corresponding target startup command would be:

[0066] The command is: `sh / home / chunjun / bin / chunjun-local.sh-job / home / chunjun / pzjson / json`. The `json` string represents the filename of the target task script.

[0067] In this embodiment of the disclosure, when the target synchronization engine is started using the target startup command corresponding to the target synchronization engine, the data synchronization process corresponding to the target synchronization engine will be started in the container corresponding to the available service node, and then the received data integration task will be executed through the started process.

[0068] This disclosure's multi-engine-based data integration platform receives configuration information for data integration tasks and obtains the target synchronization engine selected by the user through a task configuration module. Then, a task script construction module generates a target task script corresponding to the target synchronization engine based on the configuration information and the target synchronization engine. A task scheduling module determines available service nodes based on the current node load list and distributes the data integration task to these available service nodes. A task execution module serves the service nodes, with each service node corresponding to multiple synchronization engines. A containerized deployment is used to store the program packages corresponding to each synchronization engine in a designated directory within the service node's container. When an available service node receives a data integration task, it starts the target synchronization engine in the available service node according to the target startup command and target task script corresponding to the target synchronization engine, thereby starting the data synchronization process corresponding to the target synchronization engine in the container to execute the data integration task. This solution's data integration platform supports task configuration and data synchronization capabilities based on multiple synchronization engines. Users can select a suitable synchronization engine to execute the data synchronization task, enriching the platform's application scenarios and improving its applicability, demonstrating strong data integration capabilities.

[0069] In one optional embodiment of this disclosure, the task scheduling module 130 can obtain the current node load list according to a preset first query period. The node load list records the current number of running tasks and the maximum number of runnable tasks corresponding to each service node. Based on the current number of running tasks and the maximum number of runnable tasks corresponding to each service node, the idle number of runnable tasks corresponding to each service node is determined. Service nodes with an idle number of runnable tasks greater than 0 are determined as available service nodes, and the node address of the available service nodes is obtained. Based on the node address, the proxy service interface deployed on the corresponding available service node is called to send data integration tasks to the available service nodes according to the idle number of runnable tasks.

[0070] In this embodiment, the task scheduling module 130 maintains a node load list, which records the number of tasks currently running on each service node (i.e., the current number of running tasks) and the maximum number of runnable tasks for that service node. For example, Table 1 shows an example of a node load list according to an exemplary embodiment of this disclosure. As shown in Table 1, the current number of running tasks for service node ip1:port is 3, the current number of running tasks for service node ip2:port is 4, the current number of running tasks for service node ip3:port is 2, and the maximum number of runnable tasks for each service node is 10.

[0071] Table 1

[0072]

[0073] The task scheduling module 130 can adopt a timed dispatch strategy, requesting the latest node load list according to the first query cycle (e.g., 3 seconds). After obtaining the latest node load list, it calculates the number of idle runnable tasks corresponding to each service node based on the current running task count and maximum runnable task count of each service node recorded in the node load list. If the number of idle runnable tasks corresponding to a service node is 0, it indicates that the service node has no idle resources. If the number of idle runnable tasks is greater than 0, it indicates that the service node has idle resources. The service node can be identified as an available service node, and the node address of the available service node can be obtained. The node address can be obtained from the node load list, such as the ip1:port recorded in Table 1, which is the node address.

[0074] In this embodiment, if the task scheduling module 130 determines that all service nodes have no idle resources, it will not perform the task assignment operation for the time being. Instead, it will wait until the task scheduling module obtains the latest node load list at the next scheduled time and then recalculate the node load. If there are available service nodes with idle resources, the task assignment operation will be performed, and available service nodes will be allocated to the data integration task to be run. After the available service nodes are allocated, the data integration task will be assigned to the corresponding available service nodes for execution.

[0075] For example, assuming that after node load calculation, it is found that the remaining idle resources of all available service nodes can still dispatch 10 tasks to be run, that is, the total number of idle runnable tasks corresponding to available service nodes is 10, then the task scheduling module 130 can obtain 10 tasks to be run from the data integration tasks to be run, and then randomly assign available service nodes to these 10 tasks. If there are fewer than 10 tasks to be run, then available service nodes will be randomly assigned to them according to the actual number of tasks to be run. In addition, when assigning available service nodes to data integration tasks, the number of idle runnable tasks corresponding to each available service node can also be considered during the allocation process. For example, if there are 3 data integration tasks, 4 idle and runnable tasks corresponding to service node 1, 3 idle and runnable tasks corresponding to service node 2, and 2 idle and runnable tasks corresponding to service node 3, then when allocating data integration tasks, these 3 tasks can be assigned to either service node 1 or service node 2, or one data integration task can be assigned to each of these three service nodes, or two data integration tasks can be assigned to service node 3, and the remaining data integration task can be assigned to either service node 1 or service node 2. This disclosure does not restrict the task allocation method.

[0076] In this embodiment, when the task scheduling module 130 sends data integration tasks to available service nodes, it calls the corresponding proxy service interface deployed on the available service node based on the node address of the available service node. This achieves the purpose of sending data integration tasks to the available service nodes. The node address can be in the form of "IP + port number," for example, "IP1:port" in Table 1 is the node address of a service node. The proxy service interface is the interface provided by the micro-proxy service in the service node. Calling the proxy service interface through the node address to send the data integration task to the corresponding available service node ensures accurate delivery of the data integration task to the appropriate available service node.

[0077] For example, if there is an available idle task resource on service node ip1:port, after task scheduling module 130 assigns the data integration task A to be run to service node ip1:port, task scheduling module 130 will accurately distribute the data integration task A to be run to service node ip1:port by calling the interface provided by the micro-agent service on service node ip1:port.

[0078] Figure 2 A schematic diagram of the workflow of the task scheduling module 130 of an exemplary embodiment of this disclosure is shown, as follows: Figure 2As shown, the task execution module 140 provides three service nodes: Service Node 1, Service Node 2, and Service Node 3. Each service node supports both the DataX engine and the FlinkX engine for synchronization. The task scheduling module 130 manages the load status of these three service nodes and queries the node load list according to a preset first query cycle, determining the available service nodes based on the node load list. Assuming the currently queried node load list is as shown in Table 1, it can be determined from Table 1 that all three service nodes have idle resources. Assuming there are 8 data integration tasks to be run, these 8 tasks can be randomly assigned to these three service nodes, as shown in the allocation result. Figure 3 As shown, data integration tasks 1, 2, and 3 are assigned to service node 1; data integration tasks 4, 5, and 6 are assigned to service node 2; and data integration tasks 7 and 8 are assigned to service node 3. Furthermore, according to... Figure 3 As shown in the allocation results, the task scheduling module 130 can distribute each data integration task to the corresponding service node based on the node addresses corresponding to these three service nodes.

[0079] Furthermore, in one optional embodiment of this disclosure, the task scheduling module 130 also supports updating the node load list. The task scheduling module 130 can obtain the actual number of data integration tasks actually issued to the first available service node, wherein the first available service node is any service node among the available service nodes; in response to receiving the task start running status information fed back by the first available service node, the number of currently running tasks corresponding to the first available service node in the node load list is increased by the actual number of tasks.

[0080] In this embodiment, after the task scheduling module 130 successfully distributes the data integration tasks to be run to the allocated available service nodes, the task scheduling module 130 needs to update the node load list, updating the current number of running tasks for the corresponding service nodes in the node load list. Specifically, when distributing data integration tasks to each available service node, the task scheduling module 130 can record the number of data integration tasks allocated to each available service node as the actual number of data integration tasks distributed. For example, if there are 8 data integration tasks to be run and service node 1 is allocated 3, then the actual number of tasks distributed to service node 1 is 3. After each available service node receives the data integration tasks distributed by the task scheduling module 130, it can provide feedback on the task start running status information to the task scheduling module when starting to run each data integration task. After receiving the task start running status information from the first available service node, the task scheduling module 130 increases the current number of running tasks of the first available service node in the node load list by its actual number of tasks. For example, assuming that the number of currently running tasks of the first available service node originally recorded in the node load list is 3, and the actual number of tasks sent to the first available service node is 2, then after the task scheduling module 130 receives the task start running status information fed back by the first available service node, it updates the number of currently running tasks of the first available service node in the node load list to 5 (i.e., 3+2).

[0081] In this embodiment of the disclosure, when the task scheduling module 130 receives the status information of the task execution completion fed back by the second service node, in response to the status information, the task scheduling module 130 decrements the number of currently running tasks corresponding to the second service node in the node load list by 1, wherein the second service node is any service node among the service nodes.

[0082] Typically, a data synchronization process started in a service node executes a data synchronization task. After the data synchronization task in a data synchronization process is completed, the service node sends a task completion status message to the task scheduling module 130. After receiving the task completion status message, the task scheduling module 130 decrements the number of currently running tasks of the service node in the node load list by 1.

[0083] Figure 4 A schematic diagram illustrating node load updates performed by the task scheduling module in an exemplary embodiment of this disclosure is shown, such as... Figure 4As shown, for service node ip1:port, in the initial state, the number of currently running tasks for this service node is 2. After the task scheduling module 130 successfully sends a data integration task to this service node, for example, when the task scheduling module 130 receives the status information that the task has started running from the service node, the number of currently running tasks for this service node is incremented by 1, that is, the number of currently running tasks in the node load list is updated to 3. When the service node finishes executing the sent task, for example, when the task scheduling module 130 receives the status information that the task has finished executing from the service node, the number of currently running tasks for this service node is decremented by 1, that is, the number of currently running tasks in the node load list is updated to 2.

[0084] Due to network jitter or other anomalies, after a data integration task is successfully sent, the task scheduling module 130 may not receive the task completion status information. In this case, the task may actually have finished executing, but because the task scheduling module 130 has not received the task completion notification, the load corresponding to that service node cannot be updated in a timely manner. If this happens frequently, it will cause the node load calculation of the task scheduling module 130 to be incorrect, thereby blocking the sending of subsequent tasks. To address this situation, the task scheduling module 130 in this disclosure also supports node load correction. It determines whether the task has finished executing by querying whether the data synchronization for the executing task still exists, and updates the number of currently running tasks on the service node when it is determined that the task has finished executing. Therefore, in one optional embodiment of this disclosure, a data synchronization process is designed to execute a data integration task. The task scheduling module 130 is further configured to: query the number of data synchronization processes currently existing on each service node according to a preset second query period; compare the number of data synchronization processes currently existing on each service node with the number of currently running tasks corresponding to each service node in the node load list; and, in response to the inconsistency between the number of data synchronization processes currently existing on the third service node and the number of currently running tasks on the third service node, update the number of currently running tasks on the third service node in the node load list to the number of data synchronization processes.

[0085] In this embodiment, after the data integration task to be run is successfully distributed to an available service node and begins execution, the task scheduling module 130 will start interval polling. According to a preset second query period (e.g., 10 minutes), it will query whether the task's running process (i.e., the data synchronization process executing the task) still exists. If the task's running process still exists, it means the task is still running. When the query finds that the task's running process does not exist, it means the task has finished executing. Therefore, for each service node, the number of currently existing data synchronization processes can be counted, and this number can be compared with the number of currently running tasks for the same service node in the node load list. If they are the same, it means no task has finished executing. If the number of data synchronization processes is less than the number of currently running tasks, it means a task has finished executing, and the number of currently running tasks for that service node in the node load list is updated to the number of data synchronization processes. For example, for service node 1, the count of data synchronization processes on this service node is 3, while the number of currently running tasks recorded in the node load list for this service node is 5. Therefore, the number of currently running tasks for this service node in the node load list is updated to 3.

[0086] In one optional embodiment of this disclosure, the task execution module 140 uses a micro-agent service to expand the service node. In this embodiment, a micro-agent service is started and registered in the registry center to obtain a new service node.

[0087] Nacos can be used as the registry center to configure and manage micro-agent services.

[0088] In this embodiment, the micro-proxy service serves as a proxy service for the underlying synchronization engine, providing corresponding task submission interfaces. Designed as a microservice, the micro-proxy service is horizontally scalable. When a service node needs to be expanded, only one micro-proxy service needs to be started. Once the micro-proxy service is registered with the registry center, a new service node is obtained. The task scheduling module 130, through periodic polling, detects the addition of a new service node and updates the node load list, adding the new service node to the list. This completes the expansion of the service node. Thus, one micro-proxy service is deployed on each service node, forming a one-to-one relationship between the service node and the micro-proxy service. Data integration tasks are distributed to the corresponding service nodes by the task scheduling module 130, and the distributed data integration tasks are received through the interfaces provided by the micro-proxy service. Multiple service nodes with deployed micro-proxy services form a service node cluster, providing task submission capabilities to the outside world.

[0089] Furthermore, after the task scheduling module 130 detects the addition of a new service node, it can obtain the physical memory allocated to the new service node, as well as the memory occupied by the operating system and micro-agent services on the new service node. The physical memory allocated to the newly added service node can be preset according to actual needs; for example, it can be preset to 32GB for each service node. The occupied memory can also be set according to actual needs. Typically, the sum of the memory occupied by the operating system and the memory occupied by the micro-agent services deployed on a service node is approximately 6GB. Therefore, the memory occupied by the operating system and micro-agent services on each service node can be preset to 6GB. Next, the task scheduling module 130 determines the remaining memory of the new service node based on the physical memory and the occupied memory, and determines the maximum number of runnable tasks corresponding to the new service node based on the remaining memory and the preset data synchronization task allocation memory. Based on the new service node and its corresponding maximum number of runnable tasks, the node load list is updated.

[0090] The memory allocation for data synchronization tasks can be set according to actual needs. For example, the memory allocation for data synchronization tasks can be preset to 2G.

[0091] For example, assuming a new service node is allocated 32GB of physical memory and occupies 6GB of memory, then the remaining memory of the new service node is 26GB. Assuming the data synchronization task is allocated 2GB of memory, it can be determined that the maximum number of running tasks of this service node is 13. Therefore, the load information of this service node can be added to the node load list, showing that its maximum number of running tasks is 13. Since this service node is newly expanded and has not yet been assigned data integration tasks, its current number of running tasks is 0.

[0092] Furthermore, when a service node needs to be reduced, the task execution module 140 only needs to take the micro-agent service corresponding to the service node offline in the registry center. The task scheduling module 130 will detect that a service node is offline through periodic polling, and will update the node load list, deleting the load information of the offline service node from the node load list. This completes the reduction of service nodes.

[0093] In this embodiment of the disclosure, the task execution module 140 uses a micro-agent service to achieve automatic expansion of service nodes. Node expansion is simple and service node clusters can be easily built without additional configuration.

[0094] To implement the above embodiments, this disclosure also provides a data synchronization method, which is applied to the multi-engine-based data integration platform described in the above embodiments.

[0095] Figure 5This is a flowchart illustrating a data synchronization method provided in an embodiment of the present disclosure. The data processing method can be configured to be executed in the multi-engine-based data integration platform of the present disclosure, or it can be executed by an electronic device, which can be any device with data synchronization capabilities.

[0096] like Figure 5 As shown, the data synchronization method provided in this disclosure embodiment may include the following steps:

[0097] Step 201: Receive the configuration information of the data integration task through the task configuration module, and obtain the target synchronization engine selected by the user.

[0098] Step 202: Based on the configuration information and the target synchronization engine, generate the target task script corresponding to the target synchronization engine through the task script construction module.

[0099] Step 203: The task scheduling module determines the available service nodes based on the current node load list and sends the data integration task to the available service nodes.

[0100] Step 204: Through the task execution module, the available service node starts the target synchronization engine in the available service node according to the target startup command and the target task script corresponding to the target synchronization engine, so as to start the data synchronization process corresponding to the target synchronization engine in the container of the available service node to execute the data integration task. The task execution module provides service nodes, each service node corresponds to multiple synchronization engines, and the program packages corresponding to the multiple synchronization engines are stored in a specified directory in the container of the service node using containerized deployment.

[0101] It should be noted that, in the embodiments of this disclosure, the description of each step executed by each module can be found in the explanation of the relevant modules in the foregoing embodiments, and their implementation principles are similar, so they will not be repeated here.

[0102] The data synchronization method of this disclosure allows users to select a suitable data synchronization engine for data integration work according to different data integration task requirements through the task configuration module. The selected synchronization engine is associated with the task level granularity. Different types of integration tasks can use different synchronization engines to run data synchronization tasks, thereby realizing that different tasks can use different synchronization engines to perform data synchronization work. This fully leverages the functional advantages of multiple engines, has a wider range of application scenarios and applicability, and significantly improves the data integration capabilities of the data integration platform.

[0103] In one alternative embodiment of this disclosure, such as Figure 6 As shown, in Figure 5Based on the illustrated embodiment, step 202 may include the following sub-steps:

[0104] Step 301: Determine the task type corresponding to the data integration task based on the configuration information.

[0105] In this embodiment of the disclosure, the task type corresponding to the data integration task can be determined based on the configuration information received by the task scheduling module.

[0106] For example, if the source data source is MySQL and the target data source is also MySQL in the configuration information, then the task type can be determined as a mysql2mysql task.

[0107] Step 302: Using the task script construction module, a target script template corresponding to the task type and the target synchronization engine is determined from a set of preset script templates based on the task type and the target synchronization engine.

[0108] In this embodiment, the task script construction module stores multiple preset script templates. Different task types correspond to different script templates, and different synchronization engines also correspond to different script templates. Therefore, the stored script templates can be associated with synchronization engines and task types. For example, a template set can be set for each synchronization engine, storing script templates for different task types corresponding to that synchronization engine. Thus, after determining the task type, the target script template corresponding to the task type and the target synchronization engine selected by the user can be found.

[0109] Step 303: Generate the target task script based on the configuration information and the target script template.

[0110] In this embodiment of the disclosure, after determining the target script template, the target task script can be generated based on the configuration information and the target script template. For example, relevant information from the configuration information can be filled into the relevant paragraphs in the target script template to obtain the target task script. To distinguish the target task scripts for different data integration tasks, the task identifier of the data integration task can be associated with the corresponding target task script.

[0111] The data synchronization method of this disclosure determines the task type corresponding to the data integration task based on configuration information. Through a task script construction module, a target script template corresponding to the task type and target synchronization engine is determined from a plurality of preset script templates based on the task type and target synchronization engine. Then, a target task script is generated based on the configuration information and the target script template. Thus, a target task script matching the target synchronization engine and data integration task selected by the user can be obtained, ensuring successful startup of the target synchronization engine and thus ensuring the effective execution of the data synchronization task.

[0112] This disclosure also provides an electronic device, including a processor and a memory; the processor executes the steps of the data synchronization method embodiments described above by calling programs or instructions stored in the memory. To avoid repetition, these steps will not be repeated here.

[0113] This disclosure also provides a non-transitory computer-readable storage medium that stores a program or instructions that cause a computer to perform the steps of the data synchronization method described in the foregoing embodiments. To avoid repetition, these steps will not be repeated here.

[0114] This disclosure also provides a computer program product, including a computer program, wherein, when executed by a processor, the computer program implements the steps of the data synchronization method embodiments described in the foregoing embodiments.

[0115] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0116] The above description is merely a specific embodiment of this disclosure, enabling those skilled in the art to understand or implement it. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this disclosure. Therefore, this disclosure is not to be limited to the embodiments described herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A data integration platform based on multiple engines, characterized in that, include: The task configuration module, task script construction module, task scheduling module, and task execution module are included; among them, The task configuration module is used to receive configuration information for the data integration task and to obtain the target synchronization engine selected by the user. The task script construction module is used to generate a target task script corresponding to the target synchronization engine based on the configuration information and the target synchronization engine. The task scheduling module is used to determine available service nodes based on the current node load list and to send the data integration task to the available service nodes. The task execution module is used to provide service nodes, each of which corresponds to multiple synchronization engines. A containerized deployment is used to store the program packages corresponding to each synchronization engine in a designated directory within the container of the service node. When an available service node receives the data integration task, it starts the target synchronization engine in the available service node according to the target startup command and the target task script, thereby starting the data synchronization process corresponding to the target synchronization engine in the container to execute the data integration task. The task scheduling module is further configured to: According to the preset first query period, obtain the current node load list, and the node load list records the current number of running tasks and the maximum number of running tasks corresponding to each service node; The number of idle runnable tasks corresponding to each service node is determined based on the number of currently running tasks and the maximum number of runnable tasks corresponding to each service node; Service nodes with a greater than 0 number of idle runnable tasks are identified as available service nodes, and the node addresses of the available service nodes are obtained. Based on the node address, the proxy service interface deployed on the corresponding available service node is invoked to issue the data integration task to the available service node according to the number of idle runnable tasks; The task scheduling module is also used for: Obtain the actual number of data integration tasks actually issued to the first available service node, wherein the first available service node is any one of the available service nodes; In response to receiving the task start running status information from the first available service node, the number of currently running tasks corresponding to the first available service node in the node load list is increased by the actual number of tasks. In response to receiving the task execution completion status information from the second service node, the number of currently running tasks corresponding to the second service node in the node load list is decremented by 1, wherein the second service node is any one of the service nodes; A data synchronization process executes a data integration task; The task scheduling module is also used for: According to the preset second query cycle, query the number of data synchronization processes currently existing on each service node; Compare the number of data synchronization processes currently existing on each service node with the number of currently running tasks corresponding to each service node in the node load list; If the number of data synchronization processes currently existing on the third service node is inconsistent with the number of currently running tasks on the third service node, then the number of currently running tasks on the third service node in the node load list is updated to the number of data synchronization processes.

2. The data integration platform based on a multi-engine according to claim 1, characterized in that, The task execution module uses a micro-agent service to expand service nodes. Specifically, a micro-agent service is started and registered in the registry center to obtain a new service node.

3. The multi-engine-based data integration platform according to claim 2, characterized in that, The task scheduling module is also used for: Obtain the physical memory allocated to the new service node, as well as the memory occupied by the operating system and micro-agent service on the new service node; Based on the physical memory and the occupied memory, determine the remaining memory of the new service node; Based on the remaining memory and the memory allocated by the preset data synchronization tasks, the maximum number of runnable tasks corresponding to the new service node is determined. Update the node load list based on the new service nodes and their corresponding maximum number of runnable tasks.

4. A data synchronization method, characterized in that, Applied to the multi-engine-based data integration platform as described in any one of claims 1-3, the method comprises: The task configuration module receives configuration information for the data integration task and obtains the target synchronization engine selected by the user. Based on the configuration information and the target synchronization engine, the target task script corresponding to the target synchronization engine is generated by the task script construction module. The task scheduling module determines available service nodes based on the current node load list and sends the data integration task to the available service nodes. Through the task execution module, the available service node starts the target synchronization engine in the available service node according to the target startup command and the target task script corresponding to the target synchronization engine, so as to start the data synchronization process corresponding to the target synchronization engine in the container of the available service node to execute the data integration task. The task execution module provides service nodes, each service node corresponds to multiple synchronization engines, and the program packages corresponding to the multiple synchronization engines are stored in a designated directory in the container of the service node using containerized deployment.

5. The data synchronization method according to claim 4, characterized in that, The step of generating a target task script corresponding to the target synchronization engine through the task script construction module based on the configuration information and the target synchronization engine includes: Based on the configuration information, determine the task type corresponding to the data integration task; The task script construction module determines the target script template corresponding to the task type and the target synchronization engine from a set of preset script templates, based on the task type and the target synchronization engine. The target task script is generated based on the configuration information and the target script template.

6. An electronic device, characterized in that, include: Processor and memory; The processor executes the data synchronization method as described in any one of claims 4 or 5 by invoking the program or instructions stored in the memory.

7. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a program or instructions that cause the computer to perform the data synchronization method as described in any one of claims 4 or 5.

Images