A task processing method and system based on a distributed cluster datax

Abstract

The application provides a task processing method based on a distributed cluster datax, which comprises the following steps: obtaining a task to be executed; obtaining the task load condition of all current datax nodes, wherein each datax node is configured with a corresponding proxy service module, and the datax node is registered in a nacos registration center through the corresponding proxy service module; and based on a preconfigured allocation strategy, the task is distributed to the datax node. The task processing method based on the distributed cluster datax provided by the application configures the corresponding proxy service module for each datax node, and registers multiple datax nodes in the nacos registration center through the module, so that the distributed cluster of the datax is realized, and automatic registration and service discovery can be realized in the nacos registration center.

Description

Technical Field

[0001] This invention relates to the field of distributed architecture technology, and in particular to a task processing method and system based on the distributed cluster datax. Background Technology

[0002] DataX is an open-source heterogeneous data source offline synchronization tool from Alibaba, dedicated to achieving stable and efficient data synchronization between various heterogeneous data sources, including relational databases (MySQL, Oracle, etc.), HDFS, Hive, ODPS, HBase, FTP, and more. However, the current open-source version of DataX only supports single-machine mode, not distributed mode, and cannot achieve load balancing between multiple nodes, nor can it perform automatic node registration and service discovery. Summary of the Invention

[0003] The technical problem to be solved by this invention is that the current datax tool does not support distributed architecture, and a task processing method and system based on the distributed cluster datax is provided.

[0004] The technical solution adopted in this invention is a task processing method based on a distributed cluster datax, comprising: obtaining the task to be executed; obtaining the task load status of all datax nodes, wherein each datax node is configured with a corresponding proxy service module, and the datax node registers with the Nacos registry center through the corresponding proxy service module; and distributing the task to the datax node based on a pre-configured allocation strategy.

[0005] In one implementation, obtaining the task load information of all current datax nodes includes: the Redis module obtaining and storing the task load information of all current datax nodes in key-value format, wherein the task load information includes the number of allocated synchronization tasks.

[0006] In one implementation, the step of distributing the task to the datax node based on a pre-configured allocation strategy includes: distributing the task to one of the datax nodes with the lowest task load based on a pre-configured priority order of the task.

[0007] In one embodiment, the task processing method based on the distributed cluster datax further includes: dynamically adjusting the task memory of the datax node, including: configuring the task memory of the datax node using the metadata of the Nacos registry center; or configuring the task memory of the task before sending the task to the datax node.

[0008] Another aspect of the present invention provides a task processing system based on a distributed cluster datax, comprising: an acquisition module for acquiring tasks to be executed; a Redis management module for acquiring the task load of all current datax nodes and distributing the tasks to the datax nodes based on a pre-configured allocation strategy; and a Nacos registry module, wherein each datax node is configured with a corresponding proxy service module, and the datax node registers with the Nacos registry through the corresponding proxy service module.

[0009] In one implementation, the Redis module is further configured to: obtain and store the current task load information of all datax in key-value format, wherein the task load information includes the number of allocated synchronization tasks.

[0010] In one implementation, the Redis module is further configured to: distribute the task to one of the datax nodes with the lowest task load, based on the pre-configured priority order of the task.

[0011] In one embodiment, the task processing system based on the distributed cluster datax further includes a memory adjustment module, configured to: configure the task memory of the datax node using the metadata of the Nacos registry; or configure the task memory of the task before sending the task to the datax node.

[0012] Another aspect of the present invention provides an electronic device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program, when executed by the processor, implements the steps of the task processing method based on the distributed cluster datax as described in any of the preceding claims.

[0013] Another aspect of the present invention provides a computer storage medium storing a computer program, which, when executed by a processor, implements the steps of the task processing method based on the distributed cluster datax as described in any of the preceding claims.

[0014] By adopting the above technical solution, the present invention has at least the following advantages:

[0015] The task processing method based on the distributed cluster datax described in this invention configures a corresponding proxy service module for each datax node, and through this module, enables multiple datax nodes to register in the Nacos registry center, thereby realizing a distributed datax cluster and enabling automatic registration and service discovery in the Nacos registry center. Attached Figure Description

[0016] Figure 1 This is a flowchart of a task processing method based on a distributed cluster datax according to an embodiment of the present invention;

[0017] Figure 2 This is a schematic diagram of the composition structure of a task processing system based on the distributed cluster datax according to an embodiment of the present invention;

[0018] Figure 3 This is a schematic diagram of the architecture of a task processing system based on a distributed cluster datax according to an embodiment of the present invention;

[0019] Figure 4 This is a schematic diagram of an electronic device structure according to an embodiment of the present invention. Detailed Implementation

[0020] To further illustrate the technical means and effects of the present invention in achieving its intended purpose, the present invention will be described in detail below with reference to the accompanying drawings and preferred embodiments.

[0021] The steps described in the specification and the flowcharts in the accompanying drawings of this invention are not necessarily to be strictly followed according to the step numbers; the execution order of the steps can be changed. Furthermore, certain steps can be omitted, multiple steps can be combined into one step, and / or one step can be broken down into multiple steps.

[0022] The first embodiment of the present invention provides a task processing method based on a distributed cluster datax, such as... Figure 1 As shown, the specific steps include the following:

[0023] Step S1: Obtain the task to be executed.

[0024] Step S2: Obtain the task load status of all current datax nodes. Each datax node is configured with a corresponding proxy service module, and the datax node registers with the Nacos registry center through the corresponding proxy service module.

[0025] Step S3: Based on the pre-configured allocation strategy, the task is distributed to the datax node.

[0026] The following will describe in detail, step by step, a task processing method based on the distributed cluster datax provided in this embodiment.

[0027] Step S1: Obtain the task to be executed.

[0028] In this embodiment, tasks can be obtained through the Redis (Remote Dictionary Server) management module, or through other interfaces. The tasks can be multiple synchronous tasks.

[0029] Step S2: Obtain the task load status of all current datax nodes. Each datax node is configured with a corresponding proxy service module, and the datax node registers with the Nacos registry center through the corresponding proxy service module.

[0030] In this example, refer to Figure 2 You can set up at least two (usually more) datax nodes. Since the open-source version of datax only supports single-machine mode, in order to form a distributed architecture, each datax node is configured with a corresponding proxy service module. Figure 2 The agent (in this context) refers to each agent service module corresponding to a datax node. The agent service acts as a proxy for the datax, providing external interfaces to perform related operations on the datax, including starting and stopping the datax process. In addition, the datax's running logs are also collected by the agent and sent to distributed storage for unified management.

[0031] In this embodiment, the program packages in datax are mounted through the container's directory and stored in a specified location within the agent microservice's container. Therefore, the registration and discovery of datax nodes are achieved through the automatic registration and discovery of the agent microservice. Using Nacos as the registry center, when a new agent service module is registered to the Nacos registry center, the datax node corresponding to the newly added agent service module becomes available. External entities can operate the corresponding datax program through the external interface provided by the agent service module.

[0032] In this embodiment, the current distributed architecture can also be expanded or reduced in size according to the needs of actual applications.

[0033] Specifically, when expansion is needed, the agent service module corresponding to the new datax node needs to be started, and the new agent service needs to be registered with the Nacos registry. At this time, the new datax node is added to the datax cluster and can be assigned corresponding tasks to run.

[0034] When scaling down is needed, simply take the proxy service corresponding to the datax node offline. At this point, the datax node is removed from the management of the datax cluster and will no longer accept task assignments.

[0035] Step S3: Based on the pre-configured allocation strategy, the task is distributed to the datax node.

[0036] In this embodiment, load control of the datax nodes is determined based on the current task load of each datax node to decide which datax node a new synchronization task will be assigned to. Since the task distribution module adopts a microservice architecture, load control requires a unified manager. For example, a Redis distributed lock can be used to perform unified node load calculation and node load updates.

[0037] Furthermore, the load status of each datax node is also stored in Redis in key-value pairs. When a datax node is assigned a new synchronization task, its load is incremented by 1; when the task finishes execution, its load is decremented by 1. In other words, the task load of each datax node is determined by the number of synchronization tasks currently being processed by that datax node.

[0038] Furthermore, when allocating tasks in the Redis module, a task priority strategy is designed, where tasks with higher priority are allocated more runtime resources. Specifically, tasks can be divided into multiple priority levels according to actual needs, and then, in descending order of priority, tasks are sequentially distributed to one of the datax nodes with the lightest current task load, i.e., the fewest synchronization tasks.

[0039] In some possible implementations of this embodiment, in order to improve the performance of task synchronization, in addition to using task sharding of the datax node itself, the datax node sharding method can also be used to fully utilize the advantages of the datax distributed cluster, splitting the task and distributing it to multiple datax nodes for parallel execution. That is, the same task can be sharded and processed into multiple parts, which are then distributed to different datax nodes for processing.

[0040] Specifically, the source data can be split based on the splitPK field specified in the task script. This field can be numeric or time-based, and the value of this field can be used to split the source data according to the number of nodes with available resources. Ultimately, the task will be split into several smaller tasks for execution.

[0041] In this embodiment, since different types of synchronization tasks have different memory requirements during task execution, dynamic adjustment of the datax node's task execution memory is also very important. To achieve precise control of the datax node's memory, the following two methods can be used to control the datax node's execution memory:

[0042] 1) Global adjustment;

[0043] This method is based on Nacos metadata. By changing the metadata configuration in Nacos, the running memory of DataX is set. This configuration is passed to the startup script of DataX node through the agent service. The advantage of this method is that after adjusting the configuration, there is no need to restart the agent service. It can take effect in real time and is applied to all DataX nodes.

[0044] 2) Task level control;

[0045] When server resources are scarce, you may not want to implement global memory control, but only want to adjust the memory of a specific synchronization task. In this case, you can use this adjustment method. This method allows users to set the memory parameters for the data synchronization task during configuration. These parameters are passed to the agent as the configuration when starting DataX. The advantage of this approach is that it only takes effect on the task at hand and does not affect the memory settings of other tasks.

[0046] Compared with the prior art, this embodiment has at least the following advantages;

[0047] 1) By designing an agent proxy service, a proxy for datax is implemented, and a distributed cluster of datax is implemented through the cluster deployment of the agent service.

[0048] 2) Automatic node registration and service discovery: The datax package is used as part of the agent service. By leveraging the agent service's registration feature with the Nacos registry, automatic registration of the agent service and datax nodes and service discovery can be achieved.

[0049] 3) Dynamic scaling of nodes: By combining the service registration and service shutdown functions of the agent microservice, the datax cluster can be expanded and shrunk, thereby realizing the dynamic scaling of datax nodes.

[0050] 4) Redis load control and task allocation: Utilize the features of Redis distributed locks to build a unified load management system, use Redis to store the load status of each node, and use task priority strategies to control the priority of task execution.

[0051] 5) Dynamic adjustment of node memory: Dynamic adjustment of datax runtime memory is achieved through global adjustment and task-level control.

[0052] 6) Task sharding: Leveraging the features of the datax cluster, the splitPK field is used to shard the source data. Based on the sharding results, large tasks are split into multiple smaller tasks and distributed to the datax cluster for execution, thereby improving task parallelism.

[0053] The second embodiment of the present invention corresponds to the first embodiment. This embodiment introduces a task processing system based on the distributed cluster datax, such as... Figure 3 As shown, it includes the following components:

[0054] The retrieval module is configured to retrieve the tasks that need to be executed.

[0055] The Redis management module obtains the current task load of all datax nodes and distributes tasks to the datax nodes based on a pre-configured allocation strategy.

[0056] The Nacos registry module is configured with a corresponding proxy service module for each datax node. The datax node registers with the Nacos registry through the corresponding proxy service module.

[0057] In this embodiment, the Redis module is further configured to: obtain and store the current task load information of all datax in key-value format, wherein the task load information includes the number of assigned synchronization tasks.

[0058] In this embodiment, the Redis module is further configured to: distribute the task to one of the datax nodes with the lowest task load, based on the pre-configured priority order of the task.

[0059] In this embodiment, the task processing system based on the distributed cluster datax further includes a memory adjustment module, which is configured to: configure the task memory of the datax node using the metadata of the Nacos registry center; or configure the task memory of the task before sending the task to the datax node.

[0060] A third embodiment of the present invention provides an electronic device, such as... Figure 4As shown, it can be understood as a physical device, including a processor and a memory storing processor-executable instructions, which, when executed by the processor, perform the following operations:

[0061] Step S1: Obtain the task to be executed.

[0062] Step S2: Obtain the task load status of all current datax nodes. Each datax node is configured with a corresponding proxy service module, and the datax node registers with the Nacos registry center through the corresponding proxy service module.

[0063] Step S3: Based on the pre-configured allocation strategy, the task is distributed to the datax node.

[0064] The fourth embodiment of the present invention provides a task processing method based on a distributed cluster (datax) with the same flow as the first, second, or third embodiments. The difference lies in the engineering implementation: this embodiment can be implemented using software plus necessary general-purpose hardware platforms. While hardware implementation is also possible, the former is often a preferred method. Based on this understanding, the method of the present invention can be embodied in the form of a computer software product stored in a storage medium (such as ROM / RAM, magnetic disk, or optical disk), including several instructions to cause a device (such as a base station or other network device) to execute the method described in the embodiments of the present invention.

[0065] In summary, compared with the prior art, the advantages of the present invention include at least the following:

[0066] 1) By designing an agent proxy service, a proxy for datax is implemented, and a distributed cluster of datax is implemented through the cluster deployment of the agent service.

[0067] 2) Automatic node registration and service discovery: The datax package is used as part of the agent service. By leveraging the agent service's registration feature with the Nacos registry, automatic registration of the agent service and datax nodes and service discovery can be achieved.

[0068] 3) Dynamic scaling of nodes: By combining the service registration and service shutdown functions of the agent microservice, the datax cluster can be expanded and shrunk, thereby realizing the dynamic scaling of datax nodes.

[0069] 4) Redis load control and task allocation: Utilize the features of Redis distributed locks to build a unified load management system, use Redis to store the load status of each node, and use task priority strategies to control the priority of task execution.

[0070] 5) Dynamic adjustment of node memory: Dynamic adjustment of datax runtime memory is achieved through global adjustment and task-level control.

[0071] 6) Task sharding: Leveraging the features of the datax cluster, the splitPK field is used to shard the source data. Based on the sharding results, large tasks are split into multiple smaller tasks and distributed to the datax cluster for execution, thereby improving task parallelism.

[0072] Through the description of specific embodiments, a more in-depth and specific understanding should be gained of the technical means and effects adopted by the present invention to achieve the intended purpose. However, the accompanying drawings are only provided for reference and illustration and are not intended to limit the present invention.

Claims

1. A task processing method based on the distributed cluster datax, characterized in that, include: Obtain the tasks to be performed; The task load status of all current datax nodes is obtained. Each datax node is configured with a corresponding proxy service module, and the proxy service module corresponds one-to-one with the corresponding datax node. The proxy service module provides an external interface to execute the start and stop operations of the corresponding datax process, and collects the running logs of the corresponding datax node and sends them to distributed storage. The datax node is registered in the Nacos registry through the corresponding proxy service module, and the datax package is stored in the specified location of the container of the proxy service module through the directory of the container. The task load information is stored in the Redis module in key-value format. When a new synchronization task is assigned to a datax node, the load value of the node is incremented by 1. When the task finishes execution, the load value of the node is decremented by 1. The task load information is determined by the number of synchronization tasks that the datax node is currently processing. Based on a pre-configured allocation strategy, the task is distributed to the datax node. If the task is configured with a splitPK field, the source data is split according to the numeric or time type value of the splitPK field to obtain multiple subtasks, and each subtask is distributed to different datax nodes for parallel execution.

2. The task processing method based on the distributed cluster datax according to claim 1, characterized in that, The process of distributing the task to the datax node based on a pre-configured allocation strategy includes: Based on the pre-configured priority order of the tasks, the tasks are distributed to at least one of the datax nodes with the lowest task load.

3. The task processing method based on the distributed cluster datax according to claim 1, characterized in that, The method further includes: dynamically adjusting the task memory of the datax node, including: Configure the task memory of the datax node using the metadata of the Nacos registry center; or Before sending the task to the datax node, configure the task memory.

4. A task processing system based on the distributed cluster datax, characterized in that, include: Get the module to retrieve the tasks to be executed; The Redis management module obtains the current task load of all datax nodes and distributes the tasks to the datax nodes based on the pre-configured allocation strategy. The Nacos registry module includes a corresponding proxy service module for each datax node. Each proxy service module corresponds one-to-one with a datax node. The proxy service module provides an external interface to start and stop the corresponding datax process and collects the running logs of the corresponding datax node, sending them to distributed storage. The datax node registers with the Nacos registry through its corresponding proxy service module, and the datax package is mounted and stored in a designated location within the container of the proxy service module. The task load information is stored in the Redis module in key-value format. When a new synchronization task is assigned to a datax node, the load value of the node is incremented by 1. When the task finishes execution, the load value of the node is decremented by 1. The task load information is determined by the number of synchronization tasks that the datax node is currently processing. If the task is configured with a splitPK field, the source data is split according to the numeric or time type value of the splitPK field to obtain multiple subtasks, and each subtask is sent to different datax nodes for parallel execution.

5. The task processing system based on the distributed cluster datax according to claim 4, characterized in that, The Redis module is further configured as follows: Based on the pre-configured priority order of the tasks, the tasks are distributed to at least one of the datax nodes with the lowest task load.

6. The task processing system based on the distributed cluster datax according to claim 4, characterized in that, The system also includes a memory adjustment module, configured to: Configure the task memory of the datax node using the metadata of the Nacos registry center; or Before sending the task to the datax node, configure the task memory.

7. An electronic device, characterized in that, include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program, when executed by the processor, implements the steps of the method as described in any one of claims 1 to 3.

8. A computer storage medium storing a computer program, wherein the computer program, when executed by a processor, implements the steps of the task processing method based on the distributed cluster datax as described in any one of claims 1 to 3.

Images