Data transfer task migration method, device, equipment and storage medium

By detecting task status and verifying consistency, and automatically migrating data transmission tasks, the problem of configuration conflicts between data transmission tasks in Taichung is solved, and efficient and stable data migration and governance is achieved.

CN118677950BActive Publication Date: 2024-11-29ZHEJIANG DAHUA TECH CO LTD
View PDF 2 Cites 0 Cited by

Patent Information

Application Number
CN202411157218.8
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-08-22
Publication Date
2024-11-29
Estimated Expiration
2044-08-22

AI Technical Summary

Technical Problem

In the data transmission scenario where the data middle platform handles complex services, slight configuration changes may affect the accuracy of the data. The existing methods require users to understand business logic and manual operations, which can easily lead to data loss and increase risks.

Method used

By detecting the task status and verifying the task consistency of the target node, determining the migration strategy based on the task type, automatically migrating the task to be migrated to the target node, and performing task update processing to ensure the continuous operation and accuracy of the data transmission task.

Benefits of technology

It realizes automated data transmission task migration, reduces manual operation risks, simplifies workflows, and improves task migration efficiency and data transmission stability and reliability.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118677950B_ABST
    Figure CN118677950B_ABST
Patent Text Reader

Abstract

The present application discloses a migration method, device, equipment and storage medium for a data transmission task, wherein the migration method for the data transmission task comprises: in response to a received task migration request, performing a status detection on a task to be migrated corresponding to the task migration request, wherein the task migration request is used to migrate the task to be migrated from the current node to the target node; if the task to be migrated is in a running state, then verifying whether the target task in the target node is consistent with the task to be migrated; in response to the inconsistency between the target task and the task to be migrated, determining a target migration strategy from a preset task migration strategy based on the task type of the task to be migrated; migrating the task to be migrated to the target node based on the target migration strategy, and performing task update processing on the target task. The above scheme can improve the efficiency of task migration.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a method, device, equipment and storage medium for migrating data transmission tasks. Background Art

[0002] As a platform for data governance and circulation, the data middle platform can achieve unified management, efficient utilization and value mining of data through technical means such as data governance, data integration, and data services.

[0003] However, the data center also faces many challenges when dealing with complex business data transmission scenarios. Especially when updating data transmission tasks, even minor configuration changes may affect the accuracy of the final data.

[0004] At present, the solution mainly requires users to first understand the complex business logic in different scenarios and the functions of each data transmission task, and then perform manual migration. However, errors are prone to occur during manual operation, resulting in data loss, further increasing the risk of data transmission. Summary of the invention

[0005] The present application at least provides a method, apparatus, device and computer-readable storage medium for migrating a data transmission task.

[0006] The first aspect of the present application provides a migration method for data transmission tasks, including: in response to a received task migration request, performing a status detection on the task to be migrated corresponding to the task migration request, the task migration request being used to migrate the task to be migrated from the current node to the target node; if the task to be migrated is in a running state, verifying whether the target task in the target node is consistent with the task to be migrated; in response to the inconsistency between the target task and the task to be migrated, determining a target migration strategy from a preset task migration strategy based on the task type of the task to be migrated; migrating the task to be migrated to the target node based on the target migration strategy, and performing task update processing on the target task.

[0007] In one embodiment, the task migration strategy includes a full migration strategy and a local migration strategy, and the target migration strategy is determined from the preset task migration strategy based on the task type of the task to be migrated, including: if the task type of the task to be migrated is a full transfer type, then the full migration strategy is determined as the target migration strategy; if the task type of the task to be migrated is a local transfer type, then the local migration strategy is determined as the target migration strategy.

[0008] In one embodiment, if the task type of the task to be migrated is a full transfer type, then after the full migration strategy is determined as the target migration strategy, the method includes: stopping the task to be migrated and the target task to obtain a stopped task to be migrated and a stopped target task; migrating the stopped task to be migrated to the target node, and performing a full update process on the stopped target task based on the stopped task to be migrated.

[0009] In one embodiment, if the task type of the task to be migrated is a local transmission type, then after the local migration strategy is determined as the target migration strategy, the method includes: suspending the task to be migrated and the target task to obtain a suspended task to be migrated and a suspended target task; determining the local task to be migrated based on the data transmission progress of the suspended task to be migrated; migrating the local task to be migrated to the target node, and locally updating the suspended target task based on the local task to be migrated.

[0010] In one embodiment, if the task to be migrated is in a running state, then verifying whether the target task in the target node is consistent with the task to be migrated, including: performing consistency verification processing based on the task information of the task to be migrated and the task information of the target task; in response to the task information of the task to be migrated matching the task information of the target task, determining that the task to be migrated is consistent with the target task.

[0011] In one embodiment, the task information includes basic task information, task interaction information, and mapping field information, and the consistency check processing is performed based on the task information to be migrated of the task to be migrated and the candidate task information of each candidate task in the target node, including: in response to the consistency of the task basic information between the task to be migrated and the candidate task, the consistency of the task interaction information, and the consistency of the mapping field information, determining that the task information of the task to be migrated and the task information of the target task match; in response to the inconsistency of at least one of the task basic information, the task interaction information, and the mapping field information between the task to be migrated and the candidate task, determining that the task information of the task to be migrated and the task information of the target task do not match.

[0012] In one embodiment, after performing a status check on the task to be migrated corresponding to the task migration request in response to the received task migration request, the method further includes: in response to the task to be migrated being in a paused state, recording the current task progress of the task to be migrated; determining the local task to be migrated in the task to be migrated based on the current task progress of the task to be migrated; migrating the local task to be migrated to the target node, and performing task update processing on the target task.

[0013] The second aspect of the present application provides a migration device for data transmission tasks, including: a status detection module, which is used to respond to a received task migration request and perform status detection on the task to be migrated corresponding to the task migration request, and the task migration request is used to migrate the task to be migrated from the current node to the target node; a consistency verification module, which is used to verify whether the target task in the target node is consistent with the task to be migrated if the task to be migrated is in a running state; a migration strategy determination module, which is used to determine the target migration strategy from the preset task migration strategy based on the task type of the task to be migrated in response to the inconsistency between the target task and the task to be migrated; a task migration module, which is used to migrate the task to be migrated to the target node based on the target migration strategy, and perform task update processing on the target task.

[0014] A third aspect of the present application provides an electronic device, including a memory and a processor, wherein the processor is used to execute program instructions stored in the memory to implement the above-mentioned data transmission task migration method.

[0015] A fourth aspect of the present application provides a computer-readable storage medium having program instructions stored thereon, which implement the above-mentioned data transmission task migration method when the program instructions are executed by a processor.

[0016] The above scheme, in response to the received task migration request, determines the current task status of the task to be migrated by performing status detection on the task to be migrated corresponding to the task migration request, and the task migration request is used to migrate the task to be migrated from the current node to the target node; if the task to be migrated is in the running state, it is necessary to verify whether the target task in the target node is consistent with the task to be migrated, thereby determining whether the task to be migrated needs to be migrated to the target node; in response to the inconsistency between the target task and the task to be migrated, it indicates that the task to be migrated needs to be migrated to the target node, so based on the task type of the task to be migrated, a target migration strategy suitable for migrating the task to be migrated is determined from the preset task migration strategy; based on the target migration strategy, the task to be migrated is migrated to the target node, and the target task is updated so that the target node runs the updated task, thereby realizing task migration. The migration method of the data transmission task of the present application determines the difference points of the data transmission tasks in different nodes through consistency verification to ensure the continuous operation of the data transmission task before and after the migration, and based on the automated operation, eliminates the task migration risk caused by manual operation, simplifies the workflow and improves the efficiency of task migration.

[0017] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The drawings herein are incorporated into the specification and constitute a part of the specification. These drawings illustrate embodiments consistent with the present application and are used together with the specification to illustrate the technical solution of the present application.

[0019] Figure 1 is a flowchart of an exemplary embodiment of a method for migrating a data transmission task of the present application;

[0020] Figure 2 This is an exemplary overall flow chart of the migration method of the data transmission task of the present application;

[0021] Figure 3 is a block diagram of a migration device for a data transmission task shown in an exemplary embodiment of the present application;

[0022] Figure 4 It is a structural schematic diagram of an embodiment of the electronic device of the present application;

[0023] Figure 5 It is a structural diagram of an embodiment of a computer-readable storage medium of the present application. DETAILED DESCRIPTION

[0024] The scheme of the embodiment of the present application is described in detail below in conjunction with the drawings of the specification.

[0025] In the following description, for the purpose of explanation rather than limitation, specific details such as specific system structures, interfaces, and technologies are provided to facilitate a thorough understanding of the present application.

[0026] The term "and / or" in this article is only a description of the association relationship of associated objects, indicating that there may be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the objects associated before and after are in an "or" relationship. In addition, "many" in this article means two or more than two. In addition, the term "at least one" in this article means any combination of at least two of any one or more of a plurality of, for example, including at least one of A, B, and C, can mean including any one or more elements selected from the set consisting of A, B, and C.

[0027] For ease of understanding, one of the applicable scenarios of the present application is now exemplified. In complex data transmission business scenarios (such as transferring a specified software product package to a target platform for deployment and installation, etc.), the deployment and installation process may face a variety of difficulties, including data transmission task migration failures caused by data conflicts. Data conflict refers to the fact that during the data transmission task migration process, data transmission tasks with the same name may have type anomalies at the source and destination ends of the data transmission channel, data may conflict and cannot be transmitted normally, resulting in data transmission anomalies. When there is a data conflict problem, it will cause task migration failure and ultimately make the entire business unable to operate normally.

[0028] Existing solutions mainly require users to understand business logic and manually resolve conflicts after understanding the causes of data conflicts. Manual conflict resolution requires manual modification of task configuration. However, for real-time data transmission tasks, manually stopping tasks may cause data loss, reduce data transmission accuracy, increase data transmission risks, and increase user operation burdens and operation thresholds, while reducing the stability and reliability of data governance and circulation in the data center.

[0029] Therefore, the data transmission task migration method of the present application can solve the problem of detecting and resolving conflicts in the data transmission task configuration during the process of migrating the data transmission task, reduce the user's operational burden, and provide stability and reliability for data governance and circulation in the data middle station.

[0030] See also Figure 1 , Figure 1 This is a flowchart of an exemplary embodiment of the method for migrating a data transmission task of the present application. Specifically, it may include the following steps:

[0031] Step S110 , in response to the received task migration request, a status detection is performed on the task to be migrated corresponding to the task migration request, where the task migration request is used to migrate the task to be migrated from the current node to the target node.

[0032] Among them, the task migration request is used to migrate the task to be migrated from the current node to the target node. The task to be migrated in this application refers to a data transmission task. The current node is the node where the task to be migrated is currently running, and the target node is the node where the task to be migrated is running after the migration is completed. Exemplarily, the migration method of the present application can be applied to application scenarios such as server clusters, specifically to the target node itself, or to the console node in the server cluster (which can manage, control, monitor, and allocate tasks for each node in the server cluster), and is not limited here.

[0033] After receiving the task migration request, the status of the task to be migrated specified by the task migration request is detected. It should be noted that the status of the task to be migrated can at least include a running state, a paused state, and a stopped state. If the task to be migrated is in a running state, it indicates that the source end and the destination end in the task to be migrated are performing data transmission. Among them, the source end refers to the source end of the data transmission task, and the destination end refers to the destination end of the data transmission task. The data transmission task is to transmit data from the source end to the destination end. Therefore, the source end and the destination end are defined based on the data transmission task, while the current node and the target node are defined based on the task migration, and are different from each other.

[0034] Step S120: If the task to be migrated is in the running state, verify whether the target task in the target node is consistent with the task to be migrated.

[0035] Combined with the above steps, if the task to be migrated in the current node is in a running state, it indicates that the task is responsible for data transmission between the source and destination ends. During task migration, there may be a certain task (target task) running in the target node, or it may be in an idle state (if there is no running task, the task to be migrated can be directly migrated to the target node for running). Therefore, in order to avoid task conflicts or other task migration problems, the preset consistency verification algorithm module can be called in the present application to perform consistency verification on the target task in the target node and the task to be migrated in the current node to determine whether the tasks in the two nodes are consistent. Among them, the consistency verification algorithm module is preset with the consistency verification algorithm of the present application, which is used to perform consistency verification during the task migration process.

[0036] If the task to be migrated in the current node is exactly the same as the target task in the target node, it means that the target task running in the target node and the task to be migrated running in the current node have the same task purpose and task effect. Therefore, you can skip this task migration process and do not migrate the task to be migrated running in the current node. You only need to execute the target task running in the target node.

[0037] Step S130 : in response to the inconsistency between the target task and the task to be migrated, determining a target migration strategy from preset task migration strategies based on the task type of the task to be migrated.

[0038] In conjunction with the above steps, if the task to be migrated in the current node is inconsistent with the target task in the target node, that is, the target task running in the target node and the task to be migrated running in the current node have different task objectives and task effects. Therefore, it is necessary to migrate the task to be migrated running in the current node to the target node, so that the task to be migrated can continue to run after migrating to the target node, ensuring the stability of task operation, and preventing the data transmission task to be migrated from being unable to run normally due to task migration failure, causing data loss and other problems.

[0039] Specifically, the task migration method of the present application may be preset with multiple task migration strategies (such as full migration and partial migration), which are respectively used to adapt to the migration of different types of tasks to be migrated. Therefore, before migrating the tasks to be migrated, it is necessary to determine the target migration strategy from the preset task migration strategies according to the task type of the tasks to be migrated.

[0040] Step S140 , migrating the task to be migrated to the target node based on the target migration strategy, and performing task update processing on the target task.

[0041] In conjunction with the above steps, after determining the target migration strategy for migrating the tasks to be migrated, the tasks to be migrated can be migrated to the target node based on the target migration strategy, and task update processing can be performed on the target tasks in the target node based on the tasks to be migrated.

[0042] For ease of explanation, the tasks migrated to the target node will be referred to as migrated tasks in the following text. The specific update process can be to wait for the target task in the target node to be completed, update the task that the target node currently needs to execute to the migrated task and execute the task; or to pause or stop the target task currently running in the target node, update the task that the target node currently needs to execute to the migrated task and execute the task, and after the migrated task is completed, resume or restart the previously paused or stopped target task; or to update the target task based on the migrated task to obtain an updated task, so that the target node executes the updated task. The specific update method can be selected according to the actual application scenario of the task running process, and is not limited here.

[0043] It can be seen that during the operation, the task migration method of the present application responds to the received task migration request, and determines the current task status of the task to be migrated by performing status detection on the task to be migrated corresponding to the task migration request. The task migration request is used to migrate the task to be migrated from the current node to the target node; if the task to be migrated is in the running state, it is necessary to verify whether the target task in the target node is consistent with the task to be migrated, thereby determining whether the task to be migrated needs to be migrated to the target node; in response to the inconsistency between the target task and the task to be migrated, it indicates that the task to be migrated needs to be migrated to the target node, so based on the task type of the task to be migrated, a target migration strategy suitable for migrating the task to be migrated is determined from the preset task migration strategy; based on the target migration strategy, the task to be migrated is migrated to the target node, and the target task is updated so that the target node runs the updated task, thereby realizing task migration. The migration method of the data transmission task of the present application determines the difference points of the data transmission tasks in different nodes through consistency verification to ensure the continuous operation of the data transmission task before and after the migration, and based on the automated operation, eliminates the task migration risk caused by manual operation, simplifies the workflow and improves the efficiency of task migration.

[0044] Based on the above embodiments, the present embodiment of the present application describes the steps of determining the target migration strategy from the preset task migration strategy based on the task type of the task to be migrated. Among them, the task migration strategy includes a full migration strategy and a partial migration strategy. Specifically, the method of this embodiment includes the following steps:

[0045] If the task type of the task to be migrated is a full transfer type, the full migration strategy is determined as the target migration strategy; if the task type of the task to be migrated is a partial transfer type, the partial migration strategy is determined as the target migration strategy.

[0046] Full transfer means that the data to be transferred must be transferred to the destination at one time in the data transfer task. The data transfer task or the data to be transferred cannot be split and then transferred to ensure the security and integrity of the data. Full transfer is often used for data backup, data migration and other aspects.

[0047] The opposite of full transmission is partial transmission or incremental transmission, which means that in the data transmission task, the data to be transmitted specified in the source end can be split into multiple batches and transmitted to the destination end separately. Or it can be understood that each time data is transmitted, the data to be transmitted that is different or changed from the previously transmitted data is transmitted, which can effectively reduce the amount of data transmitted and reduce communication costs. Full transmission and partial transmission are commonly used in the field of data transmission technology. For details, please refer to the explanation of the two transmission methods in the prior art, which will not be elaborated here.

[0048] It should be noted that, during the task migration process, the present application needs to analyze whether the task type of the data transmission task to be migrated is full transmission or partial transmission, and adopt different migration strategies to migrate the migration task.

[0049] Exemplarily, the type and configuration of the data to be transmitted at the source end of the task to be migrated can be known through the relevant information of the task to be migrated. When the data source type of the data transmission task is mysql, oracle, mpp, sqlserver, etc. and there is no split key in the configuration information of the data to be transmitted, the data transmission task to be migrated is determined to be a full transmission type, the full migration strategy is determined as the target migration strategy, and the data transmission task is migrated. Among them, the split key is a configuration method for improving the data synchronization rate. It mainly uses a column in the data table as a split key (usually it is recommended to use the primary key or an indexed column as a split key). In this way, when reading data for transmission, data slicing can be performed according to the configured split key field to achieve concurrent reading and transmission of data, thereby improving the data synchronization rate.

[0050] If a split key is identified in the data to be transmitted, it indicates that the data to be transmitted can be split, that is, it indicates that the data transmission task to be migrated is of a local transmission type. Similarly, the data transmission task to be migrated is determined to be of a local transmission type, the local migration strategy is determined as the target migration strategy, and the data transmission task is migrated.

[0051] Based on the above embodiment, the embodiment of the present application describes the steps after determining the full migration strategy as the target migration strategy if the task type of the task to be migrated is a full transfer type. Specifically, the method of this embodiment includes the following steps:

[0052] The tasks to be migrated and the target tasks are stopped to obtain the stopped tasks to be migrated and the stopped target tasks; the stopped tasks to be migrated are migrated to the target node, and the stopped target tasks are fully updated based on the stopped tasks to be migrated.

[0053] In conjunction with the foregoing embodiment, if the task to be migrated in the current node and the target task in the target node are in a running state, and the task to be migrated is a full transmission type, the full migration strategy includes: first stopping the task to be migrated and the target task through the data transmission task state change module of the present application. Among them, the data transmission task state change module can convert the states of each task such as stop, pause, start (or restart), resume, run and update. After obtaining the stopped task to be migrated and the stopped target task, you can refer to the update method in the foregoing embodiment, for example, use the stopped task to be migrated to fully update the stopped target task, so as to achieve the migration of the stopped task to be migrated to the target node, and the migrated task in the target node also remains in a stopped state. Then, the migrated task in the target node that is in a stopped state can be started, so that the migrated task can continue to run normally in the target node, and the migrated data transmission task after the target node is run to transmit the data to be transmitted from the source end to the destination end.

[0054] Based on the above embodiment, the embodiment of the present application describes the steps after determining the local migration strategy as the target migration strategy if the task type of the task to be migrated is a local transmission type. Specifically, the method of this embodiment includes the following steps:

[0055] The task to be migrated and the target task are suspended to obtain the suspended task to be migrated and the suspended target task; the local task to be migrated is determined based on the data transmission progress of the suspended task to be migrated; the local task to be migrated is migrated to the target node, and the suspended target task is partially updated based on the local task to be migrated.

[0056] In conjunction with the above-mentioned embodiment, if the task to be migrated in the current node and the target task in the target node are in the running state, and the task to be migrated is of the local transmission type, the local migration strategy includes: first suspending the task to be migrated and the target task through the data transmission task state change module of the present application, and obtaining the suspended task to be migrated and the suspended target task. Record the data transmission progress of the suspended task to be migrated, and determine the local task to be migrated in the task to be migrated based on the data transmission progress. For example, the data transmission task to be migrated needs to transfer the A, B, and C data with the split key in the source end to the destination end, and the data transmission progress of the task to be migrated that is suspended in the current node is that the A data in the source end has been transferred to the destination end, then it can be determined that the local task to be migrated in the data transmission task is to transfer the B data and the C data from the source end to the destination end. Among them, not only the data transmission progress of the task to be migrated after the suspension can be recorded, but also the data transmission progress of the target task after the suspension can be recorded, which is used to resume the operation of the suspended target task.

[0057] It should also be noted that, in the above example, if there is a split key between B and C, the task of transmitting data B and the task of transmitting data C can be divided into two data transmission tasks, that is, the local tasks to be migrated can include the local tasks to be migrated for transmitting data B and the local tasks to be migrated for transmitting data C, thereby realizing the batch migration of data transmission tasks.

[0058] Therefore, the local task to be migrated is migrated to the target node to perform local update processing on the suspended target task. Reference can be made to the update method in the aforementioned embodiment, for example, the target task to be suspended is updated and replaced with the migrated local task, so that the local task in the suspended task to be migrated is migrated to the target node, and the migrated local task in the target node also remains in a suspended state. Then the migrated local task in the suspended state in the target node can be restored, so that the migrated local task can continue to run normally in the target node, and the data transmission local task after migration is run through the target node to transmit the remaining untransmitted data in the data transmission task from the source end to the destination end. In addition, the task update method can also include but is not limited to using the migrated local task as the task that the target node currently needs to execute and executing the task, and after the migration of the local task is completed, the previously suspended target task is restored; or after the migration of the local task is completed, the previously suspended target task is first restored to continue the task execution, and after the target task is completed, the migrated local task is restored to continue the task execution. The specific update method can be selected according to the actual application scenario of the task running process, which is not limited here.

[0059] Based on the above embodiment, the embodiment of the present application describes the steps of verifying whether the target task in the target node is consistent with the task to be migrated if the task to be migrated is in the running state. Specifically, the method of this embodiment includes the following steps:

[0060] A consistency check is performed based on the task information of the task to be migrated and the task information of the target task; in response to the task information of the task to be migrated matching the task information of the target task, it is determined that the task to be migrated and the target task are consistent.

[0061] In conjunction with the above-mentioned embodiment, if the data transmission task to be migrated is in a running state, it is necessary to determine the consistency between the task to be migrated and the target task under the condition that the target task exists in the target node. If the task to be migrated and the target task are completely consistent, the task migration process can be skipped and the task to be migrated in the current node can be paused or stopped. If the task to be migrated and the target task are inconsistent, it is necessary to select an appropriate task migration strategy according to the task type of the task to be migrated to perform task migration processing.

[0062] Specifically, the method for determining the consistency between the task to be migrated and the target task may be to determine whether the task to be migrated and the target task are consistent by determining whether the task information of the task to be migrated matches the task information of the target task. Among them, the task information may include but is not limited to task basic information (including task attribute information), task configuration information (including task configuration files), task interaction information (including source end information and destination end information that interact during task operation), and mapping field information (including field information transmitted from the source end to the destination end during data transmission). Exemplarily, if the various task information of the task to be migrated and the target task are completely matched as in the above example, it can be determined that the task to be migrated and the target task are consistent. If there is any task information mismatch between the task to be migrated and the target task, it can be determined that the task to be migrated and the target task are inconsistent.

[0063] Based on the above embodiment, the embodiment of the present application describes the steps of performing consistency check processing based on the task information to be migrated of the task to be migrated and the candidate task information of each candidate task in the target node. Among them, the task information includes task basic information, task interaction information, and mapping field information. Specifically, the method of this embodiment includes the following steps:

[0064] In response to the consistency of task basic information, task interaction information, and mapping field information between the task to be migrated and the candidate task, it is determined that the task information of the task to be migrated and the task information of the target task match; in response to the inconsistency of at least one of the task basic information, task interaction information, and mapping field information between the task to be migrated and the candidate task, it is determined that the task information of the task to be migrated and the task information of the target task do not match.

[0065] With reference to the above-mentioned embodiments, the basic task information of a data transmission task may include, for example, task name, task description information, stacking strategy, transmission mode, and basic attribute information such as the department and business system to which the task belongs.

[0066] The task interaction information of the data transmission task may include the device information of the source and destination (such as the IP address, type, name of the data table included, etc. of the source and destination), the configuration information of the source and destination (such as the partition key and primary key information in the data table included in the source, the partition field formatting type, partition field type, partition field source field and other information when the destination is of Hive type, the index template configuration, granularity, mapping method and other information when the destination is of Elasticsearch type, and the message dispatching mode, partition information and message field type when the destination is of Kafka type, etc., which are not limited here).

[0067] And the mapping field information may include: transmission constant mapping field information (for example, if the content of the column1 field in the source table source is to be transferred to the column2 field in the destination table sink, the field names and field types of column1 and column2 cannot change before and after the migration of the data transmission task).

[0068] By verifying the basic task information of the task to be migrated and the basic task information of the target task, it can be determined whether there is a difference in the relevant attribute information of the data transmission tasks in different nodes. If there is a difference, it indicates that the task to be migrated and the target task are different, so the task to be migrated needs to be migrated to the target node for operation. For example, if the task name of the task to be migrated is task1, and the task name of the target task is task2, the two tasks are different, so task1 needs to be migrated to the target node. If there is no difference, it indicates that the basic task information of the task to be migrated and the basic task information of the target task are consistent, and it is necessary to further determine whether the other information of the two tasks is consistent.

[0069] By verifying the device information of the source and destination, it can be ensured whether there are differences in the source and destination of the data in the data transmission task, that is, whether there are differences in the source and destination of the data transmission task in the current node and the target node. If the source and destination of the target task are changed compared to the source and destination of the task to be migrated, it indicates that the target task and the task to be migrated are inconsistent. Among them, the method of verifying whether there are changes in the source and destination of the two tasks may include but is not limited to matching the source name of the task to be migrated with the source name of the target task, and matching the destination name of the task to be migrated with the destination name of the target task. Or matching the source IP address of the task to be migrated with the source IP address of the target task, and matching the destination IP address of the task to be migrated with the destination IP address of the target task. It can also be a method of combining the judgment process of the name and the IP address, which is not limited here.

[0070] By verifying the configuration information of the source and destination, you can determine whether there is a difference between the source configuration of the task to be migrated and the source configuration of the target task (for example, verify the partition key and primary key information of the source table), and whether there is a difference between the destination configuration of the task to be migrated and the destination configuration of the target task (there are different verification methods for different types of destinations, please refer to the above example for details). The main purpose is to verify the destination to avoid the situation where the configuration type of the destination connected when the current node runs a data transmission task is inconsistent with the configuration type of the destination connected when the target node runs a data transmission task, which leads to data transmission errors and other problems.

[0071] By checking the transmission constant mapping field information, it is possible to determine whether there are differences in the field names and field types between the source and destination ends during the field transmission process of the data transmission task. For example, determine whether there are differences between the column1 field in the source table source in the task to be migrated and the column1 field in the source table source in the target task, and determine whether there are differences between the column2 field in the destination table sink in the task to be migrated and the column2 field in the destination table sink in the target task. If there are differences in the field names and field types of column1 and column2 of the data transmission tasks in the current node and the target node, it indicates that the data transmission tasks in the current node and the target node are inconsistent.

[0072] In addition, the consistency of data transmission tasks in different nodes can be determined by verifying the task configuration information of data transmission tasks in different nodes (such as the configuration file name and number of configuration files of the data transmission task), which will not be elaborated here.

[0073] In summary, you can refer to Figure 2 , Figure 2 It is an exemplary overall flow chart of the migration method of the data transmission task of the present application. Through the above verification links, the consistency verification module can verify the basic task information, task configuration information, source and destination related information, source and destination configuration information, and field mapping information of the data transmission task to determine the consistency between the task to be migrated in the current node and the target task in the target node. The consistency verification module can only determine consistency through verification links including but not limited to the above examples. Otherwise, if any link fails the verification, the consistency verification module will determine inconsistency.

[0074] Exemplarily, the data structure of some relevant parameters involved in the data transmission task of this application is shown here as an example, which can be understood in conjunction with Table 1. In addition, it can also be understood in conjunction with its parameter meanings and prior art, which will not be elaborated here.

[0075] Table 1 is an explanation of the names of the main parameters involved in the data transmission task of this application.

[0076]

[0077]

[0078]

[0079]

[0080]

[0081] Based on the above embodiment, the embodiment of the present application describes the steps after, in response to a received task migration request, the status of the task to be migrated corresponding to the task migration request is detected. Specifically, the method of this embodiment includes the following steps:

[0082] In response to the task to be migrated being in a suspended state, the current task progress of the task to be migrated is recorded; based on the current task progress of the task to be migrated, a partial task to be migrated in the task to be migrated is determined; the partial task to be migrated is migrated to a target node, and task update processing is performed on the target task.

[0083] The above embodiment mainly describes the steps that need to be executed when the task to be migrated is detected to be in the running state, while the present embodiment mainly describes the steps that need to be executed when the task to be migrated is detected to be in the paused state.

[0084] For example, reference may still be made to Figure 2As shown, in response to the received task migration request, the status of the task to be migrated corresponding to the task migration request is detected. If the task to be migrated is in a suspended state, the current task progress of the task to be migrated is recorded; based on the current task progress of the task to be migrated, the partial task to be migrated in the task to be migrated is determined; and the partial task to be migrated is migrated to the target node for task update processing.

[0085] If the task to be migrated is in a suspended state, it can also be determined whether the task to be migrated is a global transfer type or a local transfer type. If it is a global transfer type, it is not necessary to record the current task progress of the task to be migrated, and the entire task to be migrated can be migrated to the target node for task update processing. If it is a local transfer type, the task update processing of the local task to be migrated can be determined based on the current task progress of the task to be migrated. It should be noted that the migrated task to the target node remains in a suspended state.

[0086] It should also be noted that if after receiving the task migration request, it is detected that the task to be migrated is in a stopped state, the data transmission task state change module can directly perform an update operation to update the task to be migrated to the target node without recording the current task progress.

[0087] On the basis of the above embodiments, the embodiments of the present application also need to explain that, in the scenario of task migration, in order to ensure data security, the present application may also set encryption verification means to ensure that the task migration process between the current node and the target node is safe. Specifically, the method of this embodiment may include the following steps:

[0088] If the task migration method of the present application is applied in the current node, the corresponding keys (which may include but are not limited to symmetric keys and asymmetric keys, which are not limited here) can be set in advance in the current node and the target node. When the task to be migrated is a full transmission type, the current node uses the first key to encrypt the task to be migrated first to obtain the encrypted task to be migrated. The encrypted task to be migrated is transmitted to the target node, and the target node uses the second key to decrypt the task to be migrated first to obtain the decrypted task to be migrated, and performs task update processing based on the decrypted task to be migrated. Among them, since the type of key is not limited in this embodiment, the first key and the second key can be the same or different, which is not repeated here. When the task to be migrated is a local transmission type, the current node uses the first key to encrypt the local task to be migrated first to obtain the encrypted local task to be migrated. The encrypted local task to be migrated is transmitted to the target node, and the target node uses the second key to decrypt the local task to be migrated first to obtain the decrypted local task to be migrated, and performs task update processing based on the decrypted local task to be migrated. Among them, since the local tasks to be migrated can be divided into one or more based on the splitting key, if there are multiple local tasks to be migrated, the current node can encrypt each local task to be migrated, or encrypt some of the local tasks to be migrated among the multiple local tasks to be migrated (for example, encrypting the first local task to be migrated and the last local task to be migrated); and, multiple keys can be preset in the current node and the target node, and different keys are used when encrypting and decrypting different local tasks to be migrated.

[0089] If the task migration method of the present application is applied in a console node, then when receiving a task migration request, the status of the task to be migrated in the current node corresponding to the task migration request can be detected. If the task to be migrated is in a running state, verify whether the target task in the target node is consistent with the task to be migrated. In response to the inconsistency between the target task and the task to be migrated, determine the target migration strategy from the preset task migration strategy based on the task type of the task to be migrated. The console node then sends the real-time generated key or the preset key to the current node and the target node respectively, so that the current node encrypts the task to be migrated based on the received key to obtain the encrypted task to be migrated. The encrypted task to be migrated is migrated to the target node based on the target migration strategy, so that the target node decrypts the migrated task based on the key to obtain the decrypted task, and performs task update processing based on the decrypted task.

[0090] The specific implementation process of this embodiment can be referred to in conjunction with the explanations of the aforementioned embodiments, and will not be elaborated here.

[0091] It should be further explained that the execution subject of the data transmission task migration method may be a data transmission task migration device, for example, the data transmission task migration method may be executed by a terminal device or a server or other processing device, wherein the terminal device may be a user equipment (UE), a computer, a mobile device, a user terminal, a terminal, a cellular phone, a cordless phone, a personal digital assistant (PDA), a handheld device, a computing device, a vehicle-mounted device, a wearable device, etc. In some possible implementations, the data transmission task migration method may be implemented by a processor calling a computer-readable instruction stored in a memory.

[0092] Figure 3 FIG. 1 is a block diagram of a migration device for a data transmission task shown in an exemplary embodiment of the present application. Figure 3 As shown, the exemplary data transmission task migration device 300 includes: a state detection module 310, a consistency verification module 320, a migration strategy determination module 330 and a task migration module 340. Specifically:

[0093] The status detection module 310 is used to perform status detection on the task to be migrated corresponding to the task migration request in response to the received task migration request. The task migration request is used to migrate the task to be migrated from the current node to the target node.

[0094] The consistency verification module 320 is used to verify whether the target task in the target node is consistent with the task to be migrated if the task to be migrated is in the running state.

[0095] The migration strategy determination module 330 is configured to determine a target migration strategy from preset task migration strategies based on the task type of the task to be migrated in response to the inconsistency between the target task and the task to be migrated.

[0096] The task migration module 340 is used to migrate the task to be migrated to the target node based on the target migration strategy, and perform task update processing on the target task.

[0097] In the migration device of the exemplary data transmission task, in response to the received task migration request, the current task status of the task to be migrated is determined by performing status detection on the task to be migrated corresponding to the task migration request, and the task migration request is used to migrate the task to be migrated from the current node to the target node; if the task to be migrated is in the running state, it is necessary to verify whether the target task in the target node is consistent with the task to be migrated, thereby determining whether the task to be migrated needs to be migrated to the target node; in response to the inconsistency between the target task and the task to be migrated, it is characterized that the task to be migrated needs to be migrated to the target node, so based on the task type of the task to be migrated, a target migration strategy suitable for migrating the task to be migrated is determined from the preset task migration strategy; based on the target migration strategy, the task to be migrated is migrated to the target node, and the target task is updated so that the target node runs the updated task, thereby realizing task migration. The migration method of the data transmission task of the present application determines the difference points of the data transmission tasks in different nodes through consistency verification to ensure the continuous operation of the data transmission task before and after the migration, and based on the automated operation, eliminates the task migration risks caused by manual operation, simplifies the workflow and improves the efficiency of task migration.

[0098] It should be noted that the device provided in the above embodiment and the method provided in the above embodiment belong to the same concept, wherein the specific manner in which each module and unit performs the operation has been described in detail in the method embodiment, and will not be repeated here. In practical applications, the device provided in the above embodiment can distribute the above functions to different functional modules as needed, that is, divide the internal structure of the device into different functional modules to complete all or part of the functions described above, and this is not limited here.

[0099] The functions of each module can be found in the embodiment of the migration method of the data transmission task, which will not be described in detail here.

[0100] See also Figure 4 , Figure 4 1 is a schematic diagram of the structure of an embodiment of an electronic device of the present application. The electronic device 100 includes a memory 101 and a processor 102, and the processor 102 is used to execute program instructions stored in the memory 101 to implement the steps in the above-mentioned migration method embodiment of any data transmission task. In a specific implementation scenario, the electronic device 100 may include but is not limited to: a microcomputer, a server, and in addition, the electronic device 100 may also include a mobile device such as a laptop computer and a tablet computer, which is not limited here.

[0101] Specifically, the processor 102 is used to control itself and the memory 101 to implement the steps in the migration method embodiment of any of the above-mentioned data transmission tasks. The processor 102 can also be called a CPU (Central Processing Unit). The processor 102 may be an integrated circuit chip with signal processing capabilities. The processor 102 can also be a general-purpose processor, a digital signal processor (Digital Signal Processor, DSP), an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), a field-programmable gate array (Field-Programmable Gate Array, FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc. In addition, the processor 102 can be implemented by an integrated circuit chip.

[0102] In this exemplary electronic device, in response to a received task migration request, the current task status of the task to be migrated is determined by performing a status detection on the task to be migrated corresponding to the task migration request, and the task migration request is used to migrate the task to be migrated from the current node to the target node; if the task to be migrated is in a running state, it is necessary to verify whether the target task in the target node is consistent with the task to be migrated, thereby determining whether the task to be migrated needs to be migrated to the target node; in response to the inconsistency between the target task and the task to be migrated, it is characterized that the task to be migrated needs to be migrated to the target node, so based on the task type of the task to be migrated, a target migration strategy suitable for migrating the task to be migrated is determined from the preset task migration strategy; based on the target migration strategy, the task to be migrated is migrated to the target node, and the target task is updated so that the target node runs the updated task, thereby realizing task migration. The migration method of the data transmission task of the present application determines the difference points of the data transmission tasks in different nodes through consistency verification to ensure the continuous operation of the data transmission task before and after the migration, and based on the automated operation, eliminates the task migration risk caused by manual operation, simplifies the workflow and improves the efficiency of task migration.

[0103] See also Figure 5 , Figure 5 The computer-readable storage medium 110 stores program instructions 111 that can be executed by a processor, and the program instructions 111 are used to implement the steps in the above-mentioned migration method embodiment of any data transmission task.

[0104] In this exemplary storage medium, in response to a received task migration request, the state of the task to be migrated corresponding to the task migration request is detected by running the program instructions in the storage medium, and the current task state of the task to be migrated is determined. The task migration request is used to migrate the task to be migrated from the current node to the target node; if the task to be migrated is in a running state, it is necessary to verify whether the target task in the target node is consistent with the task to be migrated, thereby determining whether the task to be migrated needs to be migrated to the target node; in response to the inconsistency between the target task and the task to be migrated, it is characterized that the task to be migrated needs to be migrated to the target node, so based on the task type of the task to be migrated, a target migration strategy suitable for migrating the task to be migrated is determined from the preset task migration strategy; based on the target migration strategy, the task to be migrated is migrated to the target node, and the target task is updated so that the target node runs the updated task, thereby realizing task migration. The migration method of the data transmission task of the present application determines the difference points of the data transmission tasks in different nodes through consistency verification to ensure the continuous operation of the data transmission task before and after the migration, and based on the automated operation, eliminates the task migration risk caused by manual operation, simplifies the workflow and improves the efficiency of task migration.

[0105] In some embodiments, the functions or modules included in the device provided by the embodiments of the present disclosure can be used to execute the method described in the above method embodiments. The specific implementation can refer to the description of the above method embodiments, and for the sake of brevity, it will not be repeated here.

[0106] The above description of various embodiments tends to emphasize the differences between the various embodiments. The same or similar aspects can be referenced to each other, and for the sake of brevity, they will not be repeated herein.

[0107] In the several embodiments provided in the present application, it should be understood that the disclosed methods and devices can be implemented in other ways. For example, the device implementation described above is only schematic. For example, the division of modules or units is only a logical function division. There may be other division methods in actual implementation, such as units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, and the indirect coupling or communication connection of devices or units can be electrical, mechanical or other forms.

[0108] In addition, each functional unit in each embodiment of the present application can be integrated into a processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of a software functional unit. If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor (processor) to perform all or part of the steps of each implementation method of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program code.

Claims

1. A method for migrating a data transmission task, characterized in that: The method comprises: In response to a received task migration request, performing a status detection on a task to be migrated corresponding to the task migration request, wherein the task migration request is used to migrate the task to be migrated from a current node to a target node; If the task to be migrated is in a running state, verifying whether the target task in the target node is consistent with the task to be migrated; In response to the target task and the task to be migrated being inconsistent, determining a target migration strategy from preset task migration strategies based on a task type of the task to be migrated; The step of determining the target migration strategy from the preset task migration strategy based on the task type of the task to be migrated includes: if the task type of the task to be migrated is a full transfer type, determining the full migration strategy as the target migration strategy; if the task type of the task to be migrated is a partial transfer type, determining the partial migration strategy as the target migration strategy; Migrating the task to be migrated to the target node based on the target migration strategy, and performing task update processing on the target task; The step of migrating the task to be migrated to the target node based on the target migration strategy and performing task update processing on the target task includes: suspending the task to be migrated and the target task to obtain a suspended task to be migrated and a suspended target task; determining a local task to be migrated based on the data transmission progress of the suspended task to be migrated; migrating the local task to be migrated to the target node, and performing local update processing on the suspended target task based on the local task to be migrated.

2. The method according to claim 1, characterized in that After determining the full migration strategy as the target migration strategy if the task type of the task to be migrated is a full transfer type, the method includes: Performing a stop operation process on the task to be migrated and the target task to obtain a stopped task to be migrated and a stopped target task; The stopped tasks to be migrated are migrated to the target node, and a full update process is performed on the stopped target tasks based on the stopped tasks to be migrated.

3. The method according to claim 1, characterized in that If the task to be migrated is in a running state, verifying whether the target task in the target node is consistent with the task to be migrated includes: Performing consistency verification based on the task information of the task to be migrated and the task information of the target task; In response to the task information of the task to be migrated matching the task information of the target task, it is determined that the task to be migrated is consistent with the target task.

4. The method according to claim 3, characterized in that The task information includes task basic information, task interaction information, and mapping field information. The consistency check processing based on the task information to be migrated and the candidate task information of each candidate task in the target node includes: In response to the task basic information between the task to be migrated and the candidate task being consistent, the task interaction information being consistent, and the mapping field information being consistent, determining that the task information of the task to be migrated matches the task information of the target task; In response to at least one of the task basic information, the task interaction information and the mapping field information being inconsistent between the task to be migrated and the candidate task, it is determined that the task information of the task to be migrated does not match the task information of the target task.

5. The method according to claim 1, characterized in that After performing status detection on the task to be migrated corresponding to the task migration request in response to the received task migration request, the method further includes: In response to the task to be migrated being in a suspended state, recording a current task progress of the task to be migrated; Determining a partial task to be migrated among the tasks to be migrated based on the current task progress of the tasks to be migrated; Migrate the local task to be migrated to the target node, and perform task update processing on the target task.

6. A migration device for a data transmission task, characterized in that: include: A status detection module, configured to perform status detection on a task to be migrated corresponding to a received task migration request in response to the task migration request, wherein the task migration request is used to migrate the task to be migrated from a current node to a target node; A consistency verification module, used for verifying whether the target task in the target node is consistent with the task to be migrated if the task to be migrated is in a running state; a migration strategy determination module, configured to determine a target migration strategy from preset task migration strategies based on a task type of the task to be migrated in response to the target task and the task to be migrated being inconsistent; The step of determining the target migration strategy from the preset task migration strategy based on the task type of the task to be migrated includes: if the task type of the task to be migrated is a full transfer type, determining the full migration strategy as the target migration strategy; if the task type of the task to be migrated is a partial transfer type, determining the partial migration strategy as the target migration strategy; A task migration module, used to migrate the task to be migrated to the target node based on the target migration strategy, and perform task update processing on the target task; The step of migrating the task to be migrated to the target node based on the target migration strategy and performing task update processing on the target task includes: suspending the task to be migrated and the target task to obtain a suspended task to be migrated and a suspended target task; determining a local task to be migrated based on the data transmission progress of the suspended task to be migrated; migrating the local task to be migrated to the target node, and performing local update processing on the suspended target task based on the local task to be migrated.

7. An electronic device, characterized in that: The method comprises a memory and a processor, wherein the processor is used to execute program instructions stored in the memory to implement the method according to any one of claims 1 to 5.

8. A computer-readable storage medium having program instructions stored thereon, characterized in that: When the program instructions are executed by a processor, the method according to any one of claims 1 to 5 is implemented.

Citation Information

Patent Citations

  • Data migration system and method

    CN107122355A

  • Micro-grid optimization scheduling method, system, medium, equipment and terminal

    CN117498306A