Data migration method and device, equipment, storage medium and computer program

Abstract

The invention discloses a data migration method and device, equipment, a storage medium and a computer program, and relates to the technical field of information processing.The data migration method comprises the steps that consumption data is pulled from a preset source theme and subjected to grouping processing, and grouped data is obtained; the grouped data comprises a first relative position of the consumption data in a preset source theme; the grouped data are asynchronously sent to a preset target theme, a sending result is obtained, the grouped data comprise at least two pieces of sub-data, and one piece of sub-data is independently sent to the preset target theme every time according to preset interval time; the sending result is recorded and responded through a callback function, and a response result is obtained; if the response result shows that the sending result is that the consumption data is successfully sent, recording a second relative position corresponding to the first relative position in the preset target theme; and submitting the first relative position and the second relative position to a preset source theme. According to the method, after data migration, it can be guaranteed that data are not lost and repeated.

Description

Technical Field

[0001] The present application relates to the field of information processing technology, and in particular to a data migration method, device, equipment, storage medium and computer program. Background Art

[0002] Cross-cluster data synchronization migration is a process that involves migrating data from one cluster to another in real time or near real time. It is usually based on Kafka MirrorMaker to perform cross-cluster data synchronization migration, and based on Kafka expansion and partition-based data migration. MirrorMaker is only responsible for Topic data synchronization, but the most important part should be how the application system switches to the Topic of the new cluster after synchronization, which it does not solve.

[0003] There are usually two solutions to this problem: The first method is to transform the application to consume both the old and new topics at the same time, and then stop consuming after a period of time, and remove the old topic at the same time. This ensures that data is not lost during the switching process, but there will be a large amount of data duplication. The second method is to estimate a time to reset the offset for the new topic (determined by the data consumed by the application), stop the application, reset the offset, and then restart the application. This solution requires users to have a sufficient understanding of Kafka offsets to ensure that data is not lost and data duplication is reduced after the reset. In addition, Kafka MirrorMaker needs to ensure the version compatibility between the source Kafka cluster and the target Kafka cluster, otherwise compatibility issues may occur. Overall, adopting this solution will bring additional workload and data risks to the user.

[0004] Therefore, how to ensure that data is not lost and not duplicated during data migration is a problem that needs to be solved urgently. Summary of the invention

[0005] The main purpose of this application is to provide a data migration method, device, equipment, storage medium and computer program, aiming to solve the technical problem of how to ensure that data is not lost and data is not duplicated.

[0006] To achieve the above object, the present application provides a data migration method, which includes:

[0007] Pull consumption data from a preset source topic and perform grouping processing to obtain grouped data, wherein the grouped data includes a first relative position of the consumption data in the preset source topic;

[0008] Asynchronously sending the grouped data to a preset target topic to obtain a sending result, wherein the grouped data includes at least two sub-data, and each time one of the sub-data is sent to the preset target topic separately at a preset interval;

[0009] Recording and responding to the sending result through a callback function to obtain a response result;

[0010] If the response result shows that the sending result is that the consumption data is sent successfully, then the second relative position corresponding to the first relative position in the preset target topic is recorded;

[0011] Submit the first relative position and the second relative position to a preset source topic.

[0012] In one embodiment, after the step of recording and responding to the sending result through a callback function to obtain a response result, the following steps are included:

[0013] If the response result shows that the sending result is a failure to send the consumption data or a timeout of the record response, then sleep for a preset sleep time;

[0014] After the sleep mode ends, the steps of pulling consumption data from the preset source topic and performing grouping processing are executed again to obtain grouped data and record the successful data in the second relative position, where the successful data is the grouped data that is sent successfully.

[0015] In one embodiment, after the step of recording the second relative position corresponding to the first relative position in the preset target subject, the step further includes:

[0016] Constructing a relative position mapping table through the first relative position and the second relative position;

[0017] The first relative position and the second relative position are reset according to the relative position mapping table.

[0018] In one embodiment, after the step of recording and responding to the sending result through a callback function to obtain a response result, the following steps are included:

[0019] If the sending result is that all the packet data fails to be sent or the record response times out, the packet data is sent again.

[0020] In one embodiment, the data migration method further includes:

[0021] Receive a newly created synchronization task, wherein the synchronization task generates consumption data by reading and processing data;

[0022] Scan the synchronization task regularly, and if the sending result is that the consumption data fails to be sent, resend the synchronization task;

[0023] If the synchronization task waits for a timeout, the synchronization task is reissued;

[0024] After the synchronization task is completed, the synchronization task is offline through the user's offline instruction.

[0025] In one embodiment, the data migration method is executed by a query engine, the synchronization task is deployed on the first node and the second node, and the data migration method includes:

[0026] receiving a synchronization process, a sending thread, and a process heartbeat registered by the first node, wherein the synchronization process, the sending thread, and the process heartbeat belong to the query engine;

[0027] Sending the synchronization task to the second node or taking the synchronization task offline;

[0028] If the first node fails to run due to an error, the second node executes the step of issuing the synchronization task or taking the synchronization task offline;

[0029] If the second node fails to run due to an error, the registration synchronization process, sending thread and process heartbeat are executed through the first node.

[0030] In addition, to achieve the above-mentioned purpose, the present application also provides a data migration device, the data migration device comprising:

[0031] A data pulling module, used to pull consumption data from a preset source topic and perform grouping processing to obtain grouped data, wherein the grouped data includes a first relative position of the consumption data in the preset source topic;

[0032] A data sending module, used for asynchronously sending the grouped data to a preset target topic to obtain a sending result, wherein the grouped data includes at least two sub-data, and each time the sub-data is sent to the preset target topic separately at a preset interval;

[0033] A sending record module is used to record and respond to the sending result through a callback function to obtain a response result;

[0034] A relative position recording module, configured to record a second relative position corresponding to the first relative position in a preset target topic if the response result shows that the sending result is that the consumption data is sent successfully;

[0035] A relative position submission module is used to submit the first relative position and the second relative position to a preset source topic.

[0036] In addition, to achieve the above-mentioned purpose, the present application also provides a data migration device, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is used to execute the steps of the data migration method described above.

[0037] In addition, to achieve the above-mentioned purpose, the present application also provides a storage medium, which is a computer-readable storage medium, and a program for implementing the data migration method is stored on the computer-readable storage medium. The program for implementing the data migration method is executed by a processor to implement the steps of the data migration method as described above.

[0038] In addition, to achieve the above-mentioned purpose, the present application also provides a computer program product, including a computer program, which implements the steps of the above-mentioned data migration method when executed by a processor.

[0039] The present application provides a data migration method, which pulls consumption data from a preset source topic and performs group processing to obtain group data, wherein the group data contains the first relative position of the consumption data in the preset source topic; the group data is asynchronously sent to a preset target topic to obtain a sending result, wherein the group data contains at least two sub-data, and each time one of the sub-data is sent to the preset target topic separately at a preset interval; a callback function is used to record and respond to the sending result to obtain a response result; if the response result shows that the sending result is that the consumption data is sent successfully, then the second relative position corresponding to the first relative position in the preset target topic is recorded; and the position is recorded after successful sending. If the response result returned by the callback function shows that the consumption data is sent successfully, then the second relative position of the data in the target topic is recorded. In this way, even if an interruption occurs during the data migration process, it can continue from the position of the last successful sending. The first relative position and the second relative position are submitted to the preset source topic.

[0040] In summary, the present application submits the first relative position and the second relative position back to the source topic through group processing and recording relative positions combined with asynchronous sending and interval control, thereby overcoming the technical defect that MirrorMaker is only responsible for Topic data synchronization, and ensuring that data is not lost and duplicated during data migration. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0042] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0043] Figure 1 A flowchart of the first embodiment of the data migration method of this application is provided;

[0044] Figure 2 A schematic diagram of location resetting provided for an embodiment of the data migration method of the present application;

[0045] Figure 3 Another schematic diagram of resetting the position provided in the data migration method embodiment of the present application;

[0046] Figure 4 A schematic diagram of a relative position mapping process of a data migration method provided in an embodiment of the present application;

[0047] Figure 5 A flowchart of the second embodiment of the data migration method of this application is provided;

[0048] Figure 6 A synchronization task state transition diagram provided for the second embodiment of the data migration method of this application;

[0049] Figure 7 A synchronization tool execution logic diagram provided for the second embodiment of the data migration method of this application;

[0050] Figure 8 This is a schematic diagram of the module structure of the data migration device according to an embodiment of the present application;

[0051] Fig. 9 It is a structural schematic diagram of the data migration device in an embodiment of the present application.

[0052] The purpose, features and advantages of this application will be further described in conjunction with the embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0053] It should be understood that the specific embodiments described herein are only used to explain the technical solutions of the present application and are not used to limit the present application.

[0054] In order to better understand the technical solution of the present application, a detailed description will be given below in conjunction with the accompanying drawings and specific implementation methods.

[0055] The main solution of the present application is: pull consumption data from a preset source topic and perform grouping processing to obtain grouped data, wherein the grouped data includes a first relative position of the consumption data in the preset source topic; asynchronously send the grouped data to a preset target topic to obtain a sending result, wherein the grouped data includes at least two sub-data, and each sub-data is sent separately to the preset target topic at a preset interval; record and respond to the sending result through a callback function to obtain a response result; if the response result shows that the sending result is that the consumption data is sent successfully, then record the second relative position corresponding to the first relative position in the preset target topic; submit the first relative position and the second relative position to the preset source topic.

[0056] There are usually two solutions: The first method is to transform the application and consume both the old and new topics at the same time. After consuming for a period of time, stop consuming and remove the old topic at the same time. This ensures that data is not lost during the switching process, but there will be a large amount of data duplication. The second method is to estimate a time to reset the offset for the new topic (determined by the data consumed by the application), stop the application, reset the offset, and restart the application. This solution requires users to have a sufficient understanding of Kafka offsets to ensure that data is not lost and data duplication is less after the reset. In addition, Kafka MirrorMaker needs to ensure the version compatibility between the source Kafka cluster and the target Kafka cluster, otherwise compatibility issues may occur. Overall, adopting this solution will bring additional workload and data risks to the user. Therefore, how to ensure that data is not lost and data is not duplicated during data migration is a problem that needs to be solved urgently.

[0057] The present application submits the first relative position and the second relative position back to the source topic by combining group processing and recording relative positions with asynchronous sending and interval control, thereby overcoming the technical defect that MirrorMaker is only responsible for Topic data synchronization, and ensuring that data is not lost and not duplicated during data migration.

[0058] It should be noted that the execution subject of this embodiment can be a data migration device, or a computing service device with data processing, network communication and program running functions, such as a tablet computer, a personal computer, a mobile phone, etc., or a query engine capable of realizing the above functions, etc., and this embodiment does not specifically limit this. The following takes the data migration device as the execution subject as an example to illustrate this embodiment and the following embodiments.

[0059] Based on this, this application proposes a data migration method of the first embodiment, please refer to Figure 1 , the data migration method includes steps S10 to S30:

[0060] Step S10, pulling consumption data from a preset source topic and performing grouping processing to obtain grouped data, wherein the grouped data includes a first relative position of the consumption data in the preset source topic;

[0061] It should be noted that the preset source topic refers to the starting point where data is pulled (pull consumption) during the data migration process. Usually, the source topic is established as the preset source topic during data migration. Grouping data means grouping data according to specific rules or standards for easy management and operation. For example, the data of each partition is grouped by label. For example, if partition 0 contains 500 data, the data is numbered from 0 to 499.

[0062] In addition, it should be noted that consuming data refers to subscribing to specific topics or queues, and then reading and processing messages published to these topics or queues. The first relative position refers to the relative position of the consumed data after grouping in the preset source topic. For example, if the grouped data is numbered from 0 to 499, then 0 to 499 is the relative position of the corresponding data.

[0063] Step S20, asynchronously sending the grouped data to a preset target topic to obtain a sending result, wherein the grouped data includes at least two sub-data, and each time one of the sub-data is sent to the preset target topic separately at a preset interval;

[0064] It should be noted that the preset target topic refers to the end point of data transmission or flow. In message queue systems such as Apache Kafka or RocketMQ, a topic is the basic unit of data storage and distribution. It contains one or more partitions for parallel processing and improved throughput. The sending result refers to the feedback information after the data sending operation is completed. Sub-data refers to a piece of data in the grouped data. For example, when there are 500 grouped data numbered from 0 to 499, the grouped data numbered 0 is a sub-data.

[0065] In addition, it should be noted that the preset interval time refers to the interval time between each transmission of sub-data, and the preset interval time can be adjusted according to time conditions, such as 50ms and 30ms.

[0066] This embodiment ensures that data will not be lost during data migration by setting a preset interval time.

[0067] Step S30, recording and responding to the sending result through a callback function to obtain a response result;

[0068] It should be noted that a callback function is a function that accepts another function as a parameter, or registers a function in an object, and then this function is called when an event occurs. The response result refers to the feedback information received by the system during the data migration process, especially after the data is asynchronously sent from the preset source topic to the preset target topic.

[0069] It can be understood that obtaining a message (response result) indicating whether the packet data is sent successfully through a callback function provides a concise method for handling asynchronous events, allowing additional parameters and context to be passed to the called function.

[0070] In a feasible embodiment, step S30 further includes steps S301-S302:

[0071] Step S301, if the response result shows that the sending result is a failure to send consumption data or a timeout of the record response, sleep for a preset sleep time;

[0072] It should be noted that the preset sleep time refers to a period of time automatically set by the system when a sending operation fails or encounters a specific error. During this period, the consumer or migration task will be suspended and wait for another attempt to send the operation.

[0073] It is understandable that the preset sleep time can be a preset fixed time, such as 60s, or a time that changes dynamically according to the number of restarts, such as 30s multiplied by the number of restarts. A restart is required after each sleep, so the number of sleep times and the number of restarts are the same in value.

[0074] Step S302, after the sleep mode ends, re-execute the step of pulling consumption data from the preset source topic and performing grouping processing to obtain grouped data and record the successful data in the second relative position, where the successful data is the grouped data that is sent successfully.

[0075] It is understandable that after the sleep mode ends, the grouped data successfully sent before the sleep mode is recorded in the second relative position, and the consumption data is re-pulled for data transmission, ensuring that all consumption data are sent to the preset target topic.

[0076] This embodiment solves the problem of sending failure or response timeout that may be encountered during data migration by introducing a method of presetting the sleep time. When the sending operation fails or a timeout occurs, the system will automatically enter the preset sleep state, avoiding continuous failures and resource waste that may be caused by immediate retry. After the sleep period ends, the system will re-execute the data pulling and grouping operations to ensure the integrity and consistency of the data. This method effectively reduces the failure rate caused by transient failures or network fluctuations, and improves the reliability and stability of data migration. Ultimately, through this strategy, the effect of optimizing data migration performance, enhancing system fault tolerance, and ensuring business continuity is achieved.

[0077] In another feasible embodiment, step S30 records the sending result in response through a callback function, and after the step of obtaining the response result, step A10 is also included:

[0078] Step A10: if the sending result is that all the packet data fails to be sent or the record response times out, the packet data is sent again.

[0079] It is understandable that when all consumption data in the preset source data fails to be sent or all responses time out, it means that there is no data migration successfully, and there is no need to record the second relative position and resend the grouped data.

[0080] The method of this embodiment can reduce the amount of data duplication during data migration, thereby improving the accuracy of data migration.

[0081] Step S40: if the response result shows that the sending result is that the consumption data is sent successfully, then the second relative position corresponding to the first relative position in the preset target topic is recorded;

[0082] It should be noted that the second relative position refers to the offset of the data in the preset target topic, which is an identifier of the data consumption or storage location. For example, the second relative position is represented by a number. The first relative position is 499 and the second relative position can be 1499.

[0083] In a feasible embodiment, after the step of recording the second relative position corresponding to the first relative position in the preset target subject in step S40, the following steps S401 to S402 are included:

[0084] Step S401, constructing a relative position mapping table through the first relative position and the second relative position;

[0085] It should be noted that the relative position mapping table refers to the correspondence between the storage positions of data in the preset source theme and the preset target theme, that is, the mapping relationship between the first relative position and the second relative position.

[0086] Step S402: reset the first relative position and the second relative position according to the relative position mapping table.

[0087] It can be understood that the relative position mapping table is used to record the mapping relationship between the source and target topics on the offset during the synchronization process. Based on this relationship, the consumption switching of topics between clusters is completed. After recording the offset mapping relationship, the user can reset the position based on the offset mapping relationship.

[0088] In a possible embodiment, if Figure 2 , the step of resetting the position includes:

[0089] The consuming application is down (if it is not down, the data will be duplicated);

[0090] Query the latest offset consumed by the current group (0 partition offset = 550);

[0091] According to topic, group, and partition, query the corresponding offset-0 partition (corresponding interval: source [500,599], target [1600,2099], because it is a one-to-one mapping, the source offset: 550 corresponds to the target offset: 1650);

[0092] Group consumes the target topic and resets offset to 1650.

[0093] Among them, group is grouped data, topic includes preset source topic and preset target topic, source [500,599] is the range of the first relative position, and target [1600,2099] is the range of the second relative position. Source offset is a first relative position, and target offset is a second relative position.

[0094] In another possible embodiment, if Figure 3 , the step of resetting the position includes:

[0095] The consuming application is down (if it is not down, the data will be duplicated);

[0096] Query the latest offset consumed by the current group2 (0 partition offset = 1550);

[0097] The corresponding offset is not found according to the topic, group, and partition query (this means that the data near this location was written due to an exception or other reasons, and the application did not record the mapping relationship of this part. This part of the data is duplicated due to synchronization retries).

[0098] Since it is not found, we need to search backwards, so we take the earliest data > 1550, which is 1600, and the corresponding offset is 500;

[0099] The group consumes the target topic and resets offset to 500.

[0100] The above embodiment can solve the data consistency and duplication problems in the data migration process by resetting the relative positions to ensure one-to-one correspondence between the relative positions in the relative position mapping table.

[0101] Step S50: submitting the first relative position and the second relative position to a preset source topic.

[0102] In the same group of packet data, the first relative position and the second relative position have a fixed mapping relationship. For example, the first relative position is 0 to 499, and the second relative position may be 1000 to 1499.

[0103] The data migration method of this embodiment solves the problem of data loss and duplication that may occur in large-scale data transmission by asynchronously sending packet data and using a callback function to record the sending results. By accurately recording the first relative position and the second relative position of the data in the source subject and the target subject, and updating these position information after successful sending, the effect of ensuring data consistency and improving the reliability of data migration is achieved. In addition, by sending sub-data at preset intervals, load balancing is effectively achieved, avoiding pressure overload on the target system, and optimizing the utilization and scheduling of system resources. Finally, this method supports the high availability of the system by enhancing the monitorability and traceability of data migration, reduces the impact of data migration on business operations, and improves user experience and system stability.

[0104] For example, to help understand the technical concept or technical principle of the data migration method of this embodiment and the above-mentioned embodiment 1, please refer to Figure 4 , Figure 4 A relative position mapping flow chart of a data migration method is provided, as follows:

[0105] The first step is to create a consumer and start consuming data with offsets 0 to 499. After the data is sent successfully, the offsets written to the target topic are 1000 to 1499. The offset mapping relationship of this batch of synchronized data is recorded (source 0, target 1000), and then the offset = 500 is manually submitted.

[0106] The second step is to continue consuming the data with offsets 500 to 999. If all the data fails to be sent due to network reasons, the offset will not be submitted, the thread will be exited, and the synchronization task will be restarted.

[0107] The third step is to restart the synchronization task. The consumer group finds that the last offset is 500 and consumes the data from 500 to 999. This time, the data from 500 to 599 is partially successful and partially failed. The data from 500 to 599 is successfully written, and the corresponding target topic offset is 1500 to 1599. In this case, the offset is not submitted, the thread is exited, and the synchronization task is restarted.

[0108] Step 4: Restart the synchronization task. The consumer group finds that the last offset is 500, consumes the data from 500 to 999, sends it successfully, and writes the offsets from 1600 to 2099 to the target topic. Record the offset mapping relationship of this batch of synchronized data (source 500, target 1600), and manually submit offset = 1000.

[0109] The fifth step is to consume in a loop and send data.

[0110] Through the above mapping process, when switching topics across clusters, it can be ensured that data is not lost and offsets are accurately mapped, so that duplicate data will not be generated.

[0111] It should be noted that the above examples are only used to assist in understanding the present application and do not constitute a limitation on the data migration method of the present application. More simple transformations based on this technical concept are all within the scope of protection of the present application.

[0112] Based on the first embodiment of the present application, in the second embodiment of the present application, the same or similar contents as those in the above-mentioned embodiment 1 can refer to the above introduction, and will not be repeated later. Figure 5 , the data migration method further includes:

[0113] Step S01, receiving a newly created synchronization task, wherein the synchronization task generates consumption data by reading and processing data;

[0114] It should be noted that a synchronization task generally refers to a planned or real-time operation, which is responsible for copying or moving data between two or more systems, databases or storage media and ensuring data consistency. In this embodiment, the synchronization task is a task of migrating data from a preset source topic to a preset target topic.

[0115] Step S02, regularly scanning the synchronization task, if the sending result is that the consumption data fails to be sent, re-issuing the synchronization task;

[0116] Step S03, if the synchronization task waits for timeout, re-issuing the synchronization task;

[0117] Step S04: after the synchronization task is completed, the synchronization task is offline through the user's offline instruction.

[0118] It should be noted that the offline instruction is a command issued by the user and is used to notify the user to stop the running synchronization task.

[0119] It can be understood that this embodiment manages synchronization tasks through synchronization, and the state transition of the synchronization tasks is as follows: Figure 6 , as follows:

[0120] The first step is to receive a newly created synchronization task, which generates consumption data by reading and processing the data:

[0121] In the flowchart, this step may correspond to the "New Course" state, which is the starting point of the process. At this stage, the system receives new synchronization tasks and starts reading and processing the data to generate consumption data.

[0122] The second step is to periodically scan the synchronization task. If the sending result is that the consumption data fails to be sent, the synchronization task is resent:

[0123] In the flowchart, this step may correspond to the "Synchronous Task Scan" and "Scheduled Task Scan" links. The system will periodically check the execution status of the synchronization task. If it is found that the consumption data fails to be sent (possibly at a certain checkpoint in the "Synchronizing" or "In Progress" state), the system will re-issue the synchronization task, which may be reflected in the flowchart as returning to the "New Lesson" state or re-entering the "Synchronous Task Scan" link.

[0124] Step 3: If the synchronization task waits for a timeout, re-send the synchronization task:

[0125] In the flowchart, this step may correspond to the "Synchronizing" or "In Progress" state. If the task waiting timeouts, the system will re-issue the synchronization task. This may be represented by a decision point in the flowchart, such as "Check whether the task has timed out". If it has timed out, re-enter the "New Course" state or the "Synchronization Task Scan" link.

[0126] Step 4: After the synchronization task is completed, the synchronization task is offline through the user's offline instruction:

[0127] In the flowchart, this step may correspond to the situation where the user in the "Synchronizing" or "In Progress" state chooses to log off. After the user issues the log off instruction, the system updates the task status to "Offline" and finally completes the offline process of the synchronization task. This may be shown in the flowchart as a transition from the "Synchronizing" or "In Progress" state to the "Offline" state, and finally ends the process.

[0128] Among them, new course: receive new synchronization tasks and start data reading and processing. Synchronous task scanning and scheduled task scanning: regularly check the task status and handle the failure of sending. Decision points in the synchronization and ongoing states: check whether the task has timed out and handle the timeout. User chooses to go offline: in the synchronization or ongoing state, the user can issue an offline command, causing the task to enter the offline state.

[0129] In a feasible embodiment, the sending and offline of synchronization tasks are implemented through a synchronization management module, and the synchronization management module includes a synchronization tool management end, and interacts with the synchronization process end and the scheduled task end to implement the data migration method of this embodiment. The execution logic of the synchronization tool management end is as follows: Figure 7 , as follows:

[0130] User added tasks: Users add synchronization tasks through the management end.

[0131] Generate synchronization tasks: The system generates specific synchronization tasks based on the tasks added by the user.

[0132] Call the synchronous task interface: The system calls the synchronous task interface to process the generated task.

[0133] Set the task status to Synchronizing: When the synchronization task starts executing, the system updates the task status to "Synchronizing".

[0134] Scan new status tasks: The system will periodically scan new status tasks.

[0135] Scan tasks that have been issued but are still being executed.

[0136] User-initiated offline operation: Users can initiate an offline operation through the management end and request to stop a synchronization task.

[0137] Set the task status to offline: After receiving an offline request, the system updates the task status to "offline".

[0138] Scan offline tasks: The system will scan tasks with the status of "offline".

[0139] Offline task: For tasks with the status of "offline", the system calls the offline task interface to process them.

[0140] Process restart: If the process is restarted, the system needs to re-issue the synchronization task.

[0141] Scan process to start task distribution: The system will scan tasks that need to be re-distributed due to process restart.

[0142] Scan thread tasks and resend them: For thread tasks, if sending fails, the system will scan and resend these tasks.

[0143] Thread send failure restart: If a thread fails when sending a task, the system will restart the thread.

[0144] Scheduled tasks: The system includes a scheduled task function, which may be used to periodically execute synchronization tasks.

[0145] Add process record: The system records the added process information.

[0146] Process flow record: The system records the operation flow of the process, which may be used for auditing or monitoring.

[0147] Task flow: The system records the operation flow of tasks, which may also be used for auditing or monitoring.

[0148] Scan tasks in the "In-Task" state: The system scans tasks in the "In-Task" state for further processing.

[0149] Calling the interface returns success: After the interface is called successfully, the system may update the task status or perform other operations.

[0150] As can be seen from the figure, the logic of this synchronization tool management end involves task creation, status management, periodic scanning, user interaction, and task synchronization and offline processing. The entire process emphasizes the change of task status and the processing logic of tasks in different statuses, as well as the restart and retry mechanism in case of failure. The above system is a query engine or data migration device for implementing this embodiment.

[0151] In another feasible embodiment, synchronization jobs are selected for deployment based on node load conditions, and node, process, and thread indicator data are regularly checked based on actively collected monitoring indicator data to achieve self-healing restart of faults.

[0152] It is understandable that the node responsible for synchronization tasks may become unavailable.

[0153] Therefore, in a feasible embodiment, the data migration method is executed by a query engine, the synchronization task is deployed on the first node and the second node, and the data migration method includes steps B10 to B40:

[0154] Step B10, receiving the synchronization process, sending thread and process heartbeat registered by the first node, wherein the synchronization process, sending thread and process heartbeat belong to the query engine;

[0155] Step B20, sending the synchronization task to the second node or taking the synchronization task offline;

[0156] Step B30, if the first node fails to run due to an error, executing the step of issuing the synchronization task or taking the synchronization task offline through the second node;

[0157] Step B40: If the second node fails to run due to an error, the first node is used to execute the registration synchronization process, send thread and process heartbeat.

[0158] It should be noted that the first node refers to the node responsible for the registered synchronization process, sending thread and process heartbeat, and the second node refers to the node responsible for sending and offline synchronization tasks.

[0159] It is understandable that the synchronization process refers to a process created to complete synchronization management, that is, to manage node registration, synchronization job online, offline, dual-active deployment and monitoring, etc. The sending thread refers to the thread that completes the data migration in the first embodiment. The process heartbeat is a health check mechanism for detecting the operating status of the system or service. The regular sending of heartbeat signals indicates that the node is still active and operating normally.

[0160] In this embodiment, synchronization tasks are deployed through two nodes. When one node fails to operate normally, the synchronization tasks can be sent through the other node, thereby ensuring high availability of synchronization.

[0161] For further information, please refer to Figure 7 , Figure 7 The synchronization task deployment diagram of the data migration method is provided as follows:

[0162] It contains two parts: synchronization node 1 (synchronization module) and synchronization node 2 (synchronization module). Each node contains multiple threads, such as Thread1, Thread2, etc. These threads pull data from different topics (such as topicA, topicB, topicC, topicD) to the memory queue and then process the data. The synchronization tool management end monitors the execution of synchronization tasks and intervenes when necessary. For example, when an exception occurs in the synchronization process, the management end stops the synchronization task and performs corresponding error handling. In addition, the management end is also responsible for updating the synchronization status to ensure that all nodes can understand the current synchronization status in a timely manner.

[0163] The present application also provides a data migration device. Figure 8 , the data migration device comprises:

[0164] The data pulling module 10 is used to pull consumption data from a preset source topic and perform grouping processing to obtain grouped data, wherein the grouped data includes a first relative position of the consumption data in the preset source topic;

[0165] The data sending module 20 is used to asynchronously send the grouped data to a preset target topic to obtain a sending result, wherein the grouped data includes at least two sub-data, and each time the sub-data is sent to the preset target topic separately at a preset interval;

[0166] The sending record module 30 is used to record and respond to the sending result through a callback function to obtain a response result;

[0167] A relative position recording module 40 is used to record a second relative position corresponding to the first relative position in a preset target topic if the response result shows that the sending result is that the consumption data is sent successfully;

[0168] The relative position submission module 50 is used to submit the first relative position and the second relative position to a preset source topic.

[0169] Optionally, the heat dissipation device includes a heat dissipation fan, the heat dissipation parameter includes a heat dissipation rotation speed, and the maximum parameter value includes a maximum rotation speed value;

[0170] The sending record module 30 is also used for:

[0171] If the response result shows that the sending result is a failure to send the consumption data or a timeout of the record response, then sleep for a preset sleep time;

[0172] After the sleep mode ends, the steps of pulling consumption data from the preset source topic and performing grouping processing are executed again to obtain grouped data and record the successful data in the second relative position, where the successful data is the grouped data that is sent successfully.

[0173] Optionally, the relative position recording module 40 is further used for:

[0174] If the sending result is that all the packet data fails to be sent or the record response times out, the packet data is sent again.

[0175] Optionally, the data migration device further includes:

[0176] If it is detected that the estimated temperature is less than or equal to the second preset temperature threshold, obtaining a current coefficient value of the system power coefficient;

[0177] Determining whether the current coefficient value is less than a preset conventional system power coefficient value, wherein the conventional system power coefficient value is greater than the target system power coefficient value;

[0178] If so, the coefficient value of the system power coefficient is increased to the conventional system power coefficient value.

[0179] Optionally, the data migration device further includes:

[0180] Receive a newly created synchronization task, wherein the synchronization task generates consumption data by reading and processing data;

[0181] Scan the synchronization task regularly, and if the sending result is that the consumption data fails to be sent, resend the synchronization task;

[0182] If the synchronization task waits for a timeout, the synchronization task is reissued;

[0183] After the synchronization task is completed, the synchronization task is offline through the user's offline instruction.

[0184] Optionally, the data migration method is executed by a query engine, the synchronization task is deployed on the first node and the second node, and the data migration device further includes:

[0185] receiving a synchronization process, a sending thread, and a process heartbeat registered by the first node, wherein the synchronization process, the sending thread, and the process heartbeat belong to the query engine;

[0186] Sending the synchronization task to the second node or taking the synchronization task offline;

[0187] If the first node fails to run due to an error, the second node executes the step of issuing the synchronization task or taking the synchronization task offline;

[0188] If the second node fails to run due to an error, the registration synchronization process, sending thread and process heartbeat are executed through the first node.

[0189] The data migration device provided in the embodiment of the present application adopts the data migration method in the above embodiment, which can solve the technical problem of how to ensure that data is not lost and data is not repeated. Compared with the prior art, the beneficial effects of the data migration device provided in the embodiment of the present application are the same as the beneficial effects of the data migration method provided in the above embodiment, and other technical features in the data migration device are the same as the features disclosed in the above embodiment method, which will not be repeated here.

[0190] The present application provides a data migration device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the data migration method in the above-mentioned embodiment one.

[0191] Reference below Fig. 9, which shows a schematic diagram of the structure of a data migration device suitable for implementing an embodiment of the present application. The data migration device in the embodiment of the present application may include but is not limited to mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Descriptions), PMPs (Portable Media Players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Fig. 9 The data migration device shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0192] like Fig. 9 As shown, the data migration device may include a processing device 1001 (e.g., a central processing unit, a graphics processor, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM: Read Only Memory) 1002 or a program loaded from a storage device 1003 to a random access memory (RAM: Random Access Memory) 1004. In RAM1004, various programs and data required for the operation of the data migration device are also stored. The processing device 1001, ROM1002, and RAM1004 are connected to each other through a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Generally, the following systems can be connected to the I / O interface 1006: an input device 1007 including, for example, a touch screen, a touch pad, a keyboard, a mouse, an image sensor, a microphone, an accelerometer, a gyroscope, etc.; an output device 1008 including, for example, a liquid crystal display (LCD: Liquid Crystal Display), a speaker, a vibrator, etc.; a storage device 1003 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 1009. The communication device 1009 can allow the data migration device to communicate with other devices wirelessly or by wire to exchange data. Although the data migration device with various systems is shown in the figure, it should be understood that it is not required to implement or have all the systems shown. More or fewer systems can be implemented or have alternatively.

[0193] In particular, according to the embodiments disclosed in the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present application include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device, or installed from a storage device 1003, or installed from a ROM 1002. When the computer program is executed by the processing device 1001, the above-mentioned functions defined in the method of the embodiment disclosed in the present application are executed.

[0194] The data migration device provided by the present application adopts the data migration method in the above embodiment, which can solve the technical problem of how to ensure that data is not lost and data is not repeated. Compared with the prior art, the beneficial effects of the data migration device provided by the present application are the same as the beneficial effects of the data migration method provided by the above embodiment, and the other technical features in the data migration device are the same as the features disclosed in the method of the previous embodiment, which will not be repeated here.

[0195] It should be understood that the various parts disclosed in this application can be implemented by hardware, software, firmware or a combination thereof. In the description of the above embodiments, specific features, structures, materials or characteristics can be combined in any one or more embodiments or examples in a suitable manner.

[0196] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

[0197] The present application provides a computer-readable storage medium having computer-readable program instructions (ie, computer programs) stored thereon, wherein the computer-readable program instructions are used to execute the data migration method in the above-mentioned embodiment.

[0198] The computer-readable storage medium provided in the present application may be, for example, a USB flash drive, but is not limited to electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, systems or devices, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, system or device. The program code contained on the computer-readable storage medium may be transmitted using any appropriate medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination of the above.

[0199] The computer-readable storage medium may be included in the data migration device; or may exist independently without being assembled into the data migration device.

[0200] The above-mentioned computer-readable storage medium carries one or more programs. When the above-mentioned one or more programs are executed by the data migration device, the data migration device: pulls consumption data from a preset source topic and performs group processing to obtain grouped data, wherein the grouped data includes a first relative position of the consumption data in the preset source topic; asynchronously sends the grouped data to a preset target topic to obtain a sending result, wherein the grouped data includes at least two sub-data, and each time the sub-data is sent to the preset target topic separately at a preset interval; records and responds to the sending result through a callback function to obtain a response result; if the response result shows that the sending result is that the consumption data is sent successfully, then the second relative position corresponding to the first relative position in the preset target topic is recorded; and the first relative position and the second relative position are submitted to the preset source topic.

[0201] Computer program code for performing the operations of the present application may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0202] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present application. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0203] The modules involved in the embodiments described in this application may be implemented by software or hardware, wherein the name of the module does not constitute a limitation on the unit itself in some cases.

[0204] The readable storage medium provided in the present application is a computer-readable storage medium, which stores computer-readable program instructions (i.e., computer programs) for executing the above-mentioned data migration method, and can solve the technical problem of how to ensure that data is not lost and data is not duplicated. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in the present application are the same as the beneficial effects of the data migration method provided in the above-mentioned embodiment, and will not be repeated here.

[0205] An embodiment of the present application provides a computer program product, including a computer program, which implements the steps of the above-mentioned data migration method when executed by a processor.

[0206] The computer program product provided in this application can solve the technical problem of how to ensure that data is not lost and data is not duplicated. Compared with the prior art, the beneficial effects of the computer program product provided in the embodiment of this application are the same as the beneficial effects of the data migration method provided in the above embodiment, which will not be repeated here.

[0207] The above are only preferred embodiments of the present application, and are not intended to limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the present application specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent processing scope of the present application.

Claims

1. A data migration method, characterized in that: The data migration method comprises: Pull consumption data from a preset source topic and perform grouping processing to obtain grouped data, wherein the grouped data includes a first relative position of the consumption data in the preset source topic; Asynchronously sending the grouped data to a preset target topic to obtain a sending result, wherein the grouped data includes at least two sub-data, and each time one of the sub-data is sent to the preset target topic separately at a preset interval; Recording and responding to the sending result through a callback function to obtain a response result; If the response result shows that the sending result is that the consumption data is sent successfully, then the second relative position corresponding to the first relative position in the preset target topic is recorded; Submit the first relative position and the second relative position to a preset source topic.

2. The method according to claim 1, characterized in that After the step of recording and responding to the sending result through the callback function to obtain the response result, the method includes: If the response result shows that the sending result is a failure to send the consumption data or a timeout of the record response, then sleep for a preset sleep time; After the sleep mode ends, the steps of pulling consumption data from the preset source topic and performing grouping processing are executed again to obtain grouped data and record the successful data in the second relative position, where the successful data is the grouped data that is sent successfully.

3. The method according to claim 1, characterized in that After the step of recording the second relative position corresponding to the first relative position in the preset target subject, the following step further includes: Constructing a relative position mapping table through the first relative position and the second relative position; The first relative position and the second relative position are reset according to the relative position mapping table.

4. The method according to claim 1, characterized in that After the step of recording and responding to the sending result through the callback function to obtain the response result, the method includes: If the sending result is that all the packet data fails to be sent or the record response times out, the packet data is sent again.

5. The method according to claim 1, characterized in that The data migration method further includes: Receive a newly created synchronization task, wherein the synchronization task generates consumption data by reading and processing data; Scan the synchronization task regularly, and if the sending result is that the consumption data fails to be sent, resend the synchronization task; If the synchronization task waits for a timeout, the synchronization task is reissued; After the synchronization task is completed, the synchronization task is offline through the user's offline instruction.

6. The method according to claim 5, characterized in that The data migration method is executed by a query engine, the synchronization task is deployed on a first node and a second node, and the data migration method includes: receiving a synchronization process, a sending thread, and a process heartbeat registered by the first node, wherein the synchronization process, the sending thread, and the process heartbeat belong to the query engine; Sending the synchronization task to the second node or taking the synchronization task offline; If the first node fails to run due to an error, the second node executes the step of issuing the synchronization task or taking the synchronization task offline; If the second node fails to run due to an error, the registration synchronization process, sending thread and process heartbeat are executed through the first node.

7. A data migration device, characterized in that: The device comprises: A data pulling module, used to pull consumption data from a preset source topic and perform grouping processing to obtain grouped data, wherein the grouped data includes a first relative position of the consumption data in the preset source topic; A data sending module, used for asynchronously sending the grouped data to a preset target topic to obtain a sending result, wherein the grouped data includes at least two sub-data, and each time the sub-data is sent to the preset target topic separately at a preset interval; A sending record module is used to record and respond to the sending result through a callback function to obtain a response result; A relative position recording module, configured to record a second relative position corresponding to the first relative position in a preset target topic if the response result shows that the sending result is that the consumption data is sent successfully; A relative position submission module is used to submit the first relative position and the second relative position to a preset source topic.

8. A data migration device, characterized in that: The device comprises: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the data migration method according to any one of claims 1 to 6.

9. A storage medium, characterized in that: The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, the steps of the data migration method according to any one of claims 1 to 6 are implemented.

10. A computer program product, characterized in that The computer program product comprises a computer program, and when the computer program is executed by a processor, the steps of the data migration method according to any one of claims 1 to 6 are implemented.

Images