Data migration method and device, equipment and storage medium

Abstract

The invention relates to the technical field of data processing, can be applied to business system platforms of financial science and technology, medical health and the like, and discloses a data migration method, which comprises the following steps of: dividing a to-be-migrated data set in a source database into a preset number of data batches, and writing a batch label of each data batch into a preset task list file; and creating a corresponding record entry for each batch label in a preset progress record file, generating a loading task for each data batch according to the division sequence, sending the loading task to the target database, monitoring the loading state, and marking the state and timestamp of each stage in the progress record file. According to the method, the breakpoint is accurately identified by utilizing the task list file and the progress record file, the recovery mode is automatically entered, and automatic and intelligent data migration is realized.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and in particular to a data migration method, apparatus, device, and storage medium. Background Technology

[0002] With the evolution of big data architecture, enterprises need to migrate massive amounts of historical data from source systems (such as Druid) to target analytical databases (such as Doris) in scenarios such as data warehouse upgrades and storage system migrations. In real-world scenarios such as medical record management, financial transaction reconciliation, and e-commerce user behavior analysis, the datasets to be migrated are typically divided into dozens to hundreds of batches for sequential processing based on time spans, and the entire migration task may take several days or even weeks. Existing technologies generally employ centralized state management or rely on the target system's metadata for progress tracking, but these methods have the following problems: 1. When the migration process terminates abnormally due to system crashes, operational errors, or network interruptions, the progress information in memory is completely lost. Persistent state data may also fail to reflect the true completion status due to outdated updates or file corruption. After a restart, a full rerun is often necessary. For example, in the healthcare field, hospitals need to migrate years' worth of electronic medical record data in hundreds of batches each month. If the process is interrupted late at night due to network instability, the completed batches cannot be identified upon restarting the next day due to invalid state data, forcing a complete rerun. This not only wastes several days' worth of computing resources but also delays the launch of clinical research services.

[0003] 2. The lack of reliable records of sent and downloaded batches makes it impossible to accurately determine whether a batch has already been sent to the target database. This can easily lead to the same batch being sent repeatedly when the migration process restarts, generating a large number of invalid import tasks and potentially causing duplicate data writes. For example, in the fintech sector, financial institutions need to divide billions of historical transaction records into hundreds of batches per day for migration to support regulatory reporting. If the script is restarted repeatedly due to misoperation, the inability to identify sent batches can lead to duplicate imports of transaction data, resulting in serious errors such as doubling the summarization results.

[0004] In summary, existing technologies cannot achieve accurate resume transmission and avoid duplicate transmissions when the migration process is abnormally terminated. This problem affects the data accuracy requirements of medical, financial, and other industries. Therefore, there is an urgent need for a data migration mechanism that features self-healing migration status, accurate breakpoint resume transmission, and avoidance of duplicate transmissions. Summary of the Invention

[0005] In view of the above, it is necessary to provide a data migration method. The purpose of this method is to provide a precise breakpoint resumption and a data migration mechanism that avoids duplicate transmissions, addressing the data breakpoint problem that occurs during the data migration process.

[0006] Firstly, a data migration method is provided, including: The dataset to be migrated in the source database is divided into a preset number of data batches. According to the division order, a corresponding batch label is generated for each data batch. All batch labels are written into a preset task list file. A corresponding record entry is created for each batch label in a preset progress record file. In the record entry, the status of each batch label is marked as pending and the creation timestamp is recorded. According to the division order, load tasks are generated by loading data in each batch. The constructed migration process is called to send the loading tasks to the target database. The status of the batch tag corresponding to the loading task in the progress record file is marked as the execution status and the loading operation timestamp is recorded. Monitor the loading status of the loading tasks corresponding to the data of each batch tag in the progress record file in the target database. When it is confirmed that the data corresponding to the current loading task has been loaded into the target database, mark the status of the batch tag of the data corresponding to the current loading task as completed in the progress record file, and record the storage path and completion timestamp of the current data batch in the record entry.

[0007] Secondly, a data migration apparatus is provided, comprising: The partitioning module is used to divide the dataset to be migrated in the source database into a preset number of data batches. According to the partitioning order, a corresponding batch label is generated for each data batch. All batch labels are written into a preset task list file. A corresponding record entry is created for each batch label in a preset progress record file. In the record entry, the status of each batch label is marked as pending and the creation timestamp is recorded. The loading module is used to generate loading tasks by loading data in each batch according to the division order, call the constructed migration process to send the loading tasks to the target database, and mark the status of the batch tag corresponding to the loading task in the progress record file as the execution status and record the loading operation timestamp. The recording module is used to monitor the loading status of the loading tasks corresponding to the data of each batch tag in the progress record file in the target database. When it is confirmed that the data corresponding to the current loading task has been loaded into the target database, the status of the batch tag of the data corresponding to the current loading task is marked as completed in the progress record file, and the storage path and completion timestamp of the current data batch are recorded in the record entry.

[0008] Thirdly, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the above-described data migration method.

[0009] Fourthly, a computer-readable storage medium is provided, which stores a computer program that, when executed by a processor, implements the steps of the above-described data migration method.

[0010] Compared to existing technologies, this invention divides the dataset to be migrated in the source database into a preset number of data batches to avoid overload in a single operation. Each batch's batch label is written to a preset task list file, forming an immutable migration plan file. A corresponding record entry is created for each batch label in a preset progress log file. Even if the migration process crashes, the metadata of completed tasks is not lost, supporting reliable restart. Loading tasks are generated and sent to the preset target database in batch sequence, ensuring data dependencies. The status changes from pending to executing instantly upon task sending, utilizing file system atomicity to avoid duplicate submissions in concurrent scenarios, effectively preventing unnecessary pressure on the target database. The loading status of the target database is monitored, accurately synchronizing the migration data from the source database to the target database's local persistent storage. Recording the storage path achieves strong data-location binding, allowing for precise location during subsequent data cleanup, balancing, and troubleshooting, avoiding full table scans.

[0011] In the healthcare field, this invention ensures the automatic continuation of electronic medical records / medical imaging data after a hospital information system outage, preventing the loss of patient treatment data and meeting the requirement for full traceability of protected health information. In the fintech field, this invention guarantees the reliability of cross-center transaction migration, prevents duplicate entries, and ensures that the immutability of batch tags and timestamp auditing meet financial data security standards. Attached Figure Description

[0012] Figure 1 This is a schematic diagram of an application environment for a data migration method according to an embodiment of the present invention; Figure 2 This is a flowchart illustrating a data migration method provided in an embodiment of the present invention; Figure 3 This is a schematic diagram of a data migration device provided in an embodiment of the present invention; Figure 4 This is a schematic diagram of the structure of a computer device according to an embodiment of the present invention; Figure 5 This is another structural schematic diagram of a computer device according to one embodiment of the present invention.

[0013] The objectives, features, and advantages of this invention will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0014] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without inventive effort are within the scope of protection of this invention.

[0015] It should be noted that the descriptions involving "first," "second," etc., in this invention are for descriptive purposes only and should not be construed as indicating or implying their relative importance or implicitly specifying the number of technical features indicated. Therefore, a feature defined with "first" or "second" may explicitly or implicitly include at least one of that feature. Furthermore, the technical solutions of the various embodiments can be combined with each other, but this must be based on the ability of those skilled in the art to implement them. If the combination of technical solutions is contradictory or impossible to implement, it should be considered that such a combination of technical solutions does not exist and is not within the scope of protection claimed by this invention.

[0016] The data migration method provided in this embodiment of the invention can be applied to, for example, Figure 1 In this application environment, the client communicates with the server via a network. The server divides the dataset to be migrated in the source database into a preset number of data batches, generates corresponding batch labels for each data batch according to the division order, writes all batch labels into a preset task list file, and creates a corresponding record entry for each batch label in a preset progress record file. In the record entry, the status of each batch label is marked as pending and the creation timestamp is recorded. The server loads data from each batch according to the division order, generates a loading task, calls the constructed migration process to send the loading task to the target database, and marks the status of the batch tag corresponding to the loading task in the progress record file as the execution status and records the loading operation timestamp. The server monitors the loading status of the loading tasks corresponding to the data of each batch tag in the progress log file in the target database. When it is confirmed that the data corresponding to the current loading task has been loaded into the target database, the server marks the status of the batch tag of the data corresponding to the current loading task as completed in the progress log file, and records the storage path and completion timestamp of the current data batch in the record entry.

[0017] This invention targets fields such as financial enterprises, healthcare, market analysis, and engineering design. It utilizes task list files and progress log files to accurately identify breakpoints and automatically enters recovery mode for abnormal tags. Breakpoint identification and avoidance of duplicate sending are achieved without manual intervention, realizing automated and intelligent data migration. The client can be, but is not limited to, various personal computers, laptops, smartphones, tablets, and portable wearable devices. The server can be implemented using a standalone server or a server cluster consisting of multiple servers. The invention will be described in detail below through specific embodiments.

[0018] Reference Figure 2 The diagram shown is a flowchart illustrating a data migration method according to an embodiment of the present invention. This method is executed by a device.

[0019] In this embodiment, the data migration method includes: S10. Divide the dataset to be migrated in the source database into a preset number of data batches. Generate corresponding batch labels for each data batch according to the division order. Write all batch labels into a preset task list file. Create a corresponding record entry for each batch label in a preset progress record file. Mark the status of each batch label as pending and record the creation timestamp in the record entry.

[0020] In this embodiment, a migration request is received to migrate the dataset to be migrated from the source database (such as the Druid database) to the target database (such as the Doris database). A migration process is constructed based on the migration request, and the migration process is used to migrate the dataset to be migrated to the target database. The migration process can be deployed on a standalone server, the source database, a distributed microservice, or a cloud-native architecture.

[0021] Based on the load capacity of the source and target databases, the dataset to be migrated is split into N data batches using a balanced partitioning principle. The partitioning dimension can be selected as a primary key range (e.g., ID range), time window (e.g., date slice), or logical sharding (e.g., business type). The preset quantity can be dynamically calculated based on the number of rows in the source table of the dataset to be migrated and a recommended threshold for a single import, for example, controlling each batch to within 100 million rows or 10GB.

[0022] In the healthcare sector, for electronic medical record migration scenarios, data can be divided according to patient visit time windows, such as generating a preset number of data batches monthly or quarterly. Alternatively, in the fintech sector, for transaction record migration of core banking systems, a user ID hash grouping strategy can be used to evenly distribute hundreds of millions of accounts across a preset number of batches, ensuring that transaction records of the same user belong to the same batch to maintain data integrity.

[0023] Generate a globally unique batch label for each data batch that contains business semantics. The naming rule adopts a four-tuple concatenation format: [batch sequence number]_[partition method]_[business identifier]_[timestamp].

[0024] Create a plain text file named `.batch_labels` as the task list file. Each batch label should be on a separate line, arranged in ascending order of sequence number. Append a newline character to the end of the file to ensure completeness and readability. After generation, the task list file should be set to read-only access via operating system-level permission control, preventing it from participating in runtime state updates and avoiding accidental modification during runtime. In the healthcare field, the file can be named `hospital_migrate_20231114.batch_labels` and stored in a pre-defined path.

[0025] Construct a JSON-formatted .status file as the progress log file. Use batch label strings as keys in the progress log file, with each key corresponding to a structured object value. Iterate through the task list file, reading batch labels line by line, and create a record entry for each label in the progress log file.

[0026] Each batch label record entry contains three core fields: 1. Status field (e.g., initial value is assigned to pending status), 2. Creation time field (e.g., system standard timestamp when the record entry was created), 3. Data batch field (e.g., storage path template after the batch data is exported (e.g., / data / export / {label} / , which facilitates direct location in subsequent steps).

[0027] In one embodiment, the step of generating corresponding batch labels for each data batch according to the division order and writing all batch labels into a preset task list file includes: According to the predefined label naming rules, a batch label including a batch sequence number, data partitioning method, data partitioning timestamp, and data identifier is generated for each data batch. Create a blank document as a task list file, and set up a separate cell for each batch label in the task list file; After writing each batch label to the corresponding cell, set all batch labels to read-only.

[0028] Create a blank document as a task list file in the specified directory of the file system (e.g., / migration / tasks / ). Each batch label occupies a separate line, and the line number corresponds to its logical cell number. The first line of the file is reserved as a header line, and the batch label is written as a column identifier. Starting from the second line, the contents of the batch labels are written sequentially.

[0029] In ascending order of sequence number, each tag string is precisely written to its corresponding line position. After each tag is written, the file buffer is immediately flushed to ensure it is written to disk. After all tags have been written, a file integrity check is performed (e.g., to verify that the number of lines matches the batch quantity). If the check is successful, the write permission bit is revoked through the operating system's file permission control interface, retaining only read permission, thus putting the file into a read-only protected state. This operation is completed once after the task list file is generated and before the migration process starts to prevent accidental modification or overwriting during runtime.

[0030] In one embodiment, before creating a corresponding record entry for each batch label in a preset progress record file, the method further includes: Build a JSON document as a progress log file; Read the batch serial number of each batch tag from the task list file as the key of the corresponding batch tag; Use the object containing the status value of each batch label, the data storage path, and the data partition timestamp as the value of the corresponding batch label; Write the key and the value into the progress record file.

[0031] Create a blank JSON document as the progress log file (e.g., migration_status.json). The progress log file adopts a key-value pair hierarchical structure, with the top level being a set of keys consisting of batch label strings, and each key mapping to a standardized status object.

[0032] The batch label strings are read line by line from the task list file, and each label is used as a unique key in a JSON object. Simultaneously, a corresponding value object is constructed, which is structured data containing three attributes: status value, data storage path, and timestamp.

[0033] The writing strategy for the task list file adopts an atomic commit mechanism: first, a temporary file (such as migration_status.json.tmp) is generated in the target directory, all contents are written and forced to disk is executed, and finally, the temporary file is moved to the official progress record file name through an atomic rename operation, ensuring that even if the process crashes in the middle of writing, no partially written corrupted file will be generated.

[0034] In one embodiment, creating a corresponding record entry for each batch label in a preset progress record file includes: Retrieve the status field and creation time field of the corresponding batch tag from the key and value of each batch tag; Generate a JSON object based on the status field and creation time field of each batch label and use it as the record entry for the corresponding batch label.

[0035] Batch label strings are read line by line from the task list file, with each label serving as a unique primary key in the progress log file. A standardized value object is constructed for each batch label, containing two core fields: a status field and a create_time field. These two fields constitute the minimal dataset for the record entry, ensuring that each batch label has an independent status tracking entry in the progress log file.

[0036] A separate JSON object is generated for the status field and creation time field of each batch label. All batch label JSON objects are aggregated into a top-level JSON object, which is then written to the progress record file.

[0037] The core design of this invention, which includes a task list file and a progress log file, enables deduplication, resume interrupted downloads, and state persistence. The task list file is written once and read multiple times, storing only the complete set of batch tags. Its content remains unchanged throughout its lifecycle, providing a unique reference for deduplication checks and avoiding repeated queries to the source database metadata. The progress log file uses JSON format and is frequently updated, dynamically recording the real-time status of each tag (pending → executing → completed / failed). Its purpose is to support resume interrupted downloads and exception recovery.

[0038] The task list file and progress log file are linked by batch tag primary keys, forming a closed loop where static planning drives dynamic execution and dynamic execution feeds back to the static plan. This together supports the three core objectives of deduplication, resume download, and prevention of heavy load. If these two files are not designed separately, the list containing all tags must be rewritten for each status update, leading to a surge in IO and a sharp increase in the risk of concurrency conflicts.

[0039] S20. Loading tasks are generated by loading data in each batch according to the division order. The constructed migration process is called to send the loading tasks to the target database. The status of the batch tag corresponding to the loading task in the progress record file is marked as the execution status and the loading operation timestamp is recorded.

[0040] In this embodiment, the task list file is traversed line by line according to the division order. Starting from the first line, the string of batch label is read line by line. After reading each batch label, it is used as a unique identifier to locate the corresponding data batch range in the source database. For example, in the field of healthcare, the source database is usually a hospital information system (HIS) or an electronic medical record system (EMR). When locating data batches, it is necessary to query based on the patient's visit time window or medical record number range.

[0041] For each batch tag, a loading task consisting of data loading instructions is generated. The migration process is invoked to submit the loading instructions through the target database's API interface or command-line tool. Upon successful submission, the progress record file is updated immediately, such as changing the status field value of the corresponding batch tag entry from pending to executing, and adding a loading operation timestamp field to the entry.

[0042] The update operation follows the principle of atomicity: first, a complete JSON object is constructed in memory, written to a temporary file (.status.tmp), and forced to be written to disk; then, the original progress record file is overwritten through a rename operation, ensuring that even if the process is interrupted during the update, no historical state is lost or file corruption occurs. Once the update is complete, the resources for the current batch are released, and the processing loop for the next tag begins.

[0043] In one embodiment, generating a loading task by sequentially loading data from each data batch according to the partitioning order includes: Read the batch tags in the task list file according to the division order; Locate the corresponding data batch based on the read batch label; Generate loading tasks containing loading instructions for the location data in batches.

[0044] The loading instructions are encapsulated into loading tasks in sequence. After the loading tasks are generated, they are temporarily stored in the memory task pool, waiting for the migration process to submit the loading tasks to the target database for execution.

[0045] Data Loading Command: Generates an import command specific to the target database (such as Doris's Stream Load or Broker Load statement), specifying the data file path, batch label, target table name, and format options (column separator, row separator). In the healthcare field, the data loading command must point to Doris's medical subject database, with target table names such as ods_patient_records or dwd_medical_events, and column mappings set to handle HL7 nested structures. The command must include the `label = batch label` parameter to ensure that Doris can avoid importing duplicate medical record data from the same batch through its label deduplication mechanism.

[0046] In one embodiment, marking the status of the batch tag corresponding to the loading task in the progress record file as the execution status and recording the loading operation timestamp includes: Read the record entries corresponding to the batch tags in the progress record file; Modify the value of the status field of the record entry to an identifier representing the execution status; A new operation time field is added to the record entry, and the loading operation timestamp is generated based on the value of the operation time field.

[0047] Using the currently processed batch label string as the query key, perform an exact match search within the data structure to locate the record entry corresponding to that label. Within the record entry, locate the status field and overwrite its original value (usually "PENDING") with an identifier indicating the execution status (such as "EXECUTING"). Add an operation time field to the record entry.

[0048] S30. Monitor the loading status of the loading tasks corresponding to the data of each batch tag in the progress record file in the target database. When it is confirmed that the data corresponding to the current loading task has been loaded into the target database, mark the status of the batch tag of the data corresponding to the current loading task as completed in the progress record file, and record the storage path and completion timestamp of the current data batch in the record entry.

[0049] In this embodiment, for each submitted batch tag, an independent monitoring thread is initiated or a sequential polling strategy is adopted to send status query commands to the target database at preset time intervals (e.g., every 10 seconds). The batch tag must be used as an exact matching condition during the query to avoid returning irrelevant tasks. For example, in the healthcare field, the monitoring thread needs to additionally check the anonymization status of patient privacy data; the query command must include not only the batch tag but also a PHI (Protected Health Information) processing identifier for verification. Alternatively, in the fintech field, the monitoring strategy needs to meet the real-time requirements of financial regulation, employing high-frequency polling (e.g., every 3 seconds) and enabling a multi-level alarm mechanism.

[0050] In one embodiment, monitoring the loading status of the loading tasks corresponding to the data of each batch tag in the progress log file in the target database includes: Within a preset time interval, a query instruction is sent to the target database, using the batch tag as the query condition; Receive loading status information returned by the target database according to the query instruction, until a confirmation message indicating completion or failure is obtained.

[0051] Maintain a time wheel or timer in memory to trigger a query action at a preset time interval (e.g., 10 seconds). After each query command is sent, wait synchronously for the target database to return a response. Upon receiving a response, perform a final state judgment on the loading status information to obtain confirmation information of completion or failure.

[0052] In one embodiment, when it is confirmed that the data corresponding to the current loading task has been loaded into the target database, the batch tag status of the data corresponding to the current loading task is marked as completed in the progress record file, and the storage path and completion timestamp of the current data batch are recorded in the record entry, including: Receive confirmation information from the target database indicating that loading is complete; Based on the confirmation information, read the record entry corresponding to the batch label in the progress record file; Modify the value of the status field of the record entry to an identifier indicating the completion status; Add a storage path field and a completion time field to the record entry, and generate the completion timestamp based on the value of the completion time field.

[0053] Upon receiving confirmation from the target database, loading is considered complete when the returned status code is "Completed," the number of loaded rows equals the total number of rows, and the number of error rows is 0. If the returned status code is "Cancelled," "Failed," or the number of error rows exceeds the threshold, it is marked as failed. After confirming the loading task is complete, locate the record entry corresponding to the batch label in the progress log file and perform the following modifications: 1. Status Field Override: Update the status field value from "Executing" to "Completed". 2. Storage Path Supplement: Add a storage path field, specifying the actual storage path of the batch data files in the target database (e.g., Doris tablet path or HDFS directory). This path is derived from the return value of the load command. 3. Completion Timestamp Injection: Add a completion time field.

[0054] Implementing these three modifications transforms instantaneous execution results into persistent, traceable, and auditable data. When the migration script or process crashes and restarts, the progress log file can be read to identify completed batches and skip them directly, achieving "continue from the breakpoint" instead of "starting from scratch." If there is no successful marker, it will be impossible to distinguish between completed and incomplete tasks after a restart, leading to duplicate imports or data loss.

[0055] Recording the storage path establishes a physical mapping between the source file and the target location. When data anomalies are detected, it is possible to quickly trace which node and which shard the batch of data is stored in, facilitating accurate troubleshooting.

[0056] In step S30, monitoring the loading status of the target database for each batch of tags is the core technical means to achieve reliable migration. The loading of the target database is an asynchronous and time-consuming process (which may last from several seconds to several hours). The script returns immediately after the task is submitted. Without continuous monitoring, it is impossible to know when the data is actually written to disk or whether it has been successfully written. The monitoring mechanism feeds back the execution results of the external database to the local status file, forming a complete chain of "submission → execution → confirmation". This ensures that the local progress is strictly consistent with the actual remote status, avoiding the data inconsistency disaster of "I thought it succeeded, but it actually failed".

[0057] S40. When the migration process is detected to have terminated or restarted abnormally, read the latest status of each batch tag in the progress record file and the timestamp corresponding to the latest status. Batch tags with the latest status being "complete" will be skipped and not processed. Batch labels whose latest status is "Execution" and whose timestamps are more than a preset threshold from the current time are marked as abnormal labels.

[0058] In this embodiment, when the migration process is interrupted by system crashes, network outages, misoperations, or restarts, the first step is to clean up the frozen temporary files while preserving the integrity of the original task list file and progress log file. In the healthcare field, migration process anomalies may occur during the migration of electronic medical records or medical imaging data; such interruptions can affect the continuity of hospital clinical operations. After a restart, in addition to cleaning up temporary files, the digital signature of the task list file must be verified to prevent patient data from being tampered with during system crashes.

[0059] In one embodiment, reading the latest status of each batch tag in the progress record file and the timestamp corresponding to the latest status includes: Parse the record entries corresponding to each batch label stored in JSON format in the progress record file; Extract the value of the status field of the last record in the record entry as the latest status, and extract the value of the time field of the record entry for the latest status as the timestamp corresponding to the latest status.

[0060] The latest status refers to the enumeration value currently stored in the status field of the record entry corresponding to the batch label as the key in the progress record file. This enumeration value reflects the execution stage of the data batch corresponding to the batch label after the most recent status update operation in the migration process.

[0061] The process loads a JSON-formatted progress log file, deserializes it into a key-value pair mapping structure, iterates through the mapping structure, and parses the latest status field (status field, timestamp field corresponding to the latest status, and storage path field) and the timestamp corresponding to the latest status for each batch tag key. The parsed latest status field and its corresponding timestamp are then used to construct a snapshot of the actual execution status. In the healthcare field, when loading the task list file, patient privacy level screening is required to automatically identify highly sensitive batch tags containing protected health information and add protected field identifiers to the status snapshot. When parsing the progress log file, the data lineage information for each batch must be extracted to ensure a complete and traceable mapping relationship from the HIS system to Doris. The timestamp corresponding to the latest status must be synchronized with the hospital's standard NTP server to ensure time consistency for multi-hospital migration tasks and support the reliability of subsequent clinical data analysis.

[0062] For each batch tag, query the status snapshot. If the status value is "complete", no subsequent operation is performed on that batch tag, thus ensuring that completed batches are never executed again, which realizes the core skip retry logic of resuming interrupted downloads.

[0063] In one embodiment, marking batches whose latest status is "execution status" and whose timestamps are more than a preset threshold from the current time as "abnormal" tags includes: For batch tags whose latest status is execution status, calculate the time difference between the current timestamp and the timestamp corresponding to the latest status. When the time difference is greater than the preset threshold, the batch label is marked as the abnormal label; When the time difference is less than or equal to the preset threshold, the loading task corresponding to the abnormal label is retried using the migration process.

[0064] For batch tags marked as being in execution status, calculate the time difference between the current system timestamp and the timestamp of the loading operation in the record entry. If the time difference is greater than a preset timeout threshold (e.g., 2 hours), the batch tag is determined to be dead due to the migration process interruption and is marked as an abnormal tag. Batch tags in the execution status that have not timed out (e.g., in cases with extremely short restart intervals) are not processed temporarily and await the next round of scheduling by the migration process to take over again.

[0065] S50. Reset the status of the exception label to the pending status; use the exception label to query the task list file to obtain the data batch that failed to migrate, and re-execute the loading task for the data batch that failed to migrate.

[0066] In this embodiment, a status rollback operation is performed for each exception label. The record entry corresponding to the exception label is located in the JSON structure of the progress log file, and the following modifications are performed: 1. Status field write-back: Forcefully reset the status field value from the execution state to the pending state, and clear the zombie state indicator.

[0067] 2. Timestamp cleanup: Delete the loading operation timestamp field and its value to avoid old timestamps interfering with subsequent timeout calculations.

[0068] 3. Add an exception flag: Add a retry field to record the number of times this batch has been retried (set to 1 for the first retry).

[0069] Using the exception label string as the search key, an exact match query is performed in the read-only task manifest file to obtain the batches of data that failed to migrate. The loading task is then re-executed for these batches, retrieving the export and load instruction templates from the previous execution of the batch from logs or cache, preserving the original parameter configuration. During retries, the original batch labels must be reused; new labels cannot be generated. This is key to utilizing the Doris Label deduplication mechanism—even if a submission is made again, the database will recognize that the label already exists and the migration failed, allowing for overwriting or generating a new file.

[0070] This invention uses the task list file as a static blueprint to handle plan management: it stores only the complete set of batch tags, and sets it to read-only after each write to avoid frequent I / O. Its row-based structure supports sequential streaming reads, has extremely low memory usage, and is suitable for scales of tens of millions of tags.

[0071] The progress log file serves as a dynamic log, responsible for status tracking: frequently updating the real-time status of each batch. The JSON key-value structure supports random access, allowing direct status location via batch tags without traversing the entire dataset.

[0072] If the task list file and progress log file are merged into one file, a massive number of static tags need to be rewritten every time the status is updated, resulting in a surge in IO and a sharp increase in the risk of concurrency conflicts. After separation, the progress file is usually less than 100MB, and the atomic writing overhead is controllable, thus forming a dual guarantee mechanism with the task list file and progress log file.

[0073] In the field of healthcare, this invention ensures that electronic medical records / medical imaging data are automatically resumed after a hospital information system crashes, preventing the loss of patient treatment data and meeting the requirement for full traceability of protected health information.

[0074] In the field of fintech, this invention ensures the reliability of cross-center migration of transaction flows, prevents duplicate accounting, and ensures that the immutability of batch tags and timestamp auditing meet the requirements of financial data security standards.

[0075] like Figure 3 The diagram shown is a schematic diagram of a data migration device provided in an embodiment of the present invention.

[0076] The data migration device 100 of this invention can be installed in a device. Depending on the functions implemented, the data migration device 100 may include a partitioning module 110, a loading module 120, and a recording module 130. The module described in this invention can also be called a unit, which refers to a series of computer program segments that can be executed by the device processor and perform a fixed function, and which are stored in the device's memory.

[0077] In this embodiment, the functions of each module / unit are as follows: The partitioning module 110 is used to partition the dataset to be migrated in the source database into a preset number of data batches, generate corresponding batch labels for each data batch according to the partitioning order, write all batch labels into a preset task list file, create a corresponding record entry for each batch label in a preset progress record file, mark the status of each batch label as pending processing in the record entry and record the creation timestamp. The loading module 120 is used to generate a loading task by loading the data of each data batch according to the division order, call the constructed migration process to send the loading task to the target database, and mark the status of the batch tag corresponding to the loading task in the progress record file as the execution status and record the loading operation timestamp. The recording module 130 is used to monitor the loading status of the loading tasks corresponding to the data of each batch tag in the progress recording file in the target database. When it is confirmed that the data corresponding to the current loading task has been loaded into the target database, the status of the batch tag of the data corresponding to the current loading task is marked as completed in the progress recording file, and the storage path and completion timestamp of the current data batch are recorded in the record entry.

[0078] In one embodiment, the partitioning module 110 is specifically used for: According to the predefined label naming rules, a batch label including a batch sequence number, data partitioning method, data partitioning timestamp, and data identifier is generated for each data batch. Create a blank document as a task list file, and set up a separate cell for each batch label in the task list file; After writing each batch label to the corresponding cell, set all batch labels to read-only.

[0079] In one embodiment, the partitioning module 110 is specifically used for: Build a JSON document as a progress log file; Read the batch serial number of each batch tag from the task list file as the key of the corresponding batch tag; Use the object containing the status value of each batch label, the data storage path, and the data partition timestamp as the value of the corresponding batch label; Write the key and the value into the progress record file.

[0080] In one embodiment, the partitioning module 110 is specifically used for: Retrieve the status field and creation time field of the corresponding batch tag from the key and value of each batch tag; Generate a JSON object based on the status field and creation time field of each batch label and use it as the record entry for the corresponding batch label.

[0081] In one embodiment, the loading module 120 is specifically used for: Read the batch tags in the task list file according to the division order; Locate the corresponding data batch based on the read batch label; Generate a loading task containing data loading instructions for the location data batch.

[0082] In one embodiment, the recording module 130 is specifically used for: Receive confirmation information from the target database indicating that loading is complete; Based on the confirmation information, read the record entry corresponding to the batch label in the progress record file; Modify the value of the status field of the record entry to an identifier indicating the completion status; Add a storage path field and a completion time field to the record entry, and generate the completion timestamp based on the value of the completion time field.

[0083] In one embodiment, the recording module 130 is specifically used for: When the migration process is detected to have terminated or restarted abnormally, the latest status of each batch tag and the timestamp corresponding to the latest status are read from the progress record file. Batch tags with the latest status being "complete" will be skipped and not processed. Batch labels whose latest status is "Execution" and whose timestamps are more than a preset threshold from the current time are marked as abnormal labels.

[0084] In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 4As shown, the computer device includes a processor, memory, network interface, and database connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile and / or volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and database. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The network interface is used to communicate with external clients via a network connection. When the computer program is executed by the processor, it implements the functions or steps of a data migration method on the server side.

[0085] In one embodiment, a computer device is provided, which may be a client, and its internal structure diagram may be as follows: Figure 5 As shown, the computer device includes a processor, memory, network interface, display screen, and input devices connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The network interface is used to communicate with an external server via a network connection. When the computer program is executed by the processor, it implements the functions or steps of a data migration method on the client side.

[0086] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to perform the following steps: The dataset to be migrated in the source database is divided into a preset number of data batches. According to the division order, a corresponding batch label is generated for each data batch. All batch labels are written into a preset task list file. A corresponding record entry is created for each batch label in a preset progress record file. In the record entry, the status of each batch label is marked as pending and the creation timestamp is recorded. According to the division order, load tasks are generated by loading data in each batch. The constructed migration process is called to send the loading tasks to the target database. The status of the batch tag corresponding to the loading task in the progress record file is marked as the execution status and the loading operation timestamp is recorded. Monitor the loading status of the loading tasks corresponding to the data of each batch tag in the progress record file in the target database. When it is confirmed that the data corresponding to the current loading task has been loaded into the target database, mark the status of the batch tag of the data corresponding to the current loading task as completed in the progress record file, and record the storage path and completion timestamp of the current data batch in the record entry.

[0087] It should be noted that the functions or steps that can be implemented by the computer-readable storage medium or computer device described above can be referred to the relevant descriptions on the server side and client side in the foregoing method embodiments. To avoid repetition, they will not be described one by one here.

[0088] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0089] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is used as an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0090] The above-described embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit them; if any software tools or components other than those of our company appear in the embodiments of the application, they are merely for illustrative purposes and do not represent actual use; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included within the protection scope of the present invention.

Claims

1. A data migration method, characterized in that, include: The dataset to be migrated in the source database is divided into a preset number of data batches. According to the division order, a corresponding batch label is generated for each data batch. All batch labels are written into a preset task list file. A corresponding record entry is created for each batch label in a preset progress record file. In the record entry, the status of each batch label is marked as pending and the creation timestamp is recorded. According to the division order, load tasks are generated by loading data in each batch of data. The constructed migration process is called to send the loading tasks to the target database. The status of the batch tag corresponding to the loading task in the progress record file is marked as the execution status and the loading operation timestamp is recorded. Monitor the loading status of the loading tasks corresponding to the data of each batch tag in the progress record file in the target database. When it is confirmed that the data corresponding to the current loading task has been loaded into the target database, mark the status of the batch tag of the data corresponding to the current loading task as completed in the progress record file, and record the storage path and completion timestamp of the current data batch in the record entry.

2. The data migration method as described in claim 1, characterized in that, The process of generating corresponding batch labels for each data batch according to the division order and writing all batch labels into a preset task list file includes: According to the predefined label naming rules, a batch label including a batch sequence number, data partitioning method, data partitioning timestamp, and data identifier is generated for each data batch. Create a blank document as a task list file, and set up a separate cell for each batch label in the task list file; After writing each batch label to the corresponding cell, set all batch labels to read-only.

3. The data migration method as described in claim 1, characterized in that, Before creating a corresponding record entry for each batch label in the preset progress record file, the method further includes: Build a JSON document as a progress log file; Read the batch serial number of each batch tag from the task list file as the key of the corresponding batch tag; Use the object containing the status value of each batch label, the data storage path, and the data partition timestamp as the value of the corresponding batch label; Write the key and the value into the progress record file.

4. The data migration method as described in claim 1, characterized in that, The step of creating a corresponding record entry for each batch label in a preset progress record file includes: Retrieve the status field and creation time field of the corresponding batch tag from the key and value of each batch tag; Generate a JSON object based on the status field and creation time field of each batch label and use it as the record entry for the corresponding batch label.

5. The data migration method as described in claim 1, characterized in that, The step of generating a loading task by sequentially loading data from each data batch according to the division order includes: Read the batch tags in the task list file according to the division order; Locate the corresponding data batch based on the read batch label; Generate a loading task containing data loading instructions for the location data batch.

6. The data migration method as described in claim 1, characterized in that, When it is confirmed that the data corresponding to the current loading task has been loaded into the target database, the batch tag status of the data corresponding to the current loading task is marked as completed in the progress record file, and the storage path and completion timestamp of the current data batch are recorded in the record entry, including: Receive confirmation information from the target database indicating that loading is complete; Based on the confirmation information, read the record entry corresponding to the batch label in the progress record file; Modify the value of the status field of the record entry to an identifier indicating the completion status; Add a storage path field and a completion time field to the record entry, and generate the completion timestamp based on the value of the completion time field.

7. The data migration method as described in claim 1, characterized in that, After the record entry records the storage path and completion timestamp of the current data batch, the method further includes: When the migration process is detected to have terminated or restarted abnormally, the latest status of each batch tag and the timestamp corresponding to the latest status are read from the progress record file. Batch tags with the latest status being "complete" will be skipped and not processed. Batch labels whose latest status is "Execution" and whose timestamps are more than a preset threshold from the current time are marked as abnormal labels.

8. A data migration device, characterized in that, The device includes: The partitioning module is used to divide the dataset to be migrated in the source database into a preset number of data batches. According to the partitioning order, a corresponding batch label is generated for each data batch. All batch labels are written into a preset task list file. A corresponding record entry is created for each batch label in a preset progress record file. In the record entry, the status of each batch label is marked as pending and the creation timestamp is recorded. The loading module is used to generate loading tasks by loading data in each batch according to the division order, call the constructed migration process to send the loading tasks to the target database, and mark the status of the batch tag corresponding to the loading task in the progress record file as the execution status and record the loading operation timestamp. The recording module is used to monitor the loading status of the loading tasks corresponding to the data of each batch tag in the progress record file in the target database. When it is confirmed that the data corresponding to the current loading task has been loaded into the target database, the status of the batch tag of the data corresponding to the current loading task is marked as completed in the progress record file, and the storage path and completion timestamp of the current data batch are recorded in the record entry.

9. A device, characterized in that, The device includes: At least one processor; and, A memory communicatively connected to the at least one processor; wherein, The memory stores a data migration program that can be executed by the at least one processor, the data migration program being executed by the at least one processor to enable the at least one processor to perform the data migration method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a data migration program, which can be executed by one or more processors to implement the data migration method as described in any one of claims 1 to 7.

Images