Data migration platform, data migration method, equipment and storage medium

Abstract

The invention discloses a data migration platform, a data migration method, equipment and a storage medium, and relates to the technical field of data migration, and the data migration platform comprises a data source module used for obtaining to-be-migrated data and obtaining a migration configuration information base corresponding to the to-be-migrated data; the data migration module is used for dividing the to-be-migrated data into a plurality of to-be-migrated files based on migration configuration information in the migration configuration information base and generating migration tasks corresponding to the to-be-migrated files so as to migrate the to-be-migrated files to a target database through multi-instance scheduling processing; and the data verification module is used for carrying out data verification on the migration data in the target database to obtain a data migration verification result. According to the method, multiple migration tasks are divided, and multi-instance scheduling processing is combined, so that migration of multiple files can be processed in parallel at the same time, the migration time under a large data volume scene is greatly shortened, and the problems that a traditional method is long in backup time and insufficient in concurrency capability are solved.

Description

Technical Field

[0001] This application relates to the field of data migration technology, and in particular to a data migration platform, data migration method, device and storage medium. Background Technology

[0002] Currently, the industry mainly relies on traditional database backup technology for data migration between databases. This technology usually takes full table backup as the core method and uses scripts or basic tools to perform data reading, transmission and storage operations. Some solutions will combine simple task scheduling mechanisms to try to improve migration efficiency. At the same time, after the data migration is completed, basic field non-null checks are used to confirm data integrity in order to meet basic migration requirements.

[0003] However, due to the excessively long backup and migration time, especially when the data volume is large, the full table processing mode results in long single-task processing time and weak concurrency capabilities, making it impossible to handle the migration needs of multiple target databases at the same time, which seriously affects the data transmission efficiency. Therefore, the existing technology faces many key defects in practical applications and is difficult to adapt to large-scale, highly flexible data migration scenarios between databases. Summary of the Invention

[0004] The main purpose of this application is to provide a data migration platform, data migration method, device and storage medium, which aims to solve the problems of long backup time and insufficient concurrency of traditional methods.

[0005] To achieve the above objectives, this application proposes a data migration platform, which includes a data source module, a data migration module, and a data verification module. The data source module is used to obtain the data to be migrated and to obtain the migration configuration information database corresponding to the data to be migrated. The data migration module is used to divide the data to be migrated into several files to be migrated based on the migration configuration information in the migration configuration information database, and generate a migration task corresponding to each file to be migrated, so as to migrate the files to be migrated to the target database through multi-instance scheduling. The data verification module is used to verify the migration data in the target database and obtain the data migration verification result.

[0006] In one embodiment, the data migration platform further includes a configuration management module, which is used to receive data migration rules, workflow configurations and custom verification rules input by users through a visual interface and / or configuration files, and generate a migration configuration information database.

[0007] In one embodiment, the configuration management module is further configured to store the migration tasks and their corresponding dependencies in a tree structure model.

[0008] In one embodiment, the data migration platform further includes a data processing module, which is used to obtain raw migration data from the source database and perform cleaning and transformation operations on the raw migration data according to the migration configuration information database to obtain the data to be migrated.

[0009] In one embodiment, the data migration platform further includes a monitoring and alarm module, which is used to capture the task execution status and execution logs of the data source module, the data verification module, and the data migration module in real time, and generate a real-time migration progress report in combination with the alarm rules in the configuration management module. The monitoring and alarm module is also used to generate an abnormal alarm and push it to the designated person in charge when an abnormality is detected in the execution of the migration task.

[0010] To achieve the above objectives, this application proposes a data migration method, comprising: Obtain the data to be migrated, and obtain the migration configuration information database corresponding to the data to be migrated; Based on the migration configuration information in the migration configuration information database, the data to be migrated is divided into several files to be migrated, and a migration task corresponding to each file to be migrated is generated, so as to migrate the files to be migrated to the target database through multi-instance scheduling. Data verification is performed on the migration data in the target database to obtain the data migration verification result.

[0011] In one embodiment, the step of dividing the data to be migrated into several files to be migrated based on the migration configuration information in the migration configuration information database, and generating migration tasks corresponding to each file to be migrated, so as to migrate the files to be migrated to the target database through multi-instance scheduling processing, includes: Based on the batch size threshold of the migration configuration information, the data to be migrated is divided into several files to be migrated; Generate migration tasks corresponding to each of the files to be migrated, forming a task queue; Establish communication connections between multiple instances and the task queue, and migrate the files to be migrated to the target database through multi-instance scheduling processing according to each migration task.

[0012] In one embodiment, migrating the files to be migrated to the target database through multi-instance scheduling processing according to each migration task includes: Obtain the task priority of the task queue; Generate a migration sort corresponding to each migration task according to the task priority; For any of the migration tasks, based on the migration sorting and the migration database connection information corresponding to the migration task, the files to be migrated corresponding to the migration task are migrated to the target database through multi-instance scheduling.

[0013] In addition, to achieve the above objectives, this application also proposes a data migration device, the device comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the data migration method as described above.

[0014] In addition, to achieve the above objectives, this application also proposes a storage medium, which is a computer-readable storage medium, on which a computer program is stored, and which, when executed by a processor, implements the steps of the data migration method described above.

[0015] In addition, to achieve the above objectives, this application also provides a computer program product, which includes a computer program that, when executed by a processor, implements the steps of the data migration method described above.

[0016] This application provides a data migration platform, method, device, and storage medium. The data migration platform includes: a data source module for acquiring data to be migrated and a migration configuration information database corresponding to the data to be migrated; a data migration module for dividing the data to be migrated into several files to be migrated based on the migration configuration information in the migration configuration information database, and generating migration tasks corresponding to each file to be migrated, so as to migrate the files to be migrated to the target database through multi-instance scheduling; and a data verification module for verifying the migrated data in the target database and obtaining data migration verification results. This allows for flexible definition of the range of data to be migrated, the target database address, etc., according to the configuration, solving the limitation of traditional methods that require fixed full table migration, adapting to diverse data migration needs between MySQL databases, and simultaneously processing the migration of multiple files (such as migrating data from multiple instances to different target MySQL online databases), significantly shortening the migration time in scenarios with large amounts of data, solving the problems of long backup time and insufficient concurrency in traditional methods, and further verifying the migrated data in the target database after the migration is completed, avoiding the problem of insufficient verification in traditional methods leading to erroneous data being entered into the database, and ensuring the consistency and accuracy of the migrated data with the source data. The entire process relies on the migration configuration information database to achieve standardized operations. From data partitioning and task generation to multi-instance scheduling, everything is executed automatically according to the configuration, reducing manual intervention. Attached Figure Description

[0017] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0018] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 The system architecture diagram provided for the data migration platform of this application; Figure 2 This is a flowchart illustrating an embodiment of the data migration method of this application. Figure 3 A simplified flowchart illustrating the data migration method described in this application; Figure 4 This is a schematic diagram of the device structure of the hardware operating environment involved in the data migration method in the embodiments of this application.

[0020] The purpose, features, and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0021] It should be understood that the specific embodiments described herein are merely illustrative of the technical solutions of this application and are not intended to limit this application.

[0022] Existing technologies face several key shortcomings in practical applications of inter-database data migration, making them unsuitable for large-scale, highly flexible inter-database data migration scenarios: First, backup and migration times are too long, especially with large data volumes. The full-table processing mode results in long single-task processing times and weak concurrency capabilities, making it impossible to handle the migration needs of multiple target databases simultaneously, severely impacting data transmission efficiency. Second, migration flexibility is insufficient. Traditional solutions are mostly fixed processes, unable to flexibly select the migration data range according to business needs, and it is difficult to adjust migration rules and workflows through configuration, requiring a large amount of manual development and intervention, resulting in poor scalability. Third, data validation is insufficient, only basic format validation can be performed, lacking a custom validation mechanism, making it impossible to identify abnormal data in a timely manner, resulting in difficulty in guaranteeing the accuracy of data after migration. Fourth, operation and maintenance and monitoring capabilities are weak, unable to monitor migration progress in real time, with delayed detection of abnormal nodes, and the failure of a single instance can easily lead to the interruption of the entire migration, resulting in insufficient stability and maintainability.

[0023] To better understand the technical solution of this application, a detailed description will be provided below in conjunction with the accompanying drawings and specific implementation methods.

[0024] It should be noted that the executing entity in this embodiment can be a computing service device with data processing, network communication, and program execution functions, such as a tablet computer, personal computer, or mobile phone, or an electronic device, big data service platform, or data migration system capable of performing the above functions. The following description uses a data migration system as an example to illustrate this embodiment and the subsequent embodiments.

[0025] Based on this, embodiments of this application provide a data migration platform, which includes a data source module, a data migration module, and a data verification module, as described above. Figure 1 , Figure 1 The system architecture diagram provided for the data migration platform of this application.

[0026] The data source module is used to obtain the data to be migrated and to obtain the migration configuration information database corresponding to the data to be migrated. It should be noted that the data source module refers to the functional module in the data migration platform responsible for reading initial data from the source database and associating it with the migration configuration basis; the data to be migrated refers to standardized data that has been cleaned and transformed by the data processing module and meets the basic migration format requirements, rather than the original data in the source database; the migration configuration information library refers to a standardized information collection generated by the configuration management module that stores the rules for the entire data migration process, including data migration rules (such as the source and target database addresses, table names and fields to be migrated, and data filtering conditions), workflow configurations (such as task execution order and task dependencies), batch size thresholds (such as the upper limit of single file data volume), and custom validation rules (such as field value range and data format requirements).

[0027] It should be further explained that the core purpose of this module is to accurately acquire the data to be migrated that meets the migration requirements, while also taking over the rules in the migration configuration information database. This avoids a chaotic migration process due to unclear data sources or missing rules, and solves the problem of "no standard for data reading and reliance on manual rule transmission" in traditional technologies. In one possible implementation, the data source module can support a timed reading mechanism. Based on the "data reading cycle" in the migration configuration information database (e.g., 2 AM daily), it automatically reads the latest data to be migrated from the source database and synchronously updates the migration configuration information database (if the user adjusts the migration rules), achieving dynamic synchronization between data and rules.

[0028] Specifically, the data source module first establishes a communication connection with the data processing module, receiving the data to be migrated output by the data processing module. This data is standardized data after the data processing module has cleaned (removed garbled characters and null values) and transformed (standardized field formats) the original data in the source database, and has been filtered according to the "data filtering conditions" (such as "creation time > 2025-01-01") in the migration configuration information database. Subsequently, the data source module calls the configuration management module through an interface to obtain the migration configuration information database corresponding to the data to be migrated. For example, if the data to be migrated comes from the "user table of the mysql005 database", then obtain the migration rules (such as the target database being MySQL online database A) and workflow configuration (such as "trigger file partitioning task after data reading is completed") preset for the "user table of the mysql005 database" from the configuration management module.

[0029] Furthermore, after the acquisition is completed, the data source module performs integrity verification on the received data to be migrated (such as checking whether the key field "User ID" is missing) and format verification on the migration configuration information database (such as checking whether the target database address contains IP and port) to ensure that both meet the subsequent migration requirements. Then, the data to be migrated is associated with the migration configuration information database and stored for the data migration module to call.

[0030] The data migration module is used to divide the data to be migrated into several files to be migrated based on the migration configuration information in the migration configuration information database, and generate a migration task corresponding to each file to be migrated, so as to migrate the files to be migrated to the target database through multi-instance scheduling. It should be noted that the data migration module refers to the core functional module in the data migration platform responsible for file partitioning, task generation, and multi-instance scheduling; a migration task refers to a task unit containing execution information corresponding to a single file to be migrated. Each task includes a task ID, the storage path of the file to be migrated, the target database identifier, the task priority, and the timeout (e.g., 30 minutes); multi-instance scheduling refers to the process of starting multiple migration instance programs, obtaining migration tasks through a competition mechanism, and importing the files to be migrated into the target database in parallel. The number of instances can be dynamically adjusted according to the limits in the migration configuration information database; the target database refers to the MySQL database that receives the data to be migrated, including MySQL online databases and other databases that carry daily business operations, as well as historical databases, backup databases, and other non-real-time business databases.

[0031] It should be further explained that the core purpose of this module is to solve the problems of "long backup time and insufficient concurrency" in traditional technologies by "batch splitting + multi-instance concurrency": batch splitting reduces the data volume of a single file and shortens the processing time of a single task; multi-instance scheduling enables parallel migration of multiple target databases, improving overall migration efficiency. At the same time, relying on the migration configuration information database ensures that file partitioning, task generation, and instance scheduling are all based on rules, avoiding inefficiencies caused by manual intervention. In one possible implementation, the data migration module can support dynamic adjustment of batch size—if the "target database IO load" field in the migration configuration information database shows that the current load is too high (e.g., exceeding 80%), the batch size threshold is automatically reduced by 50% (e.g., from 1000 records / file to 500 records / file), reducing the pressure on the target database when importing a single file.

[0032] Specifically, in one embodiment, the data migration module first obtains the data to be migrated and the corresponding migration configuration information database from the data source module, and performs the following three steps: The first step is to divide the files to be migrated. Read the "batch size threshold" (e.g., 1000 records / file) from the migration configuration information database and split the continuous data to be migrated: if the total amount of data to be migrated is 8500 records, it will be split into 9 files to be migrated. The first 8 files each contain 1000 records, and the 9th file contains 500 records. All files are named in the format of "source database identifier-table name-batch number" (e.g., "mysql003-order table-001" to "mysql003-order table-009") and stored in the specified path on the server (e.g., " / data / migration / files / ").

[0033] The second step is to generate migration tasks. For each file to be migrated, a corresponding migration task is automatically generated: the task ID is generated in the format of "Task-Date-Sequence Number" (e.g., "Task-20250520-001"), the storage path of the file to be migrated is the server path mentioned above, the target database identifier is taken from the migration configuration information database (e.g., "MySQL Online Database A"), the task priority is assigned according to the configuration rules (e.g., "order table data has high priority"), the timeout is set to 30 minutes, and all tasks are summarized to form a task queue, sorted from high to low priority.

[0034] The third step is multi-instance scheduling and migration. Based on the "multi-instance quantity limit" in the migration configuration information database (e.g., if the server has 8 CPU cores, start 12 instances), start the corresponding number of migration instance programs and establish communication connections between the instances and the task queue. When the instance is in an "idle" state, it automatically competes to obtain tasks from the task queue (tasks with higher priority are assigned first). The instance that obtains the task establishes a connection with the target MySQL database according to the target database identifier in the task (including account and password authentication, data transmission protocol configuration), reads the data to be migrated, and imports it into the target data table. During the import process, the instance synchronizes the execution status (e.g., "importing", "success", "failure") to the monitoring and alarm module in real time.

[0035] The data verification module is used to verify the migration data in the target database and obtain the data migration verification result.

[0036] It should be noted that the data verification module refers to the functional module in the data migration platform responsible for verifying the accuracy of the migrated data in the target database; the migrated data refers to the data to be migrated that has been imported into the target database, that is, the data stored in the target database data table after the files to be migrated have been scheduled through multiple instances; data verification refers to the process of comparing the consistency and compliance of the migrated data with the source data (or standard data) according to preset rules, and identifying abnormal data; the data migration verification result refers to the structured report output by the data verification module that contains the verification conclusion, including information such as "amount of data that passed verification", "amount of data that failed verification", "list of abnormal data (including abnormal data ID and reason for abnormality)" and "verification pass rate".

[0037] It should be further explained that the core purpose of this module is to solve the problem of "insufficient data validation" in traditional technologies. Through rigorous validation, it ensures the accuracy of data migrated to the target database, preventing abnormal data (such as format errors, numerical anomalies, or missing data) from affecting the operation of the business systems hosted by the target database. Simultaneously, it generates validation results for users to trace anomalies. In one possible implementation, the data validation module can support "two-way validation": on the one hand, it verifies whether the quantity of migrated data in the target database matches the quantity of data to be migrated in the source database (e.g., if the source database has 5800 records, the target database should also have 5800 records); on the other hand, it verifies whether the content of the migrated data matches the source data (e.g., for a data entry "User ID=1001, Amount=200", the corresponding data in the target database should be exactly the same), further improving the rigor of the validation.

[0038] Specifically, in one embodiment, the data verification module first reads migration data from the target database—based on the "target database address" and "table name" in the migration configuration information base, it connects to the target database (e.g., MySQL online database A), reads all migration data from the corresponding tables, and stores them sorted by "data ID"; simultaneously, it obtains the data to be migrated from the data source module (as the verification benchmark, i.e., the source data benchmark), also sorted by "data ID". Subsequently, the data verification module performs verification according to the "custom verification rules" in the migration configuration information base. The core verification dimensions include: First, quantity verification: check whether the total amount of data migrated from the target database is consistent with the total amount of data in the source database. If they are inconsistent, mark it as "data missing or duplicate". Second, format validation checks whether the format of each field in the migration data conforms to the rules (such as whether "creation time" is in YYYY-MM-DD format and whether "amount" is numeric). If it does not conform, it is marked as "format abnormal". Third, numerical verification checks whether the numerical fields in the migration data are within the preset range (such as "amount > 0" or "age 18-60 years old"). If they are outside the range, they are marked as "numerical abnormal". Fourth, content consistency verification: compare the field content of the migrated data with the source data benchmark one by one according to the "data ID". If the value of a certain field is different (e.g., source data "amount = 200", migrated data "amount = 20"), then mark "content inconsistent".

[0039] After verification, the data verification module counts the number of various abnormal data and generates a structured data migration verification result containing "number of data that passed verification (e.g., 5798 records)," "number of data that failed verification (e.g., 2 records)," "list of abnormal data (e.g., data ID=1005: format abnormal; data ID=2010: numerical abnormal)" and "verification pass rate (e.g., 99.97%)." This result is synchronized to the monitoring and alarm module. If the verification pass rate is lower than the "preset compliance threshold (e.g., 99.9%)" in the migration configuration information database, an abnormal alarm is triggered.

[0040] This embodiment uses a data source module to acquire the data to be migrated and the corresponding migration configuration information database. A data migration module, based on the migration configuration information in the database, divides the data to be migrated into several files and generates migration tasks corresponding to each file. These tasks are then used to migrate the files to the target database via multi-instance scheduling. A data verification module verifies the migrated data in the target database to obtain the migration verification results. This allows for flexible definition of the range of data to be migrated and the target database address, overcoming the limitations of traditional methods that require fixed full-table migrations. It adapts to diverse MySQL database migration needs and can process the migration of multiple files simultaneously (e.g., multiple instances migrating data to different target MySQL online databases). This significantly shortens migration time in scenarios with large amounts of data, solving the problems of long backup times and insufficient concurrency in traditional methods. Furthermore, after migration, data verification is performed on the migrated data in the target database to avoid insufficient verification leading to incorrect data entry, ensuring the consistency and accuracy of the migrated data with the source data. The entire process relies on the migration configuration information database to achieve standardized operations. From data partitioning and task generation to multi-instance scheduling, everything is executed automatically according to the configuration, reducing manual intervention.

[0041] In one feasible implementation, the data migration platform further includes a configuration management module, which is used to receive data migration rules, workflow configurations and custom verification rules input by users through a visual interface and / or configuration files, and generate a migration configuration information database.

[0042] It should be noted that the configuration management module refers to the functional module in the data migration platform responsible for receiving and verifying user configuration information and generating a standardized set of rules; the visual interface refers to the graphical operation interface provided by the configuration management module, which allows users to complete configurations through drag-and-drop, drop-down selection, and input box filling, without writing code; the configuration file refers to the file containing configuration information written by the user in a preset format (such as XML or JSON), which can be uploaded to the configuration management module to complete the configuration; the data migration rules refer to the rules defined by the user regarding the scope of data migration and the relationship between the source and target databases, including the source database address, target database address, table name to be migrated, data filtering conditions, etc.; the workflow configuration refers to the configuration of the execution order and dependencies of migration tasks defined by the user, such as "data processing can only be performed after data reading is completed, and data verification can only be performed after data processing is completed"; and the custom verification rules refer to the rules defined by the user according to business needs to verify the accuracy of data, including field format rules, numerical range rules, and cross-field logic rules, etc.

[0043] It should be further explained that the core purpose of this module is to address the problems of "poor scalability and low configurability" in traditional technologies. By supporting both visual interfaces and configuration files as input methods, it lowers the configuration threshold for users. Simultaneously, it generates a migration configuration information library to provide a unified rule basis for the entire migration process, reducing manual intervention and development support costs. In one possible implementation, the configuration management module can provide a configuration template library, including templates such as "MySQL online database data migration templates" and "historical database archive migration templates." Users can modify some configuration items based on these templates to further improve configuration efficiency.

[0044] Specifically, in one embodiment, the configuration management module first provides two channels for inputting configuration information: First, there is a visual interface channel. Users can log in to the web interface of the configuration management module, fill in the source database IP, port, account password, and table name to be migrated in the "Source Database Configuration" section, select fields (such as "Creation Time") from the drop-down list in the "Data Filtering" section, set filter conditions (such as "greater than" and "2025-01-01"), drag and drop task nodes (such as "Data Reading", "Data Processing", and "Data Migration") and connect them to set dependencies in the "Workflow Configuration" section, and click "Add Rule" in the "Validation Rules" section to select the rule type (such as "Field Format") and set the rule content (such as "Date Format YYYY-MM-DD"). Secondly, there is the configuration file channel. Users write configuration files according to the JSON format template provided by the configuration management module. The configuration files contain three core fields: "dataMigrationRules", "workflowConfig", and "customValidationRules". Users fill in the data migration rules, workflow configuration, and custom validation rules respectively, and then upload the configuration files to the module through the upload button on the interface.

[0045] Furthermore, the configuration management module validates the input configuration information: it verifies whether the source / target database address format is valid (e.g., whether the IP address is in IPv4 format), whether the data filtering condition syntax is correct (e.g., whether there are illegal operators), whether there are loops in the workflow dependencies, and whether the custom validation rules are executable. After successful validation, the module integrates the configuration information according to a preset structured format, generates a migration configuration information library, stores it in the module's database, and assigns a unique identifier to this configuration information library (e.g., "Config-20250520-001"). It also supports users in modifying, deleting, and exporting the configuration information library, ensuring that subsequent functional modules can access the corresponding configuration information through the unique identifier.

[0046] This embodiment supports dual input methods—a visual interface and configuration files—through a configuration management module, catering to the different operating habits of users (e.g., non-technical users prefer the visual interface, while technical users prefer configuration files). Simultaneously, rigorous configuration information verification ensures the correctness of rule logic and compliance with formatting, preventing migration process anomalies due to configuration errors. The generated migration configuration information library provides data filtering criteria for the data source module, file partitioning and instance scheduling criteria for the data migration module, and verification rule criteria for the data verification module, achieving rule uniformity throughout the entire migration process. This significantly improves the flexibility and maintainability of data migration and allows for adjustments to configuration information at any time, greatly enhancing scalability.

[0047] In one feasible implementation, the configuration management module is further configured to store the migration tasks and their corresponding dependencies in a tree structure model.

[0048] It should be noted that the tree structure model refers to a hierarchical data structure model, which includes three core levels: "root node - parent node - child node". The dependency logic between nodes is reflected through the parent-child relationship. The parent node represents the core process stage, and the child node represents the sub-tasks under that stage. The dependency relationship corresponding to the migration task refers to the execution order logic between different migration tasks. That is, a certain task can only start after another task is completed. For example, the "data cleaning" task depends on the "data reading" task to be completed, and the "target library import" task depends on the "data verification" task to be completed.

[0049] It should be further explained that the core purpose of this storage method adopted by the configuration management module is to solve the problem of "chaotic task dependencies, difficult to understand and adjust intuitively" in traditional linear storage. Through a hierarchical tree structure, users can clearly grasp the task execution order and dependency logic, while facilitating the quick addition, deletion, and adjustment of task nodes, thus improving the flexibility and maintainability of workflow configuration. In one possible implementation, the configuration management module's visual interface can render the tree structure model in real time. Users can modify task names by double-clicking nodes, add subtasks by right-clicking nodes, and adjust hierarchical relationships by dragging and dropping nodes. After the operation is completed, the module automatically updates the stored dependencies, eliminating the need to manually write dependency configuration code.

[0050] Specifically, in one embodiment, the configuration management module first defines the hierarchical rules of the tree structure model: the root node is the "Overall Data Migration Process Between MySQL Databases", representing the starting point of the entire data migration process; the first-level parent nodes are the core process stages, including the "Data Preparation Stage", "Data Processing Stage", "Data Migration Stage", and "Data Verification Stage"; the child nodes under each first-level parent node are the subdivided migration tasks of the corresponding stage, such as the child nodes under the "Data Preparation Stage" parent node including "Source Database Connection", "Reading Raw Migration Data", and "Obtaining Migration Configuration Information", and the child nodes under the "Data Migration Stage" parent node including "Batch Splitting of Data to be Migrated", "Generating Migration Task Queue", "Starting Multi-Instance Program", and "Importing Files to be Migrated into the Target Database"; the dependency relationship between child nodes is defined through the "Prerequisite Task ID" field, such as the "Prerequisite Task ID" of the "Reading Raw Migration Data" child node being the ID of the "Source Database Connection" child node, meaning that the "Source Database Connection" must be completed before "Reading Raw Migration Data" can be executed.

[0051] Furthermore, the configuration management module receives migration tasks and dependencies input by the user through a visual interface or configuration file, and constructs a tree structure according to the above hierarchical rules: tasks are categorized into "core stage - sub-tasks", each task is assigned a unique node ID, and dependent tasks are associated through "preceding task IDs", forming a node data table containing node ID, node name, parent node ID, preceding task ID, and task status (such as "pending configuration" or "effective").

[0052] Finally, the module stores the tree structure model and node data table in the migration configuration information repository, and supports two query methods: one is to display the tree structure through a visual interface, where users can expand / collapse parent nodes to view subtasks and dependencies; the other is to output structured data (such as JSON format) through an interface for the process control platform to call. The process control platform can determine the task execution order based on the "preceding task ID" to ensure that the migration process proceeds in an orderly manner according to dependencies.

[0053] This embodiment uses a tree structure model in the configuration management module to store migration tasks and dependencies, transforming complex multi-task processes into an intuitive hierarchical structure. This allows users to quickly identify the core stage logic of "data preparation → data processing → data migration → data verification," as well as the execution order of subdivided tasks within each stage, solving the problem of ambiguous dependencies in traditional linear storage. Furthermore, when business requirements change and the process needs adjustment, users only need to add child nodes under the parent node, and the module will automatically update the tree structure and dependencies without reconstructing the entire configuration file, significantly enhancing the scalability of the migration process.

[0054] In one feasible implementation, the data migration platform further includes a data processing module, which is used to obtain the original migration data in the source database and perform cleaning and transformation operations on the original migration data according to the migration configuration information database to obtain the data to be migrated.

[0055] It should be noted that the data processing module refers to the functional module in the data migration platform responsible for preprocessing the raw data in the source database to make it conform to the migration and target database storage requirements; the source database refers to the MySQL database that stores the raw data to be migrated; the raw migration data refers to the data read directly from the source database without any processing, which may have problems such as inconsistent format, invalid information, and field types that do not match the target database.

[0056] In one possible implementation, the data processing module can support a two-step "preview-execute" operation. Users can first select a small amount of raw migration data (such as 100 records) to perform cleaning and transformation. After previewing the processing results and confirming that the rules are correct, the full data can then be processed to avoid abnormal processing of the full data due to rule errors.

[0057] Specifically, the data processing module first establishes a communication connection with the data source module to obtain the raw migration data from the source database. The data source module connects to the source database (such as the mysql004 database) based on the source database address and the table name to be migrated in the migration configuration information database, reads the raw migration data according to the data filtering conditions (such as "order status = completed"), and passes it to the data processing module. The raw migration data may contain problems such as the "order date" field being "2025 / 05 / 20" (non-standard date format), the "user name" field containing the special character "Zhang San#", and the "order amount" field being the string "1000 yuan".

[0058] Subsequently, the data processing module reads the data cleaning and transformation rules from the migration configuration information database and executes the operations in the order of "clean first, then transform": For example, first clean the original migration data, removing "#" from "User Name", deleting 3 records with empty "Order ID", and removing 2 duplicate order data; then perform transformation on the cleaned data, adjusting the format of "Order Date", converting the type of "Order Amount", and renaming the fields.

[0059] Finally, the data processing module performs integrity verification on the processed data (such as checking whether all key fields exist and whether the data types meet the requirements of the target library). After the verification is passed, the data to be migrated is generated and passed to the data source module for storage, so that it can be called by the subsequent data migration module.

[0060] This embodiment obtains the original migration data through the data processing module and performs cleaning and transformation operations based on the migration configuration information database. This effectively solves the problems of disordered format, invalid content, and incompatibility with the target database in the original data, ensuring that the data to be migrated can be successfully imported into the target database and avoiding migration interruptions caused by data format errors. At the same time, the processing rules are entirely based on the migration configuration information database. Users can flexibly adjust the cleaning and transformation rules according to changes in the source / target database (such as adjusting the field mapping relationship after changing the target database) without modifying the module code, which significantly enhances the scalability and practicality of the data migration platform.

[0061] In one feasible implementation, the data migration platform further includes a monitoring and alarm module, which is used to capture the task execution status and execution logs of the data source module, the data verification module, and the data migration module in real time, and generate a real-time migration progress report in combination with the alarm rules in the configuration management module. It should be noted that the monitoring and alarm module refers to the functional module in the data migration platform responsible for tracking the entire migration process status in real time, recording execution information, and generating progress reports; the task execution status refers to the real-time status of each module when executing tasks, including "pending startup," "in execution," "success," "failure," and "timeout," such as "in execution (30% of the data to be migrated has been read)" for the data source module and "success (5 files to be migrated have been migrated)" for the data migration module; the execution log refers to the detailed records generated during the execution of tasks by each module, including task start time, execution steps, amount of data involved, and abnormal information; the configuration management module refers to the module responsible for generating a standardized migration configuration information library, and the alarm rules it contains refer to the preset conditions for triggering alarms, such as "task execution timeout exceeds 30 minutes"; the real-time migration progress report refers to the structured report output by the monitoring and alarm module, which contains key information about the entire migration process, including the distribution of task execution status of each module, the number of completed / incomplete migration tasks, the amount of data to be migrated / migrated, the migration progress percentage, and the current execution node, allowing users to intuitively grasp the migration dynamics.

[0062] It's worth noting that the core purpose of this module is to address the issues of "opaque migration progress and inability to trace the process in real time" in traditional technologies. By capturing status and logs in real time, it allows users to clearly understand the progress of each stage of the migration. Simultaneously, it incorporates alarm rules to provide a basis for judging subsequent anomaly alerts, improving the monitorability and operational efficiency of data migration. In one possible implementation, real-time migration progress reports can be displayed through a visual dashboard. This dashboard uses bar charts to show the completion rate of each module's tasks, line charts to show the migration progress trend over time, and lists to display details of currently executing tasks. Users can refresh the data in real time through the dashboard without manually querying log files.

[0063] Specifically, the monitoring and alarm module first establishes real-time communication connections with the data source module, data verification module, and data migration module, and then captures the task execution status and execution logs of each module at a fixed frequency (e.g., every 10 seconds) through a preset interface (such as an HTTP interface). For the data source module, capture the "data to be migrated reading status", "migration configuration information database acquisition status" and corresponding logs; for the data migration module, capture the "file to be migrated partitioning status", "migration task queue progress", "multi-instance running status" and logs; for the data verification module, capture the "target database data reading status", "verification execution status" and logs.

[0064] Subsequently, the monitoring and alarm module reads alarm rules from the standardized configuration information library of the configuration management module, and then performs structured processing on the captured status and logs: integrating the data in the format of "module name-task type-execution status-data volume-timestamp", and calculating the task completion rate of each module and the overall migration progress (e.g., 6000 / 10000 data entries have been migrated, progress is 60%).

[0065] Finally, the module generates a real-time migration progress report according to a preset template. The report includes three sections: "Overall Progress Overview", "Details of Each Module", and "Execution Log List". Users can view the report through a web interface and export it (e.g., to Excel format) for subsequent operation and maintenance analysis.

[0066] The monitoring and alarm module is also used to generate an abnormal alarm and push it to the designated person in charge when an abnormality is detected in the execution of the migration task.

[0067] It should be noted that migration tasks refer to the specific tasks executed by each module in the data migration process; migration task execution anomalies refer to situations where the execution status of a migration task does not conform to the preset normal logic, including situations such as "task execution timeout", "task execution failure", "instance operation anomaly", and "data verification anomaly rate exceeding the standard"; anomaly alarms refer to alarm notifications generated by the monitoring and alarm module in response to anomalies, containing key information, including "anomaly task ID", "anomaly module name", "anomaly type", "anomaly occurrence time", "anomaly cause (such as 'target database connection failed, error code 1045' recorded in the log)" and "suggested handling solution (such as 'check if the target database account password is correct')"; designated responsible persons refer to the personnel preset in the standardized configuration information database of the configuration management module who are responsible for handling data migration anomalies, and the information includes the responsible person's name, phone number, email address, and the recipients of "SMS alarms" and "call alarms" in the corresponding technical handover document.

[0068] In one possible implementation, anomaly alerts can support tiered push notifications: anomalies are categorized by severity as "urgent (e.g., target database connection failure causing migration interruption)," "important (e.g., a single instance anomaly does not affect the overall migration)," and "general (e.g., a data verification failure)." Urgent alerts are sent via both SMS and email, important alerts are sent via SMS, and general alerts are sent via email, ensuring that responsible personnel prioritize handling high-priority anomalies.

[0069] Specifically, the monitoring and alarm module first establishes an anomaly detection mechanism: it compares the execution status of each migration task with the alarm rules in the configuration management module in real time to determine whether an anomaly is triggered. If the execution time of the "multi-instance migration task" of the data migration module exceeds the "timeout threshold of 30 minutes" in the alarm rules, it is determined to be an "task execution timeout" anomaly. If the "migration data verification failure rate" of the data verification module exceeds the "failure rate threshold of 0.1%" in the alarm rules, it is determined to be a "data verification anomaly". If the instance running status of the data migration module is "abnormal" for 10 minutes and does not recover automatically, it is determined to be an "instance running anomaly".

[0070] When an anomaly is detected, the module immediately extracts key anomaly information from the execution log and generates an anomaly alarm according to a preset template. The alarm content is formatted as follows: "[Data Migration Anomaly Alarm] - Anomaly Task ID: Task-20240520-004; Anomaly Module: Data Migration Module; Anomaly Type: Target Database Connection Failure; Occurrence Time: 2024-05-20 10:15; Anomaly Cause: Incorrect username and password when connecting to MySQL online database B, error code 1045; Suggested Solution: Log in to the configuration management module to check the target database username and password configuration, resubmit, and restart the migration task."

[0071] Subsequently, the module retrieves the contact information (such as phone number and email address) of the designated person in charge from the standardized configuration information database of the configuration management module, and selects the push method according to the alarm classification rules: if it is an "urgent" anomaly, an SMS alarm is sent to the person in charge's mobile phone via the SMS gateway, and an email alarm is sent to the person in charge's email address via the email server; if it is an "important" or "general" anomaly, the corresponding level is pushed. After the push is completed, the module stores the anomaly alarm information and push record in the alarm log database, supporting users to query and trace it later.

[0072] This embodiment achieves real-time monitoring and proactive anomaly alerts throughout the entire data migration process through a monitoring and alarm module. On the one hand, by frequently capturing the task execution status and execution logs of the data source module, data migration module, and data verification module, and combining them with the alarm rules of the configuration management module, a structured real-time migration progress report is generated. This allows users to intuitively grasp the overall migration progress without having to query the logs of each module individually, solving the problem of opaque progress in traditional migrations and improving the convenience of operation and maintenance. On the other hand, by detecting migration task execution anomalies in real time (such as timeouts, failures, and instance anomalies), anomaly alarms containing the causes of the anomalies and handling suggestions are automatically generated and pushed to designated responsible persons according to the severity level, ensuring that anomalies can be detected and handled as soon as possible, avoiding the extension of the migration cycle or data risks due to anomaly delays.

[0073] Based on this, embodiments of this application provide a data migration method, referring to... Figure 2 , Figure 2 This is a flowchart illustrating the data migration method of this application in Embodiment 1.

[0074] In this embodiment, the data migration method includes steps S61-S63: Step S61: Obtain the data to be migrated, and obtain the migration configuration information database corresponding to the data to be migrated; Step S62: Based on the migration configuration information in the migration configuration information database, the data to be migrated is divided into several files to be migrated, and a migration task corresponding to each file to be migrated is generated, so as to migrate the files to be migrated to the target database through multi-instance scheduling. Specifically, based on the batch size threshold of the migration configuration information, the data to be migrated is divided into several files to be migrated, and then a migration task corresponding to each file to be migrated is generated to form a task queue. This establishes a communication connection between multiple instances and the task queue, and the files to be migrated are migrated to the target database through multi-instance scheduling processing according to each migration task.

[0075] Step S63: Perform data verification on the migration data in the target database to obtain the data migration verification result.

[0076] Specifically, migration data is read from the target database. Then, custom validation rules are read from the migration configuration information repository, including quantity validation rules, format validation rules, value validation rules, and content consistency validation rules. Next, validation is performed according to the rules: the total amount of migration data is counted; if it is inconsistent with the total amount of data in the file to be migrated, it is marked as "data missing / duplicate"; the format and value of each field are checked, and if they do not conform to the rules, they are marked as "format abnormal" or "value abnormal"; the field content is compared, and if they are inconsistent, they are marked as "content inconsistent". Finally, the quantity and details of various abnormal data are counted, and data migration validation results are generated. For example, the content includes "Validation passed: 5798 records, Validation failed: 2 records, Anomaly list (Data ID=1005: date format error; Data ID=2010: amount is -50, value exceeds the limit), Validation pass rate: 99.97%", and the results are synchronized to the monitoring module. If the validation pass rate is lower than the preset threshold (e.g., 99.9%) in the migration configuration information repository, an anomaly alarm is triggered.

[0077] This embodiment obtains the data to be migrated and the corresponding migration configuration information database. Based on the migration configuration information in the database, the data to be migrated is divided into several files to be migrated, and migration tasks corresponding to each file are generated. Through multi-instance scheduling, these files are migrated to the target database. Data verification is then performed on the migrated data in the target database to obtain the data migration verification results. This achieves a closed-loop process of "flexible configuration - rapid migration - quality assurance" for data between MySQL databases. On the one hand, relying on the standardized rules of the migration configuration information database, data is obtained from the data to be migrated... The migration process is fully configurable, requiring minimal manual development and intervention. Furthermore, by splitting the migration into batches and enabling concurrent migration across multiple instances, the migration time for large data volumes is significantly reduced, addressing issues of long backup times and insufficient concurrency, thus achieving rapid migration. Simultaneously, custom verification rules ensure the accuracy of the migrated data, resolving the problem of insufficient data verification and guaranteeing the reliability of data in the target database. This process not only meets the core requirements of MySQL database migration in the context of explosive growth in information technology but also reduces development and maintenance costs and improves the stability and maintainability of data migration through standardized and automated design throughout the entire process.

[0078] In one feasible implementation, the step of dividing the data to be migrated into several files to be migrated based on the migration configuration information in the migration configuration information database, and generating migration tasks corresponding to each file to be migrated, so as to migrate the files to be migrated to the target database through multi-instance scheduling processing, includes: Step S71: Divide the data to be migrated into several files to be migrated according to the batch size threshold of the migration configuration information; It should be noted that the batch size threshold refers to the maximum amount of data that a single file to be migrated can carry, preset by the user through a visual interface or configuration file. This threshold can be flexibly adjusted according to the I / O capacity of the target database. The core purpose of this step is to solve the problem of "excessive data volume in a single task leading to long migration time" in traditional technologies by splitting the data into batches, breaking down the huge amount of data to be migrated into files of appropriate size, and reducing the processing time of a single file.

[0079] Specifically, the batch size threshold (e.g., preset to 1000 records / file) is read from the migration configuration information in the configuration management module, while the data to be migrated (e.g., 6800 records of data from the "mysql004 database - user table", which has already undergone standardization) is obtained from the data source module. Then, the data migration module performs a splitting operation according to the "continuous data splitting" logic: Starting with the first piece of data to be migrated, the module accumulates counts based on a batch size threshold. Each time the threshold is reached, a new file is generated to be migrated. If the remaining data is less than the threshold, the remaining data is generated into a separate file. During file generation, the module names each file according to the rule "source database identifier-table name-batch number" and stores the files in a pre-defined unified path on the server. It also records metadata such as the storage path, data size, and corresponding source database table name for each file, for use in subsequent migration task generation. After splitting, the module performs integrity checks on all files to be migrated (e.g., checking if the data size of each file meets the threshold requirement and if the file can be opened and read normally) to ensure no files are corrupted or data is missing.

[0080] Step S72: Generate migration tasks corresponding to each of the files to be migrated to form a task queue; It should be noted that the task queue refers to a structured collection of tasks formed by sorting all migration tasks according to preset rules (such as task priority and generation time). It supports multiple instances of programs to obtain tasks sequentially or through a competition mechanism and is the core carrier for realizing multi-instance concurrent scheduling. The core purpose of this step is to transform "files to be migrated" into "executable migration tasks," and to manage all migration requirements in a unified manner through the task queue, avoiding task chaos or duplicate execution, while providing a clear basis for task allocation in subsequent multi-instance scheduling.

[0081] In one possible implementation, task priority can be configured according to the business attributes of the data to be migrated: for example, the migration task priority corresponding to "order payment data" is set to "high", and the task priority corresponding to "user historical behavior data" is set to "medium", so as to ensure that core business data is migrated first and to ensure business continuity.

[0082] Specifically, it iterates through all the generated files to be migrated, and generates a corresponding migration task for each file: In one embodiment, firstly, unique task IDs are generated for 7 migration tasks, named in the format of "Task-Date-Sequence Number" to ensure that the tasks can be uniquely identified, and associated with the storage path of the files to be migrated for subsequent instances to read the files. Then, the target database identifier and data table name are determined, the target database corresponding to the files to be migrated is obtained from the migration configuration information, and the task priority is assigned. According to the rule of "user table data priority is medium" in the migration configuration information, the priority of all 7 migration tasks is set to "medium", and the task timeout time is set. According to the "default timeout threshold of 30 minutes" in the migration configuration information, the timeout time of each task is set to 30 minutes.

[0083] After all migration tasks are generated, the data migration module sorts the tasks according to the rule of "task priority from high to low, and tasks with the same priority sorted by generation time from early to late" to form a task queue (if all 7 tasks in this case have a priority of "medium", they are sorted by task ID number). At the same time, the task queue is stored in the task management database of the data migration module, and the initial status of each task is recorded as "pending assignment" for subsequent multi-instance program calls.

[0084] Step S73: Establish communication connections between multiple instances and the task queue, and migrate the files to be migrated to the target database through multi-instance scheduling processing according to each migration task.

[0085] It should be noted that multiple instances refer to migration instance programs deployed by the data migration platform for parallel execution of migration tasks. The number of instances can be dynamically adjusted according to the "multi-instance quantity limit" in the migration configuration information (such as not exceeding 1.5 times the number of CPU cores of the database server). For example, an 8-core CPU server can start 12 instances. Communication connection refers to the real-time data interaction link between the migration instance and the task queue, which is usually established based on the TCP / IP protocol to ensure that the instance can obtain the tasks to be assigned in the task queue in real time and provide feedback on the task execution status. Multi-instance scheduling processing refers to the mechanism of "idle instance competition for tasks" to allow multiple instances to obtain and execute migration tasks in the task queue in parallel, realizing the process of migrating multiple files to be migrated to the target database at the same time.

[0086] The core purpose of this step is to solve the problem of "insufficient concurrency and inability to handle multiple target database migrations at the same time" in traditional technologies by executing tasks concurrently through multiple instances, thereby significantly improving the data migration speed, shortening the overall migration cycle, and ensuring the real-time nature of task allocation and status feedback by relying on communication connections, thus guaranteeing the stability of the migration process.

[0087] In one possible implementation, multi-instance scheduling can support "target database load balancing": if there are tasks in the task queue that point to different target databases (e.g., some tasks point to MySQL online database A, and some point to MySQL online database B), the scheduling mechanism will prioritize assigning tasks to the instance corresponding to the target database with the lower current load, so as to avoid overloading a single target database.

[0088] Specifically, based on the "multi-instance quantity limit" in the migration configuration information, the corresponding number of migration instance programs are launched, and a unique identifier is assigned to each instance. Subsequently, a TCP / IP communication connection is established between each instance and the task queue to ensure that the instance can send "task retrieval requests" to the task queue in real time, the task queue can push tasks to be assigned to the instance, and the instance can provide real-time feedback on the task execution status (such as "in execution", "success", "failure").

[0089] Next, multi-instance scheduling is initiated: instances in an "idle" state automatically send task retrieval requests to the task queue, which allocates tasks according to the principle of "first come, first served, priority first." Instances that retrieve tasks perform migration according to the task information: they read file data based on the storage path of the files to be migrated, establish a connection with the target database based on the target database identifier (such as MySQL online database B) (including account and password authentication, data transmission protocol configuration), and import the file data into the target database according to the target data table structure (such as the field mapping relationship of the "user_info" table).

[0090] During the migration process, each instance reports its progress to the task queue every time it completes 10% of the data import (e.g., "Task-20240520-001 30% imported"). If the task completes within 30 minutes, its status is marked as "successful," and the instance is released to the "idle" state to acquire new tasks. If the task times out or fails (e.g., the target database connection is interrupted), its status is marked as "failed," and the task is returned to the task queue (marked as "awaiting reallocation"). Simultaneously, the abnormal instance is marked as "abnormal," and its task acquisition privileges are suspended. The monitoring module pushes an abnormal alert to the responsible person. This process continues until all tasks in the task queue have been successfully executed, completing the migration of the files to be migrated to the target database.

[0091] This embodiment divides the data to be migrated into several files based on the batch size threshold of the migration configuration information, and then generates migration tasks corresponding to each file to form a task queue. This establishes communication connections between multiple instances and the task queue. Based on each migration task, the files to be migrated are migrated to the target database through multi-instance scheduling, thereby achieving "high concurrency, high efficiency, and high stability" execution of data migration between MySQL databases. On the one hand, batch splitting breaks down the massive data to be migrated into smaller files, avoiding the time-consuming problem of traditional single-file migration and providing a foundation for multi-instance parallel processing. On the other hand, the task queue manages migration tasks uniformly, and combined with a multi-instance competitive scheduling mechanism, multiple instances can process the migration of different files to be migrated simultaneously (e.g., 12 instances migrate 7 files to MySQL online databases A and B in parallel). The concurrency capability increases with the number of instances until it matches the hardware capacity limit of the database server, significantly shortening the migration time in scenarios with large amounts of data.

[0092] In one feasible implementation, the step of migrating the files to be migrated to the target database through multi-instance scheduling processing according to each migration task includes: Step S81: Obtain the task priority of the task queue; It should be noted that task priority refers to the level identifier preset for each migration task to determine the execution order. The classification is based on the migration configuration information database generated by the configuration management module and can be customized by the user through a visual interface or configuration file. For example, it can be divided into three levels: "high", "medium" and "low" according to the business importance of the data to be migrated.

[0093] Specifically, the task queue management interface is invoked to read the complete task queue from the task management database of the data migration module. This queue contains all migration tasks generated in step S72, and each task carries metadata such as "task ID", "path to be migrated", "target database identifier", and "task priority". Subsequently, for each migration task in the task queue, the data migration module retrieves the corresponding priority configuration details from the migration configuration information database by associating the "task ID".

[0094] Step S82: Generate a migration sorting corresponding to each migration task according to the task priority; It should be noted that migration sorting refers to the final execution order of migration tasks, determined by task priority. The sorting rules for task priority are preset by the migration configuration information database, following the logic of "high-priority tasks are executed first, and tasks of the same priority are sorted according to preset secondary rules". The secondary rules may include task generation time (earlier tasks are prioritized), amount of data to be migrated (tasks with smaller data volume are prioritized to reduce instance waiting time), target database load (tasks pointing to low-load target databases are prioritized), etc.

[0095] The core purpose of this step is to sort the migration tasks in the task queue into an ordered migration order according to priority, so as to avoid the core task being delayed due to disordered competition for tasks during multi-instance scheduling. At the same time, the execution order of tasks with the same priority is optimized through secondary rules to improve the overall migration efficiency.

[0096] Specifically, the system reads the preset priority sorting primary rules ("high priority > medium priority > low priority") and secondary rules (such as "for tasks of the same priority, priority is given to those generated earlier") and secondary rules (such as "priority is given to those generated earlier") from the migration configuration information database. Then, based on the "task-priority" mapping table formed in step S81, all migration tasks in the task queue are sorted hierarchically: First, tasks are divided into "high priority task group," "medium priority task group," and "low priority task group" according to the primary rules; second, tasks within each priority group are sorted according to the secondary rules; the sorting within the medium and low priority groups is done similarly. Finally, the priority groups are concatenated according to the primary rules to form a complete migration sort, and the migration sort is synchronized to the task queue management database, updating the "sorting number" field of each task for subsequent multi-instance scheduling.

[0097] Step S83: For any of the migration tasks, based on the migration sorting and the migration database connection information corresponding to the migration task, the files to be migrated corresponding to the migration task are migrated to the target database through multi-instance scheduling.

[0098] It should be noted that the migration database connection information refers to the key parameters used to connect to the target database corresponding to the migration task. These parameters are derived from the migration configuration information database and specifically include the target database address (such as the IP and port of MySQL online library A), database account password (stored in encrypted form), target data table name (such as the "order_settlement" table), and data transmission protocol (such as TCP / IP). Multi-instance scheduling refers to the mechanism by which the data migration module starts multiple migration instance programs, allocates tasks according to the migration order, and executes them in parallel.

[0099] In one possible implementation, the migration database connection information can be dynamically updated: if the target database address or account password changes, the user can modify the migration configuration information database through the configuration management module. The instance will re-obtain the latest connection information before executing the task to avoid migration failure due to configuration changes.

[0100] Specifically, based on the "multi-instance quantity limit" in the migration configuration information database, the corresponding number of migration instance programs are started and initialized. Each instance is in an "idle" state and can read migration sorting and task queue information in real time. Subsequently, the data migration module allocates tasks to idle instances according to migration sorting: high-priority tasks are assigned to idle instances first. After the instance obtains the task, it retrieves the corresponding migration database connection information from the migration configuration information database through the "task ID" and establishes an encrypted connection with the target database based on the connection information (ensuring data transmission security through the SSL protocol).

[0101] After the connection is successfully established, the instance reads the file data according to the storage path of the files to be migrated in the task, and imports the data into the target database according to the field mapping relationship of the target data table. During the migration process, the instance provides real-time feedback on the task execution status to the data migration module: if the task is successfully completed, the instance is released to an "idle" state, waiting to be assigned the next task with a higher priority; if the task fails, the data migration module puts the task back into the corresponding priority position in the task queue and marks it as "awaiting reallocation," while isolating the abnormal instance and assigning the task to another idle instance for re-execution. This process continues until all tasks in the migration priority have been successfully executed, completing the migration of all files to be migrated to the target database.

[0102] This embodiment obtains the task priorities of the task queue and then generates a migration order corresponding to each migration task according to the task priorities. For any migration task, based on the migration order and the migration database connection information corresponding to the migration task, the files to be migrated corresponding to the migration task are migrated to the target database through multi-instance scheduling. By combining task priority and migration order, core business data is ensured to be migrated first, avoiding non-core data occupying resources and affecting core business. At the same time, relying on the migration database connection information ensures that the instance can accurately connect to the target database, avoiding migration failure due to incorrect connection parameters. Combined with multi-instance concurrent scheduling, the overall migration efficiency is improved while ensuring the execution order, shortening the migration cycle of large batches of data and solving the pain points of traditional technologies such as "insufficient concurrency and long migration time".

[0103] For example, to help understand the implementation process of the data migration method, please refer to... Figure 3 , Figure 3 A simplified flowchart illustrating the data migration method described in this application.

[0104] Specifically, first, users configure MySQL data migration information in the interface configuration module. For example, they determine that specific business data needs to be migrated from multiple databases in MySQL online database A, such as database 1 and database 2. They also set relevant migration rules and target databases (e.g., the corresponding database in MySQL online database B). Afterward, the data migration operation and maintenance phase begins. Next, the configured migration task is passed to the process control platform through a scheduling mechanism. Once the process control platform receives this detailed configuration information, it officially starts the data migration process. On one hand, it exports data from various databases in MySQL online database A (such as database 1 storing transaction order data, database 2 storing user information, etc.), generating files such as "Data File 1" containing order data, "Data File 2" containing user information, and so on, up to "Data File N" corresponding to different business data. On the other hand, the process control platform imports these generated data files into the corresponding database in MySQL online database B according to the configured rules, thus realizing the data migration.

[0105] Throughout the data migration process, the alarm platform will continuously monitor the system. If any anomalies occur, such as a connection interruption when exporting data from database 1 or a data format error when importing data into MySQL online database B, the relevant personnel will be notified promptly via SMS alarms. The platform will also alert operations and maintenance personnel via call alarms so that they can quickly handle the anomalies and ensure the smooth progress of the data migration.

[0106] It should be noted that the examples in the figure are only for understanding this application and do not constitute a limitation on the data migration method of this application. Any simple modifications based on this technical concept are within the protection scope of this application.

[0107] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0108] This application provides a data migration device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, which are executed by the at least one processor to enable the at least one processor to perform the data migration method in Embodiment 1 above.

[0109] The following is for reference. Figure 4The diagram illustrates a structural schematic of a data migration device suitable for implementing embodiments of this application. The data migration device in the embodiments of this application may include, but is not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Description), PMPs (Portable Media Players), in-vehicle terminals (e.g., in-vehicle navigation terminals), and fixed terminals such as digital TVs and desktop computers. Figure 4 The data migration device shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of this application.

[0110] like Figure 4 As shown, the data migration device may include a processing unit 1001 (e.g., a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory 1002 or a program loaded from a storage device 1003 into a random access memory 1004. The random access memory 1004 also stores various programs and data required for the operation of the data migration device. The processing unit 1001, the read-only memory 1002, and the random access memory 1004 are interconnected via a bus 1005. An input / output interface 1006 is also connected to the bus. Typically, the following systems can be connected to the input / output interface 1006: input devices 1007 including, for example, a touchscreen, touchpad, keyboard, mouse, image sensor, microphone, accelerometer, gyroscope, etc.; output devices 1008 including, for example, a liquid crystal display (LCD), speaker, vibrator, etc.; storage devices 1003 including, for example, magnetic tape, hard disk, etc.; and communication devices 1009. Communication device 1009 allows the data migration device to communicate wirelessly or wiredly with other devices to exchange data. While the figures show data migration devices with various systems, it should be understood that implementation or possession of all the systems shown is not required. More or fewer systems may be implemented alternatively.

[0111] Specifically, according to the embodiments disclosed in this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments disclosed in this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device, or installed from storage device 1003, or installed from read-only memory 1002. When the computer program is executed by processing device 1001, it performs the functions defined in the methods of the embodiments disclosed in this application.

[0112] The data migration device provided in this application, employing the data migration method described in the above embodiments, can solve the technical problems mentioned in the background section. Compared with the prior art, the beneficial effects of the data migration device provided in this application are the same as those of the data migration method provided in the above embodiments, and other technical features of the data migration device are the same as those disclosed in the previous embodiment method, and will not be repeated here.

[0113] It should be understood that the various parts disclosed in this application can be implemented using hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials, or characteristics can be combined in any suitable manner in one or more embodiments or examples.

[0114] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0115] This application provides a computer-readable storage medium having computer-readable program instructions (i.e., a computer program) stored thereon, the computer-readable program instructions being used to execute the data migration method described in the above embodiments.

[0116] The computer-readable storage medium provided in this application may be, for example, a USB flash drive, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems or devices, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system or device. The program code contained on the computer-readable storage medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination thereof.

[0117] The aforementioned computer-readable storage medium may be included in the data migration device; or it may exist independently and not be assembled into the data migration device.

[0118] The aforementioned computer-readable storage medium carries one or more programs that, when executed by the data migration device, cause the data migration device to: Obtain the data to be migrated, and obtain the migration configuration information database corresponding to the data to be migrated; Based on the migration configuration information in the migration configuration information database, the data to be migrated is divided into several files to be migrated, and a migration task corresponding to each file to be migrated is generated, so as to migrate the files to be migrated to the target database through multi-instance scheduling. Data verification is performed on the migration data in the target database to obtain the data migration verification result.

[0119] Computer program code for performing the operations of this application can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a Local Area Network (LAN) or a Wide Area Network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0120] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0121] The modules described in the embodiments of this application can be implemented in software or hardware. The names of the modules do not necessarily limit the functionality of the unit itself.

[0122] The readable storage medium provided in this application is a computer-readable storage medium that stores computer-readable program instructions (i.e., a computer program) for performing the above-described data migration method, and is able to solve the technical problems described in the background art. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in this application are the same as the beneficial effects of the data migration method provided in the above embodiments, and will not be repeated here.

[0123] This application provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the data migration method described above.

[0124] The computer program product provided in this application can solve the technical problems described in the background section. Compared with the prior art, the beneficial effects of the computer program product provided in the embodiments of this application are the same as the beneficial effects of the data migration method provided in the above embodiments, and will not be repeated here.

[0125] The above description is only a part of the embodiments of this application and does not limit the patent scope of this application. All equivalent structural transformations made under the technical concept of this application and using the contents of the specification and drawings of this application, or direct / indirect applications in other related technical fields, are included in the patent protection scope of this application.

Claims

1. A data migration platform, characterized in that, The data migration platform includes a data source module, a data migration module, and a data verification module. The data source module is used to obtain the data to be migrated and to obtain the migration configuration information database corresponding to the data to be migrated. The data migration module is used to divide the data to be migrated into several files to be migrated based on the migration configuration information in the migration configuration information database, and generate a migration task corresponding to each file to be migrated, so as to migrate the files to be migrated to the target database through multi-instance scheduling. The data verification module is used to verify the migration data in the target database and obtain the data migration verification result.

2. The data migration platform as described in claim 1, characterized in that, The data migration platform also includes a configuration management module, which is used to receive data migration rules, workflow configurations and custom verification rules input by users through a visual interface and / or configuration files, and generate a migration configuration information database.

3. The data migration platform as described in claim 1, characterized in that, The configuration management module is also used to store the migration tasks and their corresponding dependencies in a tree structure model.

4. The data migration platform as described in claim 1, characterized in that, The data migration platform also includes a data processing module, which is used to obtain the original migration data in the source database and perform cleaning and transformation operations on the original migration data according to the migration configuration information database to obtain the data to be migrated.

5. The data migration platform as described in claim 1, characterized in that, The data migration platform also includes a monitoring and alarm module, which is used to capture the task execution status and execution logs of the data source module, the data verification module, and the data migration module in real time, and generate a real-time migration progress report in combination with the alarm rules in the configuration management module. The monitoring and alarm module is also used to generate an abnormal alarm and push it to the designated person in charge when an abnormality is detected in the execution of the migration task.

6. A data migration method, characterized in that, include: Obtain the data to be migrated, and obtain the migration configuration information database corresponding to the data to be migrated; Based on the migration configuration information in the migration configuration information database, the data to be migrated is divided into several files to be migrated, and a migration task corresponding to each file to be migrated is generated, so as to migrate the files to be migrated to the target database through multi-instance scheduling. Data verification is performed on the migration data in the target database to obtain the data migration verification result.

7. The data migration method as described in claim 6, characterized in that, The process of dividing the data to be migrated into several files based on the migration configuration information in the migration configuration information database, and generating migration tasks corresponding to each file to be migrated, so as to migrate the files to be migrated to the target database through multi-instance scheduling, includes: Based on the batch size threshold of the migration configuration information, the data to be migrated is divided into several files to be migrated; Generate migration tasks corresponding to each of the files to be migrated, forming a task queue; Establish communication connections between multiple instances and the task queue, and migrate the files to be migrated to the target database through multi-instance scheduling processing according to each migration task.

8. The data migration method as described in claim 7, characterized in that, The step of migrating the files to be migrated to the target database through multi-instance scheduling processing according to each migration task includes: Obtain the task priority of the task queue; Generate a migration sort corresponding to each migration task according to the task priority; For any of the migration tasks, based on the migration sorting and the migration database connection information corresponding to the migration task, the files to be migrated corresponding to the migration task are migrated to the target database through multi-instance scheduling.

9. A data migration device, characterized in that, The data migration device includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the data migration method as described in any one of claims 6 to 8.

10. A storage medium, characterized in that, The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, it implements the steps of the data migration method as described in any one of claims 6 to 8.

Images