Double-track data migration method based on DataX plug-in

Abstract

The invention discloses a double-track data migration method based on a DataX plug-in, which specifically comprises the following steps: determining a source system and a target system, and customizing a Reader plug-in and a Write plug-in according to a DataX plug-in specification; performing data migration by using the Reader plug-in and the Write plug-in, specifically, performing one-time export on data of the source system, importing the data into the target system, and recording migration starting time as T0; subscribing a binlog through Canal or Debezium, capturing a DML change after the T0, writing the change into the target system, and performing data structure adjustment in the change process; configuring gray scale flow cutting at the configuration center to perform flow switching; when migration abnormity occurs in the data migration process, the configuration center switches back to the source system, service access rollback is carried out, new data of the target system is written back to the source system, and differences of the new data are marked and repaired.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and mainly to a dual-track data migration method based on the DataX plugin. Background Technology

[0002] With the continuous development of high technology, data is generated in different forms and stored in different ways, such as using databases like MySQL, Oracle, Elasticsearch, and MongoDB. When business operations become complex, a single storage solution cannot meet the needs of the business, and when multiple storage methods coexist, rapid data migration between databases becomes increasingly important.

[0003] Traditional data migration relies on commands built into the storage database, which can only import data from the same storage database. Importing data between different databases requires special handling, and various problems can be encountered during the migration process, such as: how to ensure data consistency before and after migration; how to ensure service does not stop during migration; how to adjust data structure during migration; how to handle new data when data is rolled back in case of service failure during migration; and how to make additional business adjustments when encountering tree structures during data structure adjustments during migration, etc. Summary of the Invention

[0004] To address the above shortcomings, according to a first aspect of the present invention, a dual-track data migration method based on the DataX plugin is proposed, the specific steps of which are as follows:

[0005] S1. Determine the source system and target system, and customize the Reader plugin and Write plugin according to the DataX plugin specification;

[0006] S2. Data migration is performed using the Reader plugin and the Write plugin, specifically including:

[0007] S2.1 Export the data from the source system in one go and import it into the target system, recording the migration start time as T0;

[0008] S2.2 Subscribe to binlog logs via Canal or Debezium, capture DML changes after T0, and write the changes to the target system. During the change process, data structure adjustments are performed simultaneously.

[0009] S2.3 Configure grayscale flow switching in the configuration center to switch traffic;

[0010] S3. When a migration anomaly occurs during the data migration process, the configuration center switches back to the source system, service access is rolled back, the new data of the target system is written back to the source system, the differences of the new data are marked and repaired.

[0011] Furthermore, the Reader plugin inherits from the BaseReader.ContainerReader class, and the Write plugin inherits from the BaseWriter.ContainerWriter class; the Reader plugin and the Write plugin are used to read data from the source system, convert the data into a Record, and write the Record to the target system.

[0012] Furthermore, the process of data migration using the Reader plugin and the Write plugin also includes:

[0013] Use the Maven packaging plugin to generate a .jar file, and place it in the DataX's plugin / reader / yourreader / and plugin / writer / yourwriter / folders;

[0014] Configure the source reader, target writer, schema column mapping, concurrency channel, and error rate limiting in job.json.

[0015] Furthermore, if the data structure is a tree-structured data table during the data structure adjustment, the migration also requires the use of a parent-child data model and a tree traversal algorithm to adjust the data structure. Specifically, this includes: constructing a tree structure using recursive SQL; calculating the left and right values ​​of each node in the tree structure using a recursive traversal algorithm; the left value of each node is less than the left values ​​of all its child nodes, and the right value is greater than the right values ​​of all its child nodes.

[0016] The verification rules for the tree structure data table during migration are as follows: the root node satisfies that the left value is equal to 1; each child node satisfies that the right value is greater than the left value and (right value - left value) % 2 = 1; cross nesting is prohibited, and the depth level and path consistency are satisfied.

[0017] Furthermore, configuring grayscale traffic switching in the configuration center specifically includes:

[0018] Define the traffic allocation ratio for different service versions as the rule for traffic switching, and the configuration center dynamically manages the rule;

[0019] The microservice gateway routes traffic based on the traffic ratio specified in the configuration center. The routing decision parameters include: request ID, timestamp, and IP address.

[0020] During the canary release process, it is necessary to monitor the metrics of both versions in real time. These metrics include: request response time, error rate, and throughput.

[0021] If an anomaly occurs in the new version, traffic will be switched back to the previous version service according to the rollback configuration, and the traffic allocation ratio will be dynamically modified in the configuration center.

[0022] Furthermore, the grayscale switching process employs a difference verification module for consistency determination, which specifically includes:

[0023] Confirm the migration tables of the source system and the target system, split the migration tables into multiple verification blocks, and initialize the block parameters of the verification blocks; record the block parameter tasks of the verification blocks in the diff_meta_relay table, with the initial state being pending verification;

[0024] The thread pool is used to verify each verification block in the pending state. Specifically, this includes: executing a SELECT query task, comparing whether the two hash values ​​are consistent. If they are consistent, the status of the verification block in the diff_meta_relay table is updated to "verification passed", the verification time and hash value are recorded, and the parameter values ​​of the verification block are dynamically adjusted. If the verification results are inconsistent, the status of the verification block in the diff_meta_relay table is updated to "verification failed", the primary key field is written to the alarm system, and the parameter values ​​of the verification block are not updated.

[0025] If the verification execution exceeds the preset time or throws an exception, the status of the verification block in the diff_meta_relay table is updated to verification failure, the size of the verification block is halved, the parameter values ​​of the verification block are dynamically adjusted, and if the number of verifications of the verification block is greater than 3, the verification is stopped and manual processing or system restart is required.

[0026] The table is considered to have passed verification when all verification blocks are in the verification status.

[0027] Furthermore, the waiting for manual processing or system restart may cause a breakpoint in the verification. A breakpoint recovery mechanism is used to connect the data. The breakpoint recovery mechanism includes: reading the status from the diff_meta_relay table at the breakpoint. If the verification is successful, it is skipped; if it is a verification block to be verified or a verification failed, it is re-verified.

[0028] According to a second aspect of the present invention, a computer program product is provided, on which one or more computer programs are stored, which, when executed by a computer processor, implement the method described above.

[0029] The above-described one or more technical solutions in the embodiments of this application have at least one of the following technical effects:

[0030] This invention solves the problem of slow and time-consuming data migration between heterogeneous data sources by configuring a read / write plugin and using DataX to migrate between heterogeneous data sources, while also reducing the difficulty of migrating heterogeneous data sources.

[0031] Ensuring data consistency before and after database migration is crucial. To avoid service interruptions, this invention supports zero-downtime migration while guaranteeing the non-intrusiveness of the migration project on existing business code. During migration, the data structure needs to be flexibly adjusted, especially when tree structures are involved, requiring additional business adjustments. Furthermore, in the event of service anomalies during migration, data rollback must be possible to ensure that new data does not affect the system. Therefore, the entire migration process is meticulously designed to guarantee a smooth system transition and data integrity. Attached Figure Description

[0032] The accompanying drawings are included to provide a further understanding of the embodiments and are incorporated in and constitute a part of this specification. The drawings illustrate embodiments and, together with the description, serve to explain the principles of the invention. Other embodiments and many anticipated advantages of the embodiments will be readily recognized as they become better understood through reference to the following detailed description. Elements in the drawings are not necessarily to scale. The same reference numerals refer to corresponding similar parts.

[0033] Figure 1 A flowchart illustrating a dual-track data migration method based on a DataX plugin according to an embodiment of the present invention is shown.

[0034] Figure 2 A flowchart illustrating a dual-track data migration process according to an embodiment of the present invention is shown.

[0035] Figure 3 A schematic diagram of the process of abnormal data rollback according to an embodiment of the present invention is shown.

[0036] Figure 4 This is a schematic diagram of the structure of a computer system suitable for implementing the electronic devices of the present application embodiments. Detailed Implementation

[0037] The present application will now be described in further detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and not intended to limit it. Furthermore, it should be noted that, for ease of description, only the parts relevant to the invention are shown in the accompanying drawings.

[0038] It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other. This application will now be described in detail with reference to the accompanying drawings and embodiments.

[0039] Figure 1 A flowchart illustrating a dual-track data migration method based on a DataX plugin according to an embodiment of the present invention is shown, as follows: Figure 1 As shown:

[0040] S1. Determine the source system and target system. Based on the DataX plugin specification, customize the Reader plugin and Write plugin. The Reader plugin inherits the BaseReader.ContainerReader class, and the Write plugin inherits the BaseWriter.ContainerWriter class. They are responsible for reading the source data and converting it into a Record, as well as writing the Record to the target system.

[0041] The source system can be a custom JDBC database, NoSQL, or HTTP interface, etc.; the target system can be StarRocks, DolphinDB, or OceanBase, etc.

[0042] Before using the Reader plugin and the Write plugin for data migration, the process also includes compiling and deploying the plugins: using Maven to package the plugins, generating .jar files, and placing them in the DataX's plugin / reader / yourreader / and plugin / writer / yourwriter / folders.

[0043] Write a DataX Job configuration JSON: Configure the source reader, target writer, schema column mapping, concurrency channel, and error rate limiting in job.json; Channel concurrency control is configured through "speed":{"channel":N}, allowing the task to run in parallel N paths.

[0044] Test run in the Dev environment, debug data precision and type conversion, observe performance, and optimize parallelism and runtime block size.

[0045] Small-batch trial runs were conducted to monitor data integrity and failure rate, and the full dataset was gradually migrated.

[0046] In some preferred embodiments, the Reader plugin customization method includes: creating a yourreader directory structure, defining a plugin.json file to describe Reader parameters, such as connection method URL / username / password, and query SQL. Tasks are split in the init / jobSplit / post stages on the Job side, and a connection is obtained in the startRead() method on the Task side, the SQL is executed, the ResultSet is read in a loop, encapsulated into a Record, and sent to the bridge.

[0047] The Writer plugin works similarly. When creating a new writer, parameters are defined such as the target system connection, batch size, and writing API address. In `startWrite()`, it receives batches of data from `RecordReceive` and constructs the target system's write format. It supports batch submissions, retries on failure, and timestamp / field conversion.

[0048] S2. Data migration is performed using the Reader plugin and the Write plugin. The data migration includes a two-track migration process: full migration and incremental migration. Figure 2 As shown, it specifically includes:

[0049] S2.1 Export the data from the source system in one go and import it into the target system, recording the migration start time as T0; during this stage, the business still only accesses the source database, and a small amount of data write delay is allowed within the migration window; full migration ensures that the source table is lock-free or read locks are controllable during migration, avoiding business blocking.

[0050] S2.2 Subscribe to binlog logs via Canal or Debezium, capture DML changes (insert / update / delete, etc.) after T0, and write the changes to the target system. During the change process, data structure adjustments are performed simultaneously.

[0051] During database migration, when encountering tree structures, the processing is adjusted according to different data models. This involves using a parent-child data model and tree traversal algorithms to adjust the data structure. The parent-child data model typically uses a field in the database to represent the ID of the parent node. Each node in the tree structure has a `parent_id` field pointing to its parent node. The root node's `parent_id` is usually NULL. Specifically, this includes: constructing the tree structure using recursive SQL; calculating the left and right values ​​of each node using a recursive traversal algorithm; ensuring that the left value of each node is less than the left values ​​of all its child nodes, and the right value is greater than the right values ​​of all its child nodes.

[0052] The validation rules for tree-structured data tables during migration are as follows: the root node satisfies that the left value is equal to 1; each child node satisfies that the right value is greater than the left value and (right value - left value) % 2 = 1; cross nesting is prohibited, and the depth level and path consistency are satisfied.

[0053] S2.3 Configure grayscale traffic switching in the configuration center, specifically including:

[0054] Define the traffic allocation ratio for different service versions as the rule for traffic switching. For example, version V1: occupies 80% of the traffic, and version V2 occupies 20% of the traffic.

[0055] During canary deployments, the configuration center plays a crucial role, dynamically managing traffic switching rules without modifying microservice code or restarting services. In some preferred embodiments, Nacos is used as the configuration center, and the traffic switching configuration is as follows:

[0056] {

[0057] "flowRules":{

[0058] "serviceV1":{

[0059] "version":"v1",

[0060] "weight":80

[0061] },

[0062] "serviceV2":{

[0063] "version":"v2",

[0064] "weight":20

[0065] }

[0066] }

[0067] }

[0068] This configuration means:

[0069] serviceV1 (version V1) will handle 80% of the request traffic.

[0070] serviceV2 (version V2) will handle 20% of the request traffic.

[0071] The microservice gateway is responsible for routing traffic based on the traffic proportions specified in the configuration center. Traffic routing decisions can be made based on the following:

[0072] Request ID: Traffic allocation is determined based on the hash value of the request ID.

[0073] Timestamp: Dynamically allocates traffic based on the time of the request.

[0074] IP address: Traffic allocation is determined by the user's IP address to ensure that different user groups use the old and new versions of the service in proportion.

[0075] When a request reaches the gateway, the gateway routes the traffic according to the rules in the configuration center. The specific steps are as follows:

[0076] The request enters the gateway, and the gateway determines the proportion of request traffic based on the routing configuration.

[0077] The gateway will route requests to different service versions proportionally. If the traffic ratio is set to 80% and 20%, the gateway will determine which version of the service the request should be routed to based on the request content (such as request ID, user ID, timestamp, etc.).

[0078] For example, on the first request, 80% of the requests will be sent to service V1, and 20% will be sent to service V2. If user A's request hits V2, subsequent requests will continue to hit V2 until the traffic ratio changes.

[0079] A key advantage is the ability to dynamically adjust traffic proportions. Depending on the actual situation, if the new version of the service is stable, the traffic proportion for the V2 service can be gradually increased, for example:

[0080] Assuming the initial traffic ratio is 80% (V1) and 20% (V2), after stabilization, the traffic ratio can be adjusted to 50% (V1) and 50% (V2). Simply update the configuration in the configuration center, and the gateway will automatically route traffic according to the new rules.

[0081] During the canary release process, it's necessary to monitor the metrics of both versions in real time. If performance issues or errors are found in version V2, its traffic proportion can be quickly reduced, or even switched back to V1. Monitored metrics include: request response time: Is the response time of V2 slower than V1? Error rate: Is the error rate of V2 higher than V1? Throughput: Is the number of requests processed by V2 within acceptable limits? If V2's performance is below expectations, traffic can be rolled back by adjusting the weights in the configuration center.

[0082] If issues arise with the new version, immediately switch traffic back to the older version. Simply adjust the traffic ratio in the configuration center to quickly reduce the V2 traffic ratio to 0% and restore the V1 ratio to 100%.

[0083] Proportional traffic allocation is a method for gradually deploying new versions and verifying feature stability. By dynamically managing traffic ratios through a configuration center, traffic distribution can be flexibly controlled, ensuring that the new version can be validated within a limited user group without affecting all users. Simultaneously, the monitoring and rollback mechanisms implemented during the canary release process guarantee service stability and mitigate the risks associated with new version deployments.

[0084] The grayscale switching process uses a difference verification module to determine consistency. The purpose is to verify whether the data in each migration table in the source system database and the target system database is consistent, so as to determine whether grayscale switching can be entered or whether the grayscale switching process is successful.

[0085] In the difference verification module, the parameters must first be preset, including:

[0086] Concurrent threads: 25;

[0087] The query time for each verification block task is 1 second.

[0088] The initial check block size is 1000 lines;

[0089] The minimum check block size is 100 lines, and the maximum is 10,000 lines.

[0090] Expected query time for a single verification block task: within 0.5 seconds;

[0091] Database difference record table: diff_meta_relay;

[0092] The verification process is as follows:

[0093] Step 0: Initialization phase;

[0094] Read the list of table names;

[0095] Establish connections between the source system database and the target system database;

[0096] Verify the consistency of table structures (fields, types, and order, etc.) between the source system database and the target system database; if inconsistent, throw an exception and adjust the data structure.

[0097] Create or update the diff_meta_relay table for each table.

[0098] Step 1: Primary key range query;

[0099] The initial verification block size (e.g., 1000) is set, the migration table is split into multiple verification block tasks, and the block parameters of the verification blocks are initialized. The block parameter tasks of the verification blocks are recorded in the diff_meta_relay table, and the initial state is pending verification.

[0100] Step 2: Execute relay verification tasks concurrently;

[0101] The thread pool is used to verify each verification block in the pending state. Specifically, this includes: executing a SELECT query task, comparing whether the two hash values ​​are consistent. If they are consistent, the status of the verification block in the diff_meta_relay table is updated to "verification passed", the verification time and hash value are recorded, and the parameter values ​​of the verification block are dynamically adjusted (e.g., if it is faster than 0.2 seconds, then the verification block size is multiplied by 2). If the verification results are inconsistent, the status of the verification block in the diff_meta_relay table is updated to "verification failed", the primary key field is written to the alarm system, and the parameter values ​​of the verification block are not updated.

[0102] If the verification execution exceeds the preset time or throws an exception, the status of the verification block in the diff_meta_relay table is updated to verification failure, the size of the verification block is halved, the parameter values ​​of the verification block are dynamically adjusted, and if the number of verifications of the verification block is greater than 3, the verification is stopped and manual processing or system restart is required.

[0103] The table is considered to have passed verification when all verification blocks are in the verification status.

[0104] Step 3: Breakpoint recovery mechanism;

[0105] Waiting for manual processing or system restart may cause breakpoints in the verification process. A breakpoint recovery mechanism is used to restore data continuity. The breakpoint recovery mechanism includes: reading the status from the diff_meta_relay table at the breakpoint; if the verification is successful, skipping it; if it is a verification block that needs to be verified or has failed, re-verifying it.

[0106] Step 4: Detailed comparison of preferred row levels;

[0107] For verification block tasks with inconsistent hash values, nested loops or Map algorithms are used to compare each row; specific difference fields are output; and finally, the differences are recorded in the log table.

[0108] In one specific implementation, the table user, with id ranging from 1 to 10000, is divided into 10 verification block tasks, each with 1000 rows. Ten hash queries are executed concurrently. The third task times out, but succeeds after two retries. The hash value queried by the seventh task is inconsistent, so fine row comparison is initiated. After all verification block tasks pass the verification, the status of the table user is updated to verification complete.

[0109] When all subsidiary tables in the diff_meta_relay table have passed verification and all failed tasks have been manually or automatically corrected, a verification report is output. The difference verification module has completed verification and can proceed to the next grayscale stream.

[0110] S3. When a migration anomaly occurs during the data migration process, the configuration center switches back to the source system, service access is rolled back, the new data of the target system is written back to the source system, the differences of the new data are marked and repaired.

[0111] like Figure 3 As shown, data rollback can automatically switch traffic back to the source system and write back new data (usually user operations) generated in the target system to the source system to ensure uninterrupted business and data consistency.

[0112] During the data migration phase, if DataX fails or the binlog structure is inconsistent, a rollback can be triggered to prevent dirty data from polluting the target database. During the canary rollout phase, the performance of the target system is monitored during the batch rollout process, and if any abnormality is found, the system is quickly switched back. During the fault recovery phase, if write anomalies, structural conflicts, or service unavailability occur, new data is written back to ensure business continuity. If tree structure calculation errors or index anomalies are encountered, the system is switched back to the source system to prevent the spread of errors.

[0113] Based on the above-mentioned method of the present invention, a dual-track data migration system using a DataX plugin is provided. The system relies on DataX at its core to migrate heterogeneous data sources. To facilitate user operation, a graphical interface has been developed, which is compatible with commonly used components and cloud components. Users only need to pull the source and target libraries in the canvas and perform the corresponding configurations according to the instructions to complete a data migration task flow.

[0114] The task JSON file serves as the basis for DataX to schedule tasks. To adapt to various heterogeneous data sources, the parameters of the JSON file are fixed. Only the core parameters and the relevant statements for extraction and data insertion need to be modified to complete the generation of the task JSON file. Subsequently, the read and write plugins of the adapted data source will complete the relevant data conversion, thereby completing the migration of heterogeneous data sources.

[0115] DataX enables rapid migration of heterogeneous data sources, primarily through read and write plugins for the data source. The read plugin reads data from the source database based on the configuration in the task's JSON file, while the write plugin writes the read data from the source database to the target database based on the corresponding configuration in the task's JSON file, thus achieving the migration of heterogeneous data sources.

[0116] The task JSON file configures the relevant configuration information for read and write actions. The read action configured by DataX scheduling uses the read plugin, which, in conjunction with the relevant configuration of the corresponding component, reads the corresponding data. After reading is complete, the write action configured by DataX scheduling uses the write plugin, which, in conjunction with the relevant configuration of the corresponding component, writes the read data, thereby completing the migration task.

[0117] In summary, this invention is applicable to business scenarios involving rapid data migration between multiple heterogeneous data sources. It can quickly respond to project needs, avoid wasting time, and save production capacity and human resources costs, showing great application potential.

[0118] The following is for reference. Figure 4 It shows a schematic diagram of the structure of a computer system 400 suitable for implementing electronic devices according to embodiments of the present application. Figure 4 The electronic device shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments of this application.

[0119] like Figure 4 As shown, the computer system 400 includes a central processing unit (CPU) 401, which can perform various appropriate actions and processes based on programs stored in read-only memory (ROM) 402 or programs loaded from storage section 408 into random access memory (RAM) 403. The RAM 403 also stores various programs and data required for the operation of the system 400. The CPU 401, ROM 402, and RAM 403 are interconnected via a bus 404. An input / output (I / O) interface 405 is also connected to the bus 404.

[0120] The following components are connected to I / O interface 405: an input section 406 including a keyboard, mouse, etc.; an output section 407 including a liquid crystal display (LCD) and speakers, etc.; a storage section 408 including a hard disk, etc.; and a communication section 409 including a network interface card such as a LAN card and a modem, etc. The communication section 409 performs communication processing via a network such as the Internet. A drive 410 is also connected to I / O interface 405 as needed. A removable medium 411, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on drive 410 as needed so that computer programs read from it can be installed into storage section 408 as needed.

[0121] Specifically, according to embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this disclosure include a computer program product comprising a computer program carried on a computer-readable storage medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication section 409, and / or installed from removable medium 411. When the computer program is executed by central processing unit (CPU) 401, it performs the functions defined in the methods of this application. It should be noted that the computer-readable storage medium of this application can be a computer-readable signal medium or a computer-readable storage medium or any combination thereof. The computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this application, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in connection with an instruction execution system, apparatus, or device. In this application, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium can also be any computer-readable storage medium other than a computer-readable storage medium that can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. Program code contained on a computer-readable storage medium may be transmitted using any suitable medium, including but not limited to: wireless, wire, optical fiber, RF, etc., or any suitable combination thereof.

[0122] Computer program code for performing the operations of this application can be written in one or more programming languages ​​or a combination thereof. Programming languages ​​include object-oriented programming languages—such as Java, Smalltalk, and C++—as well as conventional procedural programming languages—such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0123] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0124] The modules described in the embodiments of this application can be implemented in software or in hardware.

[0125] In another aspect, this application also provides a computer-readable storage medium, which may be included in the electronic device described in the above embodiments; or it may exist independently and not assembled into the electronic device. The aforementioned computer-readable storage medium carries one or more programs, which, when executed by the electronic device, cause the electronic device to: determine the source system and the target system; customize the Reader plugin and the Write plugin according to the DataX plugin specification; and perform data migration using the Reader plugin and the Write plugin, specifically including: exporting the data from the source system in one go and importing it into the target system, recording the migration start time as T0; subscribing to binlog logs through Canal or Debezium, capturing DML changes after T0, and writing the changes to the target system, while simultaneously adjusting the data structure during the change process; configuring gray-scale flow switching in the configuration center for traffic switching; and when a migration anomaly occurs during the data migration process, switching back to the source system in the configuration center, rolling back service access, writing the new data from the target system back to the source system, marking the differences in the new data, and repairing them.

[0126] The above description is merely a preferred embodiment of this application and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of the invention involved in this application is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-described inventive concept. For example, technical solutions formed by substituting the above features with (but not limited to) technical features with similar functions disclosed in this application.

Claims

1. A dual-track data migration method based on the DataX plugin, characterized in that, include: S1. Determine the source system and target system, and customize the Reader plugin and Write plugin according to the DataX plugin specification; S2. Data migration is performed using the Reader plugin and the Write plugin, specifically including: S2.1 Export the data from the source system in one go and import it into the target system, recording the migration start time as T0; S2.2 Subscribe to binlog logs via Canal or Debezium, capture DML changes after T0, and write the changes to the target system. During the change process, data structure adjustments are performed simultaneously. S2.3 Configure grayscale flow switching in the configuration center to switch traffic; S3. When a migration anomaly occurs during the data migration process, the configuration center switches back to the source system, service access is rolled back, the new data of the target system is written back to the source system, the differences of the new data are marked and repaired.

2. The dual-track data migration method according to claim 1, characterized in that, The Reader plugin inherits from the BaseReader.ContainerReader class, and the Write plugin inherits from the BaseWriter.ContainerWriter class. The Reader plugin and the Write plugin are used to read data from the source system, convert the data into a Record, and write the Record to the target system.

3. The dual-track data migration method according to claim 1, characterized in that, The process of data migration using the Reader plugin and the Write plugin also includes: Use the Maven packaging plugin to generate a .jar file, and place it in the DataX's plugin / reader / yourreader / and plugin / writer / yourwriter / folders; Configure the source reader, target writer, schema column mapping, concurrency channel, and error rate limiting in job.json.

4. The dual-track data migration method according to claim 1, characterized in that, If the data structure is a tree-structured data table during the data structure adjustment, then the migration also requires the use of a parent-child data model and a tree traversal algorithm to adjust the data structure. Specifically, this includes: constructing a tree structure using recursive SQL; calculating the left and right values ​​of each node in the tree structure using a recursive traversal algorithm; the left value of each node is less than the left values ​​of all its child nodes, and the right value is greater than the right values ​​of all its child nodes.

5. The dual-track data migration method according to claim 4, characterized in that, The verification rules for the tree structure data table during migration are as follows: the root node satisfies that the left value is equal to 1; each child node satisfies that the right value is greater than the left value and (right value - left value) % 2 = 1; Cross-nesting is prohibited, and the depth level and path must be consistent.

6. The dual-track data migration method according to claim 1, characterized in that, The step of configuring grayscale traffic switching in the configuration center specifically includes: Define the traffic allocation ratio for different service versions as the rule for traffic switching, and the configuration center dynamically manages the rule; The microservice gateway routes traffic based on the traffic ratio specified in the configuration center. The routing decision parameters include: request ID, timestamp, and IP address. During the canary release process, it is necessary to monitor the metrics of both versions in real time. These metrics include: request response time, error rate, and throughput. If an anomaly occurs in the new version, traffic will be switched back to the previous version service according to the rollback configuration, and the traffic allocation ratio will be dynamically modified in the configuration center.

7. The dual-track data migration method according to claim 1 or 6, characterized in that, The grayscale switching process uses a difference verification module to determine consistency. This difference verification module specifically includes: Confirm the migration tables of the source system and the target system, split the migration tables into multiple verification blocks, and initialize the block parameters of the verification blocks; record the block parameter tasks of the verification blocks in the diff_meta_relay table, with the initial state being pending verification; The thread pool is used to verify each verification block in the pending state. Specifically, this includes: executing a SELECT query task, comparing whether the two hash values ​​are consistent. If they are consistent, the status of the verification block in the diff_meta_relay table is updated to "verification passed", the verification time and hash value are recorded, and the parameter values ​​of the verification block are dynamically adjusted. If the verification results are inconsistent, the status of the verification block in the diff_meta_relay table is updated to "verification failed", the primary key field is written to the alarm system, and the parameter values ​​of the verification block are not updated. If the verification execution exceeds the preset time or throws an exception, the status of the verification block in the diff_meta_relay table is updated to verification failure, the size of the verification block is halved, the parameter values ​​of the verification block are dynamically adjusted, and if the number of verifications of the verification block is greater than 3, the verification is stopped and manual processing or system restart is required. The table is considered to have passed verification when all verification blocks are in the verification status.

8. The dual-track data migration method according to claim 7, characterized in that, The waiting for manual processing or system restart will cause the verification to breakpoint. A breakpoint recovery mechanism is used to connect the data. The breakpoint recovery mechanism includes: reading the status from the diff_meta_relay table at the breakpoint. If the verification is passed, it is skipped. If it is a verification block to be verified or the verification failed, it is re-verified.

9. A computer program product, characterized in that, It stores a computer program that, when executed by a processor, implements the method as described in any one of claims 1-8.

10. A computing system, characterized in that, It includes a processor and a memory, the processor being configured to perform the method as described in any one of claims 1-8.

Images