Data migration method and device, computer device and storage medium

Abstract

The application relates to a data migration method and device, computer equipment and a storage medium. The method comprises the following steps: based on a preset data table mapping script, performing full data synchronization and incremental data synchronization on a database of data to be migrated out and a database of data to be migrated in; in the case that request side identification information of a target service request for a first service system hits a gray ratio, forwarding the target service request to a second service system; respectively executing service logic corresponding to the target service request in the first service system and the second service system and finding out changed data, performing data checking on the changed data based on the data table mapping script to obtain a data checking result; periodically acquiring the data checking result of a preset period, and in the case that it is determined that the gray ratio needs to be adjusted based on the data checking result in the preset period, adjusting the gray ratio until the gray ratio reaches a target value. The method is beneficial to improving the adaptability of data migration to different data migration scenarios.

Description

Technical Field

[0001] This application relates to the field of distributed database technology, and in particular to a data migration method, apparatus, computer equipment, computer-readable storage medium, and computer program product. Background Technology

[0002] The rapid development of the consumer finance industry necessitates addressing the accumulated technical debt of legacy systems to adapt to the demands of rapid business iteration and improve the overall efficiency and security of financial services. System refactoring can resolve technical debt, enhance system maintainability, support agile business iteration and innovation, reduce long-term costs, and unlock the value of data.

[0003] However, data migration is a significant technical challenge during system restructuring, and in many cases, the development cost of the new system is lower than the cost of data migration. Data migration in related technologies typically includes the following methods: one is downtime migration, where the system is shut down during off-peak periods for migration, and services are restored after the migration is complete; another is to first store the new system's data tables as slaves and then migrate the data using MySQL master-slave synchronization; and yet another is to combine Change Data Capture (CDC) technology with Extract-Transform-Load (ETL) processes to achieve real-time or near real-time data synchronization and processing.

[0004] However, data migration based on downtime migration cannot meet the business requirements of financial systems (financial systems require less than 5 minutes of downtime per year). The prerequisite for data migration based on MySQL master-slave replication is that the master and slave data table structures are completely consistent. However, in actual system reconstruction, there are huge differences in the structure of the new table and the old table. Differences in table structure versions can lead to semantic loss. Data migration based on the CDC+ETL combination scheme can achieve real-time or near real-time data synchronization and processing, but there are security issues in data migration.

[0005] It is evident that the data migration methods used in the relevant business systems have limited adaptability to different data migration scenarios. Summary of the Invention

[0006] Therefore, it is necessary to provide a data migration method, apparatus, computer equipment, computer-readable storage medium, and computer program product that can improve the adaptability of data migration in response to the above-mentioned technical problems.

[0007] Firstly, this application provides a data migration method, including:

[0008] Based on the preset data table mapping script, the first database of data to be migrated out and the second database of data to be migrated in are synchronized with full data and incremental data.

[0009] If the identifier information of the requester for the target business request to the first business system matches the grayscale ratio in the grayscale progress table, the target business request is forwarded to the second business system. The target business request is a non-query type request. The first database is the database of the first business system, and the second database is the database of the second business system.

[0010] The business logic corresponding to the target business request is executed in the first business system and the second business system respectively, causing data changes in the first database and the second database;

[0011] Find the changed data in the first database and the second database respectively, and verify the found changed data based on the data table mapping script to obtain the data verification results;

[0012] Periodically obtain the data verification results within a preset period. If it is determined that the grayscale ratio needs to be adjusted based on the data verification results within the preset period, adjust the grayscale ratio until the grayscale ratio reaches the target value, so that all target business requests for the first business system are forwarded to the second business system.

[0013] Secondly, this application also provides a data migration apparatus, comprising:

[0014] The data synchronization module is used to perform full data synchronization and incremental data synchronization between the first database of data to be migrated out and the second database of data to be migrated in, based on a preset data table mapping script.

[0015] The business request grayscale migration module is used to forward the target business request to the second business system when the identifier information of the requester of the target business request to the first business system matches the grayscale ratio in the grayscale progress table. The target business request is a non-query type request. The first database is the database of the first business system, and the second database is the database of the second business system.

[0016] The data verification module is used to execute the business logic corresponding to the target business request in the first business system and the second business system respectively, so that the data in the first database and the second database will change; to find the changed data in the first database and the second database respectively, and to verify the found changed data based on the data table mapping script to obtain the data verification result;

[0017] The grayscale ratio adjustment module is used to periodically obtain the data verification results within a preset period. If it is determined that the grayscale ratio needs to be adjusted based on the data verification results within the preset period, the grayscale ratio is adjusted until the grayscale ratio reaches the target value, so that all target business requests for the first business system are forwarded to the second business system.

[0018] Thirdly, this application also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in any of the above-described data migration method embodiments.

[0019] Fourthly, this application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps in any of the above-described data migration method embodiments.

[0020] Fifthly, this application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps in any of the above-described data migration method embodiments.

[0021] The aforementioned data migration method, apparatus, computer equipment, computer-readable storage medium, and computer program product, firstly, address the issue of inconsistent data structures between the database to be migrated out and the database to be migrated into during data migration by pre-setting a data table mapping script. Subsequently, based on the data table mapping script, full and incremental data synchronization is performed between the first database containing the data to be migrated out and the second database containing the data to be migrated into, improving the accuracy of data migration. Secondly, for the target business request from the first business system corresponding to the first database, it detects whether the identifier information of the requester of the target business request matches the grayscale ratio in the constructed grayscale progress table. If the identifier information matches the grayscale ratio, the target business request is redirected to the second business system corresponding to the second database. Then, the business logic corresponding to the target business request is executed in both the first and second business systems, respectively, so that the first database... When data changes occur in the first and second databases, the changed data in both databases is retrieved. Based on the data table mapping script, the retrieved changed data is verified to obtain the verification results. Subsequently, the verification results within a preset period are periodically obtained. If the grayscale ratio needs to be adjusted based on the verification results within the preset period, the grayscale ratio is adjusted until the grayscale ratio reaches the target value. This ensures that all target business requests for the first business system are forwarded to the second business system. In this way, by migrating business requests from the first business system to the second business system without downtime, gradually and in stages, the stability of data migration is improved, the impact on actual business is reduced, and the grayscale ratio in the grayscale progress information is adjusted based on the data verification results, which helps to improve the security of data migration and the adaptability to different data migration scenarios. Attached Figure Description

[0022] To more clearly illustrate the technical solutions in the embodiments of this application or related technologies, the drawings used in the description of the embodiments of this application or related technologies will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0023] Figure 1 This is a diagram illustrating the application environment of a data migration method in one embodiment.

[0024] Figure 2 This is a flowchart illustrating a data migration method in one embodiment;

[0025] Figure 3 This is a flowchart illustrating the data migration method in another embodiment;

[0026] Figure 4 This is a schematic diagram of the data verification process in one embodiment;

[0027] Figure 5 This is a flowchart illustrating the data migration method in another embodiment;

[0028] Figure 6 This is a schematic diagram of the grayscale migration process for business requests in one embodiment;

[0029] Figure 7 This is a schematic diagram of the grayscale ratio adjustment process in one embodiment;

[0030] Figure 8 This is a structural block diagram of a data migration device in one embodiment;

[0031] Figure 9 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0032] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0033] The data migration method provided in this application embodiment can be applied to, for example, Figure 1 The application environment shown.

[0034] Specifically, server 102 can perform full and incremental data synchronization on the first database of data to be migrated out and the second database of data to be migrated in, based on a preset data table mapping script. Next, when the identifier information of the requester for a target business request to the first business system matches the grayscale ratio in the grayscale progress table, the target business request is forwarded to the second business system. The target business request is a non-query type request. The first database is the database of the first business system, and the second database is the database of the second business system. Then, the business logic corresponding to the target business request is executed in both the first and second business systems, causing data changes in both databases. The changed data in the first and second databases is then retrieved, and the retrieved changed data is verified based on the data table mapping script to obtain the verification results. Finally, the verification results within a preset period are periodically obtained. If the grayscale ratio needs adjustment based on the verification results within the preset period, the grayscale ratio is adjusted until it reaches the target value, ensuring that all target business requests to the first business system are forwarded to the second business system.

[0035] Among them, server 102 can be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides cloud computing services.

[0036] In one exemplary embodiment, such as Figure 2 As shown, a data migration method is provided, which is applied to... Figure 1 Taking server 102 as an example, the explanation includes the following steps (hereinafter referred to as S): S100 to S500. Wherein:

[0037] S100, based on a preset data table mapping script, performs full data synchronization and incremental data synchronization between the first database of data to be migrated out and the second database of data to be migrated in.

[0038] The full data refers to all data in the first database currently being migrated out. The incremental data refers to newly generated data during the data migration process of the first database.

[0039] In this embodiment, the data table mapping script refers to the script containing the data table structure conversion rules between the first database containing the data to be migrated out and the second database containing the data to be migrated in.

[0040] In this embodiment, the issue arises when the data table structure of a business system undergoes redesign in a business system restructuring scenario, resulting in inconsistencies between the data table structures before and after the restructuring. Based on the differences in data table structures between the first database to be migrated out and the database to be migrated into, conversion rules are pre-written to obtain a data table mapping script. These differences include, but are not limited to, the addition or removal of fields, changes in data types, and different naming conventions.

[0041] In practice, the server can use SQL queries to retrieve the data to be migrated from the first database. Then, it executes a data table mapping script to convert the data into a table structure suitable for the second database. The converted data is then batch-inserted into the second database to complete full data synchronization. Subsequently, the timestamp of the current data synchronization and the position of the binary log are recorded to determine the starting point of incremental synchronization.

[0042] For newly added data during full data synchronization, the server can either retrieve updated data since the last synchronization time based on the timestamp recorded during the last synchronization, execute a data table mapping script to convert the retrieved incremental data into table structures suitable for the second database, and then update or insert the converted data into the second database according to the data type of the incremental data. Alternatively, the server can obtain data update information from the database logs recorded during the last synchronization, retrieve incremental data from the first database based on the data change information, execute a data table mapping script to convert the retrieved incremental data into table structures suitable for the second database, and then update or insert the converted data into the second database according to the data type of the incremental data, thus completing the incremental data synchronization.

[0043] S200: If the identification information of the requester of the target business request to the first business system matches the gray scale ratio in the gray scale progress table, the target business request is forwarded to the second business system. The target business request is a non-query type request. The first database is the database of the first business system, and the second database is the database of the second business system.

[0044] The identification information may include, but is not limited to, user identifiers (user IDs), device identifiers, IP addresses, and order identifiers. The grayscale ratio can be the proportion of selected target business requests that are directed to the second business system; the grayscale progress table is the table structure that stores the grayscale ratios. Non-query type requests may include, but are not limited to, requests for data write, update, or delete operations, or may include composite requests containing data queries and at least one of data write, update, or delete operations.

[0045] In this embodiment, the first business system can be the old business system before the system reconstruction, and the second business system can be the new business system after the system reconstruction.

[0046] In practical applications, when migrating target business requests during business system upgrades or restructuring, in order to reduce risks and ensure business continuity and data consistency, a gray-scale schedule can be built in the distributed cache, and a gray-scale ratio can be set for the migration of small batches of target business requests. For example, in the initial stage of migration, the gray-scale ratio can be set to 5%.

[0047] In practice, the server can perform a hash operation on the identifier information of the target business request entering the first business system, convert the hash result into a percentage value, and check whether the converted hash result is less than the grayscale ratio. If the converted hash result is less than the grayscale ratio, it is determined that the identifier information of the requester of the target business request matches the grayscale ratio in the grayscale progress table. For example, if user A initiates a request through a front-end or upstream service to change user A's account information, the server determines that the business request involves a write operation, thus identifying it as the target business request. Subsequently, the target business request is routed to the second business system according to preset routing rules.

[0048] S300 executes the business logic corresponding to the target business request in the first business system and the second business system respectively, causing data changes in the first database and the second database.

[0049] In practice, after the server forwards the target business request to the second business system, the second business system executes the corresponding business logic for the target business request, causing data changes in the second database. Subsequently, the execution result of the business request is returned to the first business system. After the first business system obtains the execution result of the business request returned by the second business system, it forwards the result to the internal interface in the first business system used to implement the business logic to execute the corresponding business logic of the target business request through asynchronous means (such as message queue implementation). Finally, the execution result of the business request returned by the second business system is passed through to the requester.

[0050] For example, in response to a request to change account information, the second business system performs permission verification and account information format verification. After passing both verifications, it updates the user table in the second database to modify the account information. Subsequently, the second business system returns the business request execution result to the first business system. This result includes, but is not limited to, whether the business request was successfully executed and the corresponding database changes. Upon receiving the result, the first business system asynchronously sends it to an internal service or interface via a message queue (such as Kafka or RocketMQ). This allows the internal service or interface to execute relevant business logic, updating the data in the first database. During asynchronous processing, the first business system also transparently transmits the business request execution result back to the requesting party.

[0051] S400: Locate the changed data in the first database and the second database respectively, and verify the found changed data based on the data table mapping script to obtain the data verification result.

[0052] In specific implementation, after the first and second business systems execute the corresponding business logic in response to the target business request, the data in the first and second databases changes. The server obtains the change records of the data tables in real time. Based on the change records, it queries the first and second databases respectively to retrieve the changed data A and C in response to the target business request. Subsequently, a data table mapping script is executed on the changed data in the first database to convert the changed data into changed data B that conforms to the data structure of the second database. Then, the converted changed data B is compared with the changed data C in the second database to obtain the data verification result, which is recorded in a data table used to store the data verification result. The table structure may include, but is not limited to, the request serial number of the target business request, the theoretical result, the actual result, the data verification value, and the data verification time. The request serial number is used to identify the target business request. The theoretical result corresponds to the changed data B after the data table mapping script is executed on the changed data. The actual result corresponds to the changed data C in the second database. The data verification value represents the result value of the sequential comparison between changed data B and changed data C, which may include consistency or inconsistency. The data verification time represents the time point when the data verification occurred. The actual results and theoretical results may be stored as JSON strings of a batch of data, because it may involve changes to a batch of data tables.

[0053] S500 periodically obtains the data verification results within a preset period. If it is determined that the grayscale ratio needs to be adjusted based on the data verification results within the preset period, the grayscale ratio is adjusted until the grayscale ratio reaches the target value, so that all target business requests for the first business system are forwarded to the second business system.

[0054] The target value can be 100%, which is used to switch business requests from the first business system to the second business system.

[0055] In practice, the operator can pre-set a scheduled task. This task controls the server to calculate the percentage of inconsistent data verification results within a preset period, according to a pre-defined frequency. For example, the scheduled task can be set to start every other day, and its content can be to calculate the percentage of inconsistent data verification results in the most recent 24 hours, representing the percentage of abnormal data.

[0056] The grayscale ratio needs to be adjusted based on the data verification results within a preset period. This can be achieved by comparing the percentage of abnormal data counted within the preset period with a preset minimum abnormal data percentage threshold. If the percentage of abnormal data is less than the preset minimum abnormal data percentage threshold, it indicates that the performance of the second business system is relatively stable. In this case, the grayscale ratio needs to be adjusted to increase the grayscale ratio and forward more target business requests to the second business system for execution.

[0057] Subsequently, the grayscale ratio can be increased in preset steps (e.g., 5%) or based on the proportion of abnormal data; this application does not limit this. Thus, through scheduled tasks, the stability and accuracy of the second business system's functions are verified based on data verification results. The grayscale ratio is increased in stages until the target value is reached. The first business system only carries the forwarding logic of business requests and no longer processes the business logic of business requests. Subsequently, the interface calls of surrounding systems are gradually switched to the second business system, enabling the second business system to independently handle all business requests, and the first business system is taken offline.

[0058] In the aforementioned data migration method, firstly, to address the inconsistency in data structures between the database to be migrated out and the database to be migrated into, a data table mapping script is pre-set. Then, based on the data table mapping script, full and incremental data synchronization is performed between the first database (for data to be migrated out) and the second database (for data to be migrated into), improving the accuracy of the data migration. Secondly, for the target business request from the first business system corresponding to the first database, it is checked whether the identifier information of the requester matches the grayscale ratio in the constructed grayscale progress table. If the identifier information matches the grayscale ratio, the target business request is redirected to the second business system corresponding to the second database. Subsequently, the business logic corresponding to the target business request is executed in both the first and second business systems, causing data changes in both databases. The changed data in the first and second databases are identified separately, and the identified changed data is verified based on the data table mapping script to obtain the data verification results. Then, the data verification results within a preset period are obtained periodically. If the grayscale ratio needs to be adjusted based on the data verification results within the preset period, the grayscale ratio is adjusted until the grayscale ratio reaches the target value, so that all target business requests for the first business system are forwarded to the second business system. In this way, by migrating business requests from the first business system to the second business system in a non-disruptive, gradual, and phased manner, the stability of data migration is improved, the impact on actual business is reduced, and the grayscale ratio in the grayscale progress information is adjusted based on the data verification results, which helps to improve the security of data migration and the adaptability to different data migration scenarios.

[0059] To improve the accuracy of data migration, data simulation verification is performed. In one exemplary embodiment, such as... Figure 3 As shown, after performing full data synchronization and incremental data synchronization between the first database and the second database, the method further includes steps S102 to S106. Wherein:

[0060] S102, for a query type request from the first business system, execute the business logic corresponding to the query type request in both the first and second business systems to obtain the first business execution result and the second business execution result.

[0061] Among them, query-type requests can be requests involving data retrieval. The first business execution result includes the data retrieved by the first business system after executing the business logic corresponding to the query-type request, and the second business execution result includes the data retrieved by the second business system after executing the business logic corresponding to the query-type request.

[0062] In practical applications, such as Figure 4 As shown, a simulation routing module and a simulation verification module can be pre-deployed in the first business system. The simulation routing module is used to route and forward business requests according to preset routing rules, and the simulation verification module is used to perform data verification.

[0063] In practice, the requesting party can initiate a query-type request carrying a business serial number to the first business system. The simulation routing module of the first business system, according to preset routing rules, first executes the corresponding business logic within the first business system to obtain the first business execution result. This first business execution result is then written to a distributed cache using the business serial number as a unique key for subsequent verification. Subsequently, the query-type request is routed to the second business system. The second business system returns the second business execution result obtained after executing the query-type request to the simulation verification module of the first business system, and simultaneously routes the first business execution result to the upstream requesting party. For non-query-type requests, the first business system executes the corresponding business logic for the non-query-type request and returns the business execution result to the upstream requesting party.

[0064] For example, a user initiates a "query account status" request to the first business system through the front end. The simulation routing module determines that the request is a query type request. The first business system retrieves the current status of the user's account from the first database. The current status may include, but is not limited to, balance, status, and transaction records. Subsequently, the retrieved account status information is written into a distributed cache and returned to the requester. Then, the "query account status" request is routed to the second business system. The second business system executes the "query account status" request, retrieves the account status information from the second database, and returns the retrieved account status information to the simulation verification module of the first business system.

[0065] S104, perform data verification on the first business execution result and the second business execution result to obtain the data verification result.

[0066] The data verification results may include whether the parameters in the first business execution result and the second business execution result are consistent.

[0067] In practice, the simulation verification module retrieves the first business execution result from the distributed cache based on the business serial number, and compares the first business execution result with the received second business execution result parameter by parameter to determine whether they are consistent, thus obtaining the data verification result.

[0068] S106, If the data verification results represent inconsistent data, adjust the data table mapping script.

[0069] In this embodiment, during the data migration process, due to the large number of fields involved and the significant differences in data table structures, data simulation verification is required to ensure the accuracy of the data migration and to validate the accuracy of the data table mapping script. During system reconstruction, the interfaces of the second business system are categorized according to the second database operation type, and can be divided into two types: one is a query structure involving only query logic, and the other is an aggregation interface containing CRUD operations. Since it is a system reconstruction, all interfaces may have code errors. Therefore, data verification is necessary. If the data verification results indicate inconsistent parameters, it means that at least one error exists in the logic of the data table mapping script and the second business system's interface, requiring manual correction.

[0070] In practice, when inconsistencies in business execution results are detected, verification information can be pushed to enable operators to adjust data table mapping scripts or interface logic of the second business system.

[0071] Based on the preset data table mapping script, the second database to be migrated performs full data synchronization and incremental data synchronization. It also includes: based on the adjusted data table mapping script, performing incremental data synchronization between the first database and the second database.

[0072] In practical implementation, incremental data synchronization also needs to be performed simultaneously during the aforementioned data simulation and verification process. Therefore, based on the adjusted data table mapping script, incremental data synchronization is performed between the first database and the second database. Referring to the implementation method of incremental data synchronization between the first database of data to be migrated out and the second database of data to be migrated in, based on the preset data table mapping script in the above embodiment, it will not be repeated here.

[0073] In this embodiment, the accuracy of the second business system is verified through data simulation and incremental data synchronization, thereby improving the accuracy of system migration.

[0074] In an exemplary embodiment, based on a preset data table mapping script, a full data synchronization is performed between the first database and the second database, including steps S120 to S140, wherein:

[0075] S120, partition the data to be migrated in the first database and determine multiple first partition information.

[0076] The first partition information may include the full range of data to be migrated within the partition.

[0077] In practice, this can be achieved through distributed worker nodes for full data synchronization from the first database to the second database. Specifically, the management node within the distributed worker nodes can use a batch processing framework to shard the data tables in the first database by primary key or timestamp, breaking down the full synchronization task into multiple sub-data synchronization tasks, thus obtaining the data range to be migrated for each sub-data synchronization task. The batch processing framework can include, but is not limited to, Apache Spark and Apache Flink frameworks.

[0078] S140, the information of each first partition is distributed to the distributed worker nodes so that the distributed worker nodes can perform full data synchronization between the first database and the second database according to the preset data mapping script and the first partition information.

[0079] In practice, each shard can be processed by a distributed worker node, with each node handling data synchronization tasks in parallel. For each distributed worker node, based on the first partition information, the range of data to be migrated in its assigned first database is determined. The node then retrieves the data to be migrated from the first database based on this range within the partition. A data table mapping script is executed on the retrieved data to convert it into data that matches the data table structure of the second database. The converted data is then batch-inserted into the second database to complete the full data synchronization. After full data synchronization, the timestamp of the current data synchronization and the position of the binary log can be recorded to determine the starting point of incremental synchronization.

[0080] In this embodiment, partitioning improves the efficiency of full data synchronization, while data transformation based on the data table migration script improves the accuracy of data migration.

[0081] In an exemplary embodiment, incremental data synchronization is performed between the first database and the second database based on a preset data table mapping script, including steps S160 to S180, wherein:

[0082] S160, perform partitioning on the incremental data in the first database to determine multiple second partition information.

[0083] S180, the information of each second partition is distributed to the distributed worker nodes so that the distributed worker nodes can perform incremental data synchronization between the first database and the second database according to the preset data mapping script and the second partition information.

[0084] The second partition information may include data change records of incremental data within the partition.

[0085] In practice, for newly added data during full data synchronization, the management node determines the data change records of the incremental data in the first database based on the timestamp and binary log position recorded during the last synchronization. The data change records are then sharded using a batch processing framework, with each shard processed by a distributed worker node. For each distributed worker node, based on the data change records of the incremental data within its assigned shard, it queries the first database for the corresponding incremental data. A data table mapping script is then executed on the retrieved incremental data to convert it into data that matches the data table structure of the second database. The converted data is then written to the second database in batches to complete the incremental data synchronization.

[0086] Understandably, the incremental data synchronization process involves a continuous loop to determine the next incremental data to be migrated. This can be done based on the timestamp recorded during the previous synchronization and the position of the binary log, following the incremental data synchronization process described above. It will not be repeated here.

[0087] In this embodiment, partitioning improves the efficiency of incremental data synchronization, while data transformation based on the data table migration script improves the accuracy of data migration.

[0088] In one exemplary embodiment, such as Figure 5 As shown, the grayscale progress table includes grayscale ratios corresponding to different system interfaces; it detects whether the identifier information of the requester for the target business request of the first business system matches the grayscale ratio in the grayscale progress table, including S220 to S260, wherein:

[0089] S220, perform a hash operation on the identifier information of the requester of the target business request for the first business system to obtain a hash value, and take the remainder of the hash value with a first preset value to obtain the remainder result.

[0090] The first preset value is used to map the hash value to a preset numerical range, and this first preset value is set according to the migration accuracy requirements of the business request. Identification information may include, but is not limited to, user identifiers (user IDs), IP addresses, and order identifiers.

[0091] In practical applications, the preset value can be set according to the accuracy requirements of the grayscale migration of business requests in the business system. For example, if the first preset value is set to 10000, then the first preset value is used to map the hash value to the integer range of 0 to 9999. It should be understood that the first preset value set here is only an example and is not intended to be the only limitation.

[0092] In this embodiment, as Figure 6 As shown, the grayscale control module of the first business system is used to determine whether the requester's identification information matches the grayscale ratio, to perform routing operations on the target business requests entering the first business system, and to maintain a grayscale progress table in a distributed cache (Redis). Specifically, a Map structure in Redis is used in advance to store the grayscale ratios corresponding to different system interfaces. The key of the Map is the system interface identifier, and the value is the grayscale ratio. For example, as shown in the table below, the grayscale ratios of system interface 1, system interface 2, and system interface n are 0.01%, 10%, and 60%, respectively.

[0093] System interface identifier Grayscale ratio Interface 1 0.01% Interface 2 10% ... … Interface n 60%

[0094] For example, such as Figure 6 As shown, after a target business request enters the first business system, the grayscale control module performs a hash operation on the user identifier of the requester to obtain a hash value. Subsequently, the hash value is moduloed by a first preset value to obtain the remainder result. For example, the hash value is moduloed by 10000.

[0095] S240, determine the grayscale value corresponding to each grayscale ratio.

[0096] Among them, the grayscale value is used to detect whether the identification information matches the grayscale ratio.

[0097] In specific implementation, the grayscale value corresponding to each grayscale ratio can be determined by multiplying the first preset value by the grayscale ratio. For example, the grayscale value corresponding to the grayscale ratio of interface 1 is 1, the grayscale value corresponding to the grayscale ratio of interface 2 is 1000, and the grayscale value corresponding to the grayscale ratio of interface n is 6000.

[0098] S260, compare the remainder result with each gray value in turn. If the remainder result is less than the gray value, determine that the identification information matches the gray ratio corresponding to the gray value.

[0099] In practice, the grayscale adjustment module sequentially compares the remainder result with each grayscale value. If the remainder result is less than the grayscale value, it determines that the identification information matches the grayscale ratio corresponding to the grayscale value. For example, if the remainder result is less than 1, it determines that the identification information matches the grayscale ratio of interface 1; if the remainder result is less than 1000, it determines that the identification information matches the grayscale ratio of interface 2; and if the remainder result is less than 6000, it determines that the identification information matches the grayscale ratio of interface n.

[0100] In other embodiments, if the grayscale control module determines that the identification information matches the grayscale ratio in the grayscale progress table, it first forwards the target business request to the second business system. The second business system executes the corresponding business logic of the target business request to cause data changes in the second database. Subsequently, the second business system returns the execution result of the second business request to the first business system. After obtaining the execution result of the second business request returned by the second business system, the first business system forwards the execution result of the second business request to the internal interface in the first business system used to implement the business logic in an asynchronous manner (e.g., through a message queue). The internal interface executes the corresponding business logic of the target business request. At the same time, the grayscale control module transparently transmits the execution result of the second business request to the requester.

[0101] In other embodiments, if the grayscale control module determines that the identification information does not match the grayscale ratio in the grayscale progress table, the target business request is routed to the business interface of the first business system so that the business interface executes the target business request and returns the execution result of the first business request to the requester.

[0102] In this embodiment, migrating business requests based on identification information and a grayscale progress table helps improve migration efficiency.

[0103] In an exemplary embodiment, the data verification result includes the verification result value; if it is determined that the grayscale ratio needs to be adjusted based on the data verification result within a preset period, the grayscale ratio is adjusted, including S520 to S580, wherein:

[0104] S520, for each system interface, calculates the ratio of target verification result values ​​that represent inconsistent data among the verification result values ​​corresponding to the system interface within a preset period.

[0105] The ratio represents the proportion of the inconsistent target verification result value among the verification result values ​​corresponding to the system interface. The data verification result also includes a system interface identifier, used after data verification to store the identifier of the system interface corresponding to the grayscale ratio matched by the requester's identifier information.

[0106] In practice, operators can pre-set scheduled tasks. These tasks control the server to calculate the ratio of inconsistent target verification result values ​​among the verification result values ​​corresponding to system interfaces within a preset period, according to a pre-defined frequency. For example, the frequency can be set to start the scheduled task every other day. The task content is to calculate the ratio of target verification values ​​for each interface within the most recent 24 hours. For each system interface, the grayscale ratio is determined based on the ratio corresponding to the system interface to determine whether it needs adjustment. If adjustment is deemed necessary, the grayscale ratio is adjusted.

[0107] For example, such as Figure 6 As shown, when the scheduled task starts, the data verification and compensation module filters out the verification result value for each system interface in the grayscale progress table according to the system interface identifier in the data verification result, and calculates the ratio of the target verification result value.

[0108] S540, if the ratio is greater than the preset maximum error tolerance rate, the grayscale ratio of the system interface will be modified to the second preset value.

[0109] The maximum error tolerance rate represents the ratio of the largest acceptable target verification result value. The second preset value can be 0.

[0110] In practical applications, the maximum error tolerance rate can be pre-configured based on data migration requirements. For example, the maximum error tolerance rate can be pre-configured as a dynamic setting of 5%.

[0111] In specific implementation, such as Figure 7 As shown, for each system interface, the data verification and compensation module can compare the ratio with the maximum error tolerance rate. If the ratio is greater than the maximum error tolerance rate, it indicates that the data error caused by the migration of business requests is more serious. In this case, the grayscale ratio of the current system interface is modified to 0 to stop the forwarding of business requests of the system interface.

[0112] S560 maintains the grayscale ratio of the system interface when the ratio is less than or equal to the preset maximum error tolerance rate and greater than the preset minimum error tolerance rate.

[0113] Among them, the minimum error tolerance rate represents the ratio of the smallest tolerable target verification result value.

[0114] In practical applications, the minimum error tolerance rate can be pre-configured based on data migration requirements. For example, the minimum error tolerance rate can be pre-configured dynamically as 0.1%.

[0115] In practice, when the ratio of the current system interface is less than or equal to the preset maximum error tolerance rate, the data verification and compensation module compares the ratio with the minimum error tolerance rate. If the ratio is greater than the minimum error tolerance rate, it indicates that the migration of business requests of the current system interface is relatively stable, and the grayscale ratio of the current system interface is not modified.

[0116] S580, when the ratio is less than or equal to the preset minimum error tolerance rate and the grayscale ratio of the interface is greater than the preset maximum single grayscale increment value, adjusts the grayscale ratio of the interface to the sum of the current grayscale ratio and the maximum single grayscale increment value. Understandably, when the adjusted grayscale ratio is greater than 100%, the adjusted grayscale ratio is set to 100%.

[0117] Among them, the maximum single grayscale increase value represents the upper limit of the single grayscale ratio increase value of the system interface.

[0118] In practical applications, the maximum single-scale increment value can be pre-configured based on data migration requirements. For example, the maximum single-scale increment value can be dynamically configured to be 10%.

[0119] In specific implementation, such as Figure 7 As shown, when the data verification and compensation module determines that the ratio is less than or equal to the preset minimum error tolerance rate, it compares the current interface's grayscale ratio with the preset maximum single grayscale increase value. If the grayscale ratio is greater than the preset maximum single grayscale increase value, the current interface's grayscale ratio is adjusted to the sum of the current grayscale ratio and the maximum single grayscale increase value to control the increase in grayscale ratio and improve migration stability. Understandably, when the adjusted grayscale ratio is greater than 100%, the adjusted grayscale ratio is set to 100%.

[0120] In this embodiment, by periodically statistically analyzing the ratio of inconsistent data verification results, the gray-scale migration progress of business requests is dynamically adjusted according to the ratio. This improves the adaptability to different migration scenarios, enhances the stability of gray-scale migration of business requests, and thus improves the stability of the second business system.

[0121] By using the grayscale ratio of the grayscale schedule and the identifier information of the requester, business requests are gradually migrated to the second business system, ensuring a smooth and reliable migration process, which helps improve the continuity and stability of business.

[0122] In an exemplary embodiment, adjusting the grayscale ratio of the interface based on the data verification results of the grayscale traffic of the hit interface further includes: adjusting the grayscale ratio of the interface to a target multiple of the current grayscale ratio when the ratio is less than or equal to a preset minimum error tolerance rate and the grayscale ratio of the interface is less than or equal to a preset maximum single grayscale increase value.

[0123] In specific implementation, such as Figure 7 As shown, when the judgment ratio is less than or equal to the preset maximum error tolerance rate, the data verification and compensation module compares the current interface's grayscale ratio with the preset maximum single grayscale increment value. If the ratio is less than or equal to the preset maximum single grayscale increment value, it indicates a high consistency between the business execution results of the first and second business systems, allowing for an increase in the business request diversion ratio. In this case, the grayscale ratio of the current interface is adjusted to the target number of digits (e.g., twice the current grayscale ratio). Understandably, when the adjusted grayscale ratio is greater than 100%, the adjusted grayscale ratio is set to 100%.

[0124] In this embodiment, the gray-scale migration progress of business requests is dynamically adjusted according to the ratio, which improves the adaptability to different migration scenarios and helps to improve the stability of gray-scale migration of business requests, thereby improving the stability of the second business system.

[0125] In other embodiments, during incremental data synchronization, for target service requests that match the grayscale ratio, the incremental data corresponding to the target service request is directly discarded to avoid duplicate writing between the two links. For target service requests that do not match the grayscale ratio, incremental data synchronization is performed normally.

[0126] In an exemplary embodiment, the data to be migrated in the first database is partitioned to determine multiple first partition information, including S122 to S124, wherein:

[0127] S122, obtain the amount of data to be migrated and the number of worker nodes, and record the initial position of the binlog log of the first database.

[0128] S124, based on the amount of data to be migrated and the number of distributed worker nodes, the data to be migrated is evenly divided into multiple partitions according to the primary key ID, and the primary key ID range of each partition is obtained. The information of the first partition includes the primary key ID range.

[0129] The MySQL Binlog is a binary log file used to record internal changes made to the database (data modification operations) within MySQL. The initial position of the binlog indicates the position of the binlog before full data synchronization.

[0130] In practical applications, the management node in the distributed worker nodes can record the initial position of the binlog of the first database for subsequent incremental data synchronization. Then, the management node can query the total amount of data (data to be migrated) in the first database based on the primary key ID.

[0131] In practice, the management node, based on the data volume and the number of distributed worker nodes, uses the database's Spring Batch remote sharding feature to evenly divide the data to be migrated into multiple partitions according to the primary key ID. Each partition contains the same amount of data to be migrated, thus obtaining the primary key ID range for each partition. For example, if there are N distributed worker nodes, the data to be migrated will be divided into N partitions. Spring Batch is a lightweight and powerful batch processing framework provided by the Spring framework.

[0132] The information of each first partition is distributed to the distributed worker nodes, so that the distributed worker nodes can perform full data synchronization between the first database and the second database according to the preset data mapping script and the first partition information, including S142, wherein:

[0133] S142, distribute the primary key ID range of each partition to each worker node, so that each worker node can query the data in the first database according to its corresponding primary key ID range, execute the preset data table mapping script on the queried data, obtain new table data, and insert the new table data into the second database.

[0134] In practice, the management node sends the partition information, which includes the primary key ID range of the partition, to the distributed worker nodes one by one. Each distributed worker node is responsible for handling and executing the data synchronization within its respective partition.

[0135] Subsequently, the distributed worker nodes receive their respective primary key ID ranges and process the full data synchronization in parallel. Specifically, for each distributed worker node, within the primary key ID range of its assigned partition, it uses a paginated query to retrieve a portion of the data at a time. It then executes a data table mapping script on the retrieved data in the first database, converting the data into new table data conforming to the data table structure of the second database. The converted new table data is then batch-inserted into the second database. For example, if each page contains 1000 records, and the query conditions include the primary key ID range of the partitions. The management node, also a distributed worker node, also needs to perform data synchronization based on the partition information.

[0136] In this embodiment, the full data synchronization task is divided and scheduled by the management node, enabling distributed nodes to perform data mapping and writing in parallel according to the data table mapping script, thereby improving the efficiency and accuracy of data migration.

[0137] In an exemplary embodiment, incremental data in the first database is partitioned to determine multiple second partition information, including S162 to S164, wherein:

[0138] S162, obtain the current position of the binlog in the first database, and determine the target binlog data between the initial position and the current position.

[0139] S164, based on the number of distributed worker nodes, the target binlog data is evenly divided into multiple partitions to obtain the binlog data range of each partition. The second partition information includes the binlog data range.

[0140] In practice, during the full data synchronization process, the management node obtains the current position of the binlog log of the first database, takes the initial position as the start position and the current position as the end position, and determines the target binlog data between the start position and the end position. The target binlog data is used to obtain the incremental data generated during the data synchronization process.

[0141] Subsequently, based on the number of distributed worker nodes, the management node uses the database's SpringBatch remote sharding feature to divide the incremental data synchronization task into multiple partitions. Each partition independently processes a portion of the incremental data, achieving parallel incremental data synchronization. Specifically, the management node evenly divides the target binlog data into multiple partitions based on the number of distributed worker nodes, with each partition containing the same amount of target binlog data, thus obtaining the binlog data range for each partition. For example, if the number of distributed worker nodes is N, the target binlog data is divided into N partitions.

[0142] The information of each second partition is distributed to each distributed worker node, so that each distributed worker node performs incremental data synchronization between the first database and the second database according to the preset data mapping script and the second partition information, including S182 to S186, wherein:

[0143] S182, distribute the binlog data range of each partition to each worker node, so that each worker node can read the binlog data within its corresponding binlog data range, parse the read binlog data into data table data, execute the preset data table mapping script on the data table data, and obtain new table data.

[0144] S184: If data with the same primary key as the data in the new table is found in the second database, update the second database according to the data in the new table.

[0145] S186: If no data with the same primary key as the new table data is found in the second database, insert the new table data into the second database.

[0146] In practice, the management node sends the second partition information of multiple binlog data ranges containing partitions to the distributed worker nodes one by one, and each distributed worker node is responsible for handling the incremental data synchronization within its own partition.

[0147] Subsequently, the distributed worker nodes receive their respective binlog data ranges and process incremental data synchronization in parallel. Specifically, for each distributed worker node, within the binlog data range of its assigned partition, it uses a binlog parsing tool to batch read the binlog data, parses the read binlog data, extracts incremental data change records, and then obtains the corresponding incremental data (data table data) based on these records. Next, it executes a data table mapping script to convert the data table data into new table data conforming to the data table structure of the second database. Then, it checks if data with the same primary key as the new table data exists in the second database. If so, it performs an update operation to update the new table data in the second database. If no such data exists, it performs an insert operation to batch insert the new table data into the second database. The management node, also a distributed worker node, also needs to perform data synchronization based on the second partition information.

[0148] Then, return to execute S162 to S186 to achieve continuous, cyclical incremental data synchronization. In S162, the current position of the binlog is obtained, and this current position is determined as the end position. The current position of the binlog recorded during the last synchronization is determined as the initial position, and the target binlog data between the initial position and the current position is determined.

[0149] In this embodiment, data mapping is performed through a data table mapping script, and incremental data synchronization tasks are processed by partitioning, which improves the efficiency and accuracy of data migration.

[0150] To provide a clearer explanation of the data migration method provided in this application, a specific embodiment is described below, which includes the following steps:

[0151] S1. Obtain the data volume and number of worker nodes of the data to be migrated, record the initial position of the binlog of the first database, and based on the data volume and number of distributed worker nodes of the data to be migrated, divide the data to be migrated into multiple partitions evenly according to the primary key ID, and obtain the primary key ID range of each partition. The information of the first partition includes the primary key ID range.

[0152] S2, distribute the primary key ID range of each partition to each distributed worker node, so that each distributed worker node can query the data in the first database according to its corresponding primary key ID range, execute the preset data table mapping script on the queried data, obtain the new table data, and insert the new table data into the second database.

[0153] S3, obtain the current position of the binlog log of the first database, determine the target binlog data between the initial position and the current position, and divide the target binlog data evenly into multiple partitions based on the number of distributed worker nodes to obtain the binlog data range of each partition. The second partition information includes the binlog data range.

[0154] S4 distributes the binlog data range of each partition to each worker node, so that each worker node can read the binlog data within its corresponding binlog data range, parse the read binlog data into data table data, execute a preset data table mapping script on the data table data to obtain new table data, if data with the same primary key as the new table data exists in the second database, update the second database according to the new table data, if data with the same primary key as the new table data does not exist in the second database, insert the new table data into the second database.

[0155] S5, for a query type request from the first business system, execute the corresponding business logic in the first business system and the second business system respectively to obtain the first business execution result and the second business execution result. Perform data verification on the first business execution result and the second business execution result to obtain the data verification result. If the data verification result represents inconsistent data, adjust the data table mapping script.

[0156] S6. Perform a hash operation on the identifier information of the requester for the target business request of the first business system to obtain a hash value. Take the remainder of the hash value with the first preset value to obtain the remainder result. Determine the gray value corresponding to each gray scale ratio. Compare the remainder result with each gray value in turn. If the remainder result is less than the gray value, it is determined that the identifier information matches the gray scale ratio corresponding to the gray value.

[0157] S7. If the identification information of the requester of the target business request to the first business system matches the gray scale ratio in the gray scale progress table, the target business request is forwarded to the second business system. The target business request is a non-query type request. The first database is the database of the first business system, and the second database is the database of the second business system.

[0158] S8 executes the business logic corresponding to the target business request in the first business system and the second business system respectively, causing data changes in the first database and the second database. It then finds the changed data in the first database and the second database respectively, and performs data verification on the found changed data based on the data table mapping script to obtain the data verification result.

[0159] S9: Periodically acquire data verification results within a preset period. For each system interface, calculate the ratio of the target verification result values ​​representing inconsistent data among the verification result values ​​corresponding to the system interface within the preset period. If the ratio is greater than the preset maximum error tolerance rate, modify the grayscale ratio of the system interface to a second preset value. If the ratio is less than or equal to the preset maximum error tolerance rate and greater than the preset minimum error tolerance rate, maintain the grayscale ratio of the system interface.

[0160] S10, for each system interface, if the ratio is less than or equal to the preset minimum error tolerance rate and the grayscale ratio of the interface is greater than the preset maximum single grayscale increment value, adjust the grayscale ratio of the interface to the sum of the current grayscale ratio and the maximum single grayscale increment value; if the ratio is less than or equal to the preset minimum error tolerance rate and the grayscale ratio of the interface is less than or equal to the preset maximum single grayscale increment value, adjust the grayscale ratio of the interface to the target multiple of the current grayscale ratio.

[0161] S11, until the grayscale ratio reaches the target value, so that all target business requests for the first business system are forwarded to the second business system.

[0162] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0163] In one exemplary embodiment, such as Figure 8 As shown, a data migration device 600 is provided, including: a data synchronization module 610, a service request grayscale migration module 620, a data verification module 630, and a grayscale ratio adjustment module 640, wherein:

[0164] The data synchronization module 610 is used to perform full data synchronization and incremental data synchronization between the first database of data to be migrated out and the second database of data to be migrated in, based on a preset data table mapping script.

[0165] The business request grayscale migration module 620 is used to forward the target business request to the second business system when it is detected that the identifier information of the requester of the target business request to the first business system matches the grayscale ratio in the grayscale progress table. The target business request is a non-query type request. The first database is the database of the first business system, and the second database is the database of the second business system.

[0166] The data verification module 630 is used to execute business logic corresponding to the target business request in the first business system and the second business system respectively, so that the data in the first database and the second database changes; to find the changed data in the first database and the second database respectively, and to perform data verification on the found changed data based on the data table mapping script, so as to obtain the data verification result.

[0167] The grayscale ratio adjustment module 640 is used to periodically obtain the data verification results within a preset period. If it is determined that the grayscale ratio needs to be adjusted based on the data verification results within the preset period, the grayscale ratio is adjusted until the grayscale ratio reaches the target value, so that all target business requests for the first business system are forwarded to the second business system.

[0168] In an exemplary embodiment, the data migration apparatus 600 further includes a business request execution module 650 and a data verification module 660, wherein:

[0169] The business request execution module 650 is used to execute the business logic corresponding to the query type request in the first business system and the second business system respectively, and obtain the first business execution result and the second business execution result;

[0170] The data verification module 660 is used to verify the data of the first business execution result and the second business execution result to obtain the data verification result; if the data verification result represents inconsistent data, the data table mapping script is adjusted.

[0171] The data synchronization module 610 is also used to perform incremental data synchronization between the first database and the second database based on the adjusted data table mapping script.

[0172] In an exemplary embodiment, the data synchronization module 610 is further configured to partition the data to be migrated in the first database to determine multiple first partition information; and distribute each first partition information to a distributed worker node so that the distributed worker node performs full data synchronization between the first database and the second database according to a preset data mapping script and the first partition information.

[0173] In an exemplary embodiment, the data synchronization module 610 is further configured to partition the incremental data in the first database to determine multiple second partition information; and distribute each second partition information to a distributed worker node so that the distributed worker node performs incremental data synchronization between the first database and the second database according to a preset data mapping script and the second partition information.

[0174] In an exemplary embodiment, the business request grayscale migration module 620 is further configured to perform a hash operation on the identifier information of the requester of the target business request for the first business system to obtain a hash value, take the remainder of the hash value with a first preset value to obtain a remainder result; determine the grayscale value corresponding to each grayscale ratio; and compare the remainder result with each grayscale value in turn. If the remainder result is less than the grayscale value, it is determined that the identifier information matches the grayscale ratio corresponding to the grayscale value.

[0175] In an exemplary embodiment, the grayscale ratio adjustment module 640 is further configured to, for each system interface, calculate the ratio of the target verification result values ​​representing inconsistent data among the verification result values ​​corresponding to the system interface within a preset period; if the ratio is greater than the preset maximum error tolerance rate, modify the grayscale ratio of the system interface to a second preset value; if the ratio is less than or equal to the preset maximum error tolerance rate and greater than the preset minimum error tolerance rate, maintain the grayscale ratio of the system interface; if the ratio is less than or equal to the preset minimum error tolerance rate and the grayscale ratio of the interface is greater than the preset maximum single grayscale increment value, adjust the grayscale ratio of the interface to the sum of the current grayscale ratio and the maximum single grayscale increment value.

[0176] In an exemplary embodiment, the grayscale ratio adjustment module 640 is further configured to adjust the grayscale ratio of the interface to a target multiple of the current grayscale ratio when the ratio is less than or equal to a preset minimum error tolerance rate and the grayscale ratio of the interface is less than or equal to a preset maximum single grayscale increase value.

[0177] In an exemplary embodiment, the data synchronization module 610 is further configured to obtain the data volume of the data to be migrated and the number of worker nodes, and record the initial position of the binlog log of the first database; based on the data volume of the data to be migrated and the number of distributed worker nodes, the data to be migrated is evenly divided into multiple partitions according to the primary key ID, and the primary key ID range of each partition is obtained, wherein the information of the first partition includes the primary key ID range; the primary key ID range of each partition is distributed to each distributed worker node, so that each distributed worker node queries the data of the first database according to its corresponding primary key ID range, executes a preset data table mapping script on the queried data, obtains new table data, and inserts the new table data into the second database.

[0178] In an exemplary embodiment, the data synchronization module 610 is further configured to: obtain the current position of the binlog log of the first database; determine the target binlog data between the initial position and the current position; divide the target binlog data evenly into multiple partitions based on the number of distributed worker nodes to obtain the binlog data range of each partition, wherein the second partition information includes the binlog data range; distribute the binlog data range of each partition to each worker node so that each worker node reads the binlog data within its corresponding binlog data range; parse the read binlog data into data table data; execute a preset data table mapping script on the data table data to obtain new table data; update the second database according to the new table data if data with the same primary key as the new table data is found in the second database; and insert the new table data into the second database if data with the same primary key as the new table data is not found in the second database.

[0179] Each module in the aforementioned data migration device 600 can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the operations corresponding to each module.

[0180] In one exemplary embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 8As shown, this computer device includes a processor, memory, input / output (I / O) interfaces, and a communication interface. The processor, memory, and I / O interfaces are connected via a system bus, and the communication interface is also connected to the system bus via the I / O interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides the environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The database stores data. The I / O interfaces are used for exchanging information between the processor and external devices. The communication interface is used for communicating with external terminals via a network connection. When the computer program is executed by the processor, it implements a data migration method.

[0181] Those skilled in the art will understand that Figure 8 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0182] In one exemplary embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in any of the above-described data migration method embodiments.

[0183] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the steps in any of the above data migration method embodiments.

[0184] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in any of the above data migration method embodiments.

[0185] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data must comply with relevant regulations.

[0186] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, artificial intelligence (AI) processors, etc., and are not limited to these.

[0187] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0188] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A data migration method, characterized in that, The method includes: Based on the preset data table mapping script, the first database of data to be migrated out and the second database of data to be migrated in are synchronized with full data and incremental data. If the identification information of the requester of the target business request to the first business system matches the gray scale ratio in the gray scale progress table, the target business request is forwarded to the second business system. The target business request is a non-query type request. The first database is the database of the first business system, and the second database is the database of the second business system. The business logic corresponding to the target business request is executed in the first business system and the second business system respectively, causing data changes in the first database and the second database; The changed data in the first database and the second database are found respectively, and the found changed data is checked based on the data table mapping script to obtain the data check result; The system periodically obtains data verification results within a preset period. If the grayscale ratio needs to be adjusted based on the data verification results within the preset period, the grayscale ratio is adjusted until the grayscale ratio reaches the target value, so that all target business requests for the first business system are forwarded to the second business system.

2. The method according to claim 1, characterized in that, After performing full data synchronization and incremental data synchronization between the first database and the second database, the method further includes: For a query-type request from the first business system, business logic corresponding to the query-type request is executed in both the first and second business systems to obtain the first business execution result and the second business execution result. The execution results of the first and second services are compared to obtain the data comparison results. If the data verification results indicate inconsistencies in the data, adjust the data table mapping script. The second database, based on a preset data table mapping script, performs full and incremental data synchronization for the data to be migrated, and also includes: Based on the adjusted data table mapping script, incremental data synchronization is performed between the first database and the second database.

3. The method according to claim 2, characterized in that, Based on a pre-defined data table mapping script, a full data synchronization is performed between the first and second databases, including: The data to be migrated in the first database is partitioned to determine multiple first partition information; The information of each of the first partitions is distributed to the distributed worker nodes, so that the distributed worker nodes can perform full data synchronization between the first database and the second database according to the preset data mapping script and the information of the first partition.

4. The method according to claim 2 or 3, characterized in that, Based on a pre-defined data table mapping script, incremental data synchronization is performed between the first and second databases, including: The incremental data in the first database is partitioned to determine multiple second partition information; The second partition information is distributed to distributed worker nodes so that the distributed worker nodes can perform incremental data synchronization between the first database and the second database according to the preset data mapping script and the second partition information.

5. The method according to claim 1, characterized in that, The grayscale progress table includes the grayscale ratios corresponding to different system interfaces; The detection process includes checking whether the identifier information of the requester for the target business request to the first business system matches the grayscale ratio in the grayscale progress table, including: A hash operation is performed on the identifier information of the requester of the target business request for the first business system to obtain a hash value. The hash value is then moduloed by a first preset value to obtain a remainder result. Determine the grayscale value corresponding to each of the aforementioned grayscale ratios; The remainder result is compared with each of the grayscale values ​​in turn. If the remainder result is less than the grayscale value, it is determined that the identification information matches the grayscale ratio corresponding to the grayscale value.

6. The method according to claim 5, characterized in that, The data verification results include verification result values; When it is determined that the grayscale ratio needs to be adjusted based on the data verification results within a preset period, adjusting the grayscale ratio includes: For each system interface, the ratio of the target verification result values ​​representing inconsistent data among the verification result values ​​corresponding to the system interface within a preset period is calculated. If the ratio is greater than the preset maximum error tolerance rate, the grayscale ratio of the system interface will be modified to a second preset value. When the ratio is less than or equal to the preset maximum error tolerance rate and the ratio is greater than the preset minimum error tolerance rate, the grayscale ratio of the system interface is maintained. If the ratio is less than or equal to the preset minimum error tolerance rate and the grayscale ratio of the interface is greater than the preset maximum single grayscale increment value, the grayscale ratio of the interface is adjusted to the sum of the current grayscale ratio and the maximum single grayscale increment value.

7. The method according to claim 6, characterized in that, The method further includes: If the ratio is less than or equal to the preset minimum error tolerance rate and the grayscale ratio of the interface is less than or equal to the preset maximum single grayscale increase value, the grayscale ratio of the interface is adjusted to a target multiple of the current grayscale ratio.

8. The method according to claim 3, characterized in that, The step of partitioning the data to be migrated in the first database to determine multiple first partition information includes: Obtain the amount of data to be migrated and the number of worker nodes, and record the initial position of the binlog log of the first database; Based on the amount of data to be migrated and the number of distributed worker nodes, the data to be migrated is evenly divided into multiple partitions according to the primary key ID, and the primary key ID range of each partition is obtained. The first partition information includes the primary key ID range. The step of distributing the information of each of the first partitions to the distributed worker nodes, so that the distributed worker nodes perform full data synchronization between the first database and the second database according to the preset data mapping script and the first partition information, includes: The primary key ID range of each partition is distributed to each of the distributed worker nodes, so that each of the distributed worker nodes queries the data in the first database according to its corresponding primary key ID range, executes a preset data table mapping script on the queried data, obtains new table data, and inserts the new table data into the second database.

9. The method according to claim 4, characterized in that, The step of partitioning the incremental data in the first database to determine multiple second partition information includes: Obtain the amount of data to be migrated and the number of worker nodes, and record the initial position of the binlog log of the first database; Obtain the current position of the binlog in the first database, and determine the target binlog data between the initial position and the current position; Based on the number of distributed worker nodes, the target binlog data is evenly divided into multiple partitions to obtain the binlog data range of each partition. The second partition information includes the binlog data range. The step of distributing the second partition information to each of the distributed worker nodes, so that each of the distributed worker nodes performs incremental data synchronization between the first database and the second database according to the preset data mapping script and the second partition information, includes: Distribute the binlog data range of each partition to each worker node, so that each worker node reads the binlog data within its corresponding binlog data range, parses the read binlog data into data table data, and executes a preset data table mapping script on the data table data to obtain new table data. If data with the same primary key as the new table data is found in the second database, the second database is updated according to the new table data. If no data with the same primary key as the new table data is found in the second database, the new table data is inserted into the second database.

10. A data migration device, characterized in that, The device includes: The data synchronization module is used to perform full data synchronization and incremental data synchronization between the first database of data to be migrated out and the second database of data to be migrated in, based on a preset data table mapping script. The business request grayscale migration module is used to forward the target business request to the second business system when the identification information of the requester of the target business request to the first business system matches the grayscale ratio in the grayscale progress table. The target business request is a non-query type request. The first database is the database of the first business system, and the second database is the database of the second business system. The data verification module is used to execute business logic corresponding to the target business request in the first business system and the second business system respectively, so that the data in the first database and the second database will change; to find the changed data in the first database and the second database respectively, and to perform data verification on the found changed data based on the data table mapping script, so as to obtain the data verification result; The grayscale ratio adjustment module is used to periodically obtain the data verification results within a preset period. If it is determined that the grayscale ratio needs to be adjusted based on the data verification results within the preset period, the grayscale ratio is adjusted until the grayscale ratio reaches the target value, so that all target business requests for the first business system are forwarded to the second business system.

Images