Data migration test method, data migration test device and electronic equipment

Abstract

The invention provides a data migration test method, a data migration test device and electronic equipment. In the scheme, the source table is extracted from the old system database to the intermediate database, and the source table is converted in the intermediate database by using the preset conversion rule to obtain the target table matched with the structure of the new system database; performing a first comparison operation on the source table and the target table in the intermediate library to obtain a first comparison result; performing second comparison operation on the data of the target table and the data of the new system database to obtain a second comparison result; sending an interface request message to the new system, obtaining a response message, and performing third comparison operation on a field in the response message and a corresponding field in an old system database to obtain a third comparison result; and analyzing the first comparison result, the second comparison result and the third comparison result to obtain a data migration test result. According to the scheme, the problem of limitation of a data migration test method is solved, and the limitation comprises low accuracy, low efficiency and incomplete test.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a data migration testing method, a data migration testing device, a computer-readable storage medium, and an electronic device. Background Art

[0002] Data migration refers to the process of migrating data from one computer storage system to another through synchronization, extraction, conversion, and import. Verifying the consistency, accuracy, and completeness of the migrated data is a crucial task during this process. When migrating data from a centralized database to a distributed database with sharded tables, the correctness of the data sharding results must also be verified. In addition to static comparisons, dynamic verification is also required to verify the availability and correctness of existing data in the new system.

[0003] Existing technologies for migrating data to distributed databases suffer from problems such as low accuracy, low efficiency, and incomplete testing. Summary of the Invention

[0004] The main purpose of this application is to provide a data migration testing method, a data migration testing device, a computer-readable storage medium and an electronic device to at least solve the problems of limitations of data migration testing methods in the prior art, including low accuracy, low efficiency and incomplete testing.

[0005] To achieve the above-mentioned objectives, according to one aspect of the present application, a data migration testing method is provided, comprising: extracting a source table from an old system database into an intermediate database, and converting the source table in the intermediate database using preset conversion rules to obtain a target table that is compatible with the structure of a new system database, wherein the intermediate database is a temporary database established during the data migration process from the old system to the new system; performing a first comparison operation on the source table and the target table in the intermediate database to obtain a first comparison result, wherein the first comparison operation includes a total amount comparison, a detailed comparison, a primary key comparison, and a data quality comparison between the source table and the target table; performing a second comparison operation on the data of the target table and the data of the new system database to obtain a second comparison result, wherein the second comparison operation includes a total amount comparison and a detailed comparison between the data of the target table and the data of the new system database; sending an interface request message to the new system, obtaining a response message, and performing a third comparison operation on the fields in the response message with the corresponding fields in the old system database to obtain a third comparison result; parsing the first comparison result, the second comparison result, and the third comparison result to obtain a data migration test result.

[0006] Optionally, a first comparison operation is performed on the source table and the target table in the intermediate library to obtain a first comparison result, wherein the first comparison operation includes a total amount comparison, a detailed comparison, a primary key comparison and a data quality comparison between the source table and the target table, and includes: obtaining a table mapping relationship file of the new system database and the old system database, wherein the table mapping relationship file includes a correspondence between tables in the new system database and the old system database, and a matching situation between fields; generating a first comparison script according to the table mapping relationship file; performing a configuration operation on the data to be verified, wherein the configuration operation includes full data verification and partial data verification; executing the first comparison script based on the configuration operation, performing the first comparison operation on the source table and the target table to obtain the first comparison result, wherein the first comparison result includes a first total amount comparison result, a first detailed comparison result, a primary key comparison result and a data quality comparison result.

[0007] Optionally, a second comparison operation is performed on the data of the target table and the data of the new system database to obtain a second comparison result, where the second comparison operation includes a total comparison and a detailed comparison between the data of the target table and the data of the new system database, including: obtaining a new system database table structure, where the new system database table structure is obtained from a table mapping relationship file between the new system database and the old system database; generating a second comparison script for verifying the correctness of data sharding based on the new system database table structure using preset sharding calculation rules and verification algorithms; configuring connection information of the new system database corresponding to each shard, where the connection information includes a host address, database name, port number, user name and password; performing a configuration operation on the data to be verified, where the configuration operation includes full data verification and partial data verification; based on the connection information and the configuration operation, performing the second comparison operation on the data of the target table and the data of the new system database, executing the second comparison script, and obtaining the second comparison result, where the second comparison result includes a second total comparison result and a second detailed comparison result.

[0008] Optionally, based on the table structure of the new system database, using preset sharding calculation rules and verification algorithms, a second comparison script for verifying the correctness of data sharding is generated, including: based on the preset sharding calculation rules, parsing the records of each business table in the new system database, calculating the corresponding sharding value, and obtaining the sharding value calculation result; using the verification algorithm, checking the correlation between each business table in the new system database to obtain the correlation check result; generating the second comparison script based on the sharding value calculation result and the correlation check result.

[0009] Optionally, an interface request message is sent to the new system, a response message is obtained, and a third comparison operation is performed on the fields in the response message with the corresponding fields in the old system database to obtain a third comparison result, including: sending the interface request message to the new system, the interface request message including multiple business scenarios to cover the interface function of the new system; collecting the response message made by the new system to the interface request message, the response message including the data field returned after executing specific business logic; generating a third comparison script according to the response message; executing the third comparison script, performing the third comparison operation on the fields in the response message with the corresponding fields in the old system database to obtain the third comparison result.

[0010] Optionally, before sending the interface request message to the new system, the method also includes: obtaining the mapping relationship between the fields in the new system interface response message and the fields in the old system database table through data combing and system analysis; constructing a field comparison table based on the mapping relationship, and the field comparison table is used to guide the comparison of the fields in the response message with the corresponding fields in the old system database.

[0011] Optionally, parsing the first comparison result, the second comparison result and the third comparison result to obtain the data migration test result includes: parsing the first comparison result, the second comparison result and the third comparison result; when there is no abnormality in the first comparison result, the second comparison result and the third comparison result, determining that the data migration test result is correct; when there is an abnormality in any comparison result among the first comparison result, the second comparison result and the third comparison result, locating the abnormality according to a preset error code mapping rule, and determining that there is an abnormality in the data migration test result.

[0012] According to another aspect of the present application, a data migration testing device is provided, comprising: a conversion unit for extracting a source table from an old system database to an intermediate database, and converting the source table in the intermediate database using a preset conversion rule to obtain a target table that is compatible with the structure of the new system database, wherein the intermediate database is a temporary database established during the data migration process from the old system to the new system; a first comparison unit for performing a first comparison operation on the source table and the target table in the intermediate database to obtain a first comparison result, wherein the first comparison operation includes a total amount comparison, a detailed comparison, a primary key comparison, and a data quality comparison between the source table and the target table. comparison; a second comparison unit, used to perform a second comparison operation on the data of the target table with the data of the new system database to obtain a second comparison result, wherein the second comparison operation includes a total comparison and a detailed comparison between the data of the target table and the data of the new system database; a third comparison unit, used to send an interface request message to the new system, obtain a response message, and perform a third comparison operation on the fields in the response message with the corresponding fields in the old system database to obtain a third comparison result; a parsing unit, used to parse the first comparison result, the second comparison result and the third comparison result to obtain a data migration test result.

[0013] According to another aspect of the present application, a computer-readable storage medium is provided, which includes a stored program, wherein when the program is run, the device where the computer-readable storage medium is located is controlled to execute any one of the data migration testing methods.

[0014] According to another aspect of the present application, an electronic device is provided, comprising: one or more processors, a memory, and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, and the one or more programs include methods for executing any one of the data migration testing methods.

[0015] By applying the technical solution of the present application, a source table is extracted from the old system database to an intermediate database, and the source table is converted in the intermediate database using preset conversion rules to obtain a target table that is compatible with the structure of the new system database. The intermediate database is a temporary database established during the data migration process from the old system to the new system; a first comparison operation is performed on the source table and the target table in the intermediate database to obtain a first comparison result. The first comparison operation includes a total amount comparison, a detailed comparison, a primary key comparison and a data quality comparison between the source table and the target table; a second comparison operation is performed on the data of the target table and the data of the new system database to obtain a second comparison result. The second comparison operation includes a total amount comparison and a detailed comparison between the data of the target table and the data of the new system database; an interface request message is sent to the new system, a response message is obtained, and a third comparison operation is performed on the fields in the response message and the corresponding fields in the old system database to obtain a third comparison result; the first comparison result, the second comparison result and the third comparison result are parsed to obtain a data migration test result. In this solution, the accuracy of data conversion is effectively verified by comparing the source and target tables, including total amounts, details, primary keys, and data quality. The integrity and consistency of data imported into the new system are ensured by comparing the target table data with the data in the new system database. By sending interface request messages to the new system and comparing the response messages with the old system database, the correctness of the new system logic and data interaction is verified, thus resolving the limitations of data migration testing methods, including low accuracy, low efficiency, and incomplete testing. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] The drawings that constitute part of this application are used to provide a further understanding of this application. The illustrative embodiments of this application and their descriptions are used to explain this application and do not constitute an improper limitation on this application. In the drawings:

[0017] Figure 1 A hardware structure block diagram of a mobile terminal for performing a data migration test method provided in an embodiment of the present application is shown;

[0018] Figure 2 A flow chart of a data migration testing method provided according to an embodiment of the present application is shown;

[0019] Figure 3 A data migration and data verification flow chart of a specific data migration test method provided according to an embodiment of the present application is shown;

[0020] Figure 4 A module diagram of a specific data migration testing method provided according to an embodiment of the present application is shown;

[0021] Figure 5A schematic diagram of an implementation flow of a specific data migration testing method provided according to an embodiment of the present application is shown;

[0022] Figure 6 A structural block diagram of a data migration testing device provided according to an embodiment of the present application is shown.

[0023] The above drawings include the following reference numerals:

[0024] 102. Processor; 104. Memory; 106. Transmission device; 108. Input / output device. DETAILED DESCRIPTION

[0025] It should be noted that, in the absence of conflict, the embodiments and features of the embodiments in this application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0026] In order to enable those skilled in the art to better understand the present invention, the following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of this application.

[0027] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequential order. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present application described here. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0028] As introduced in the background technology, the prior art has low accuracy, low efficiency and incomplete testing in data migration testing. In order to solve the problems of limitations of the data migration testing method in the prior art, including low accuracy, low efficiency and incomplete testing, the embodiments of the present application provide a data migration testing method, a data migration testing device, a computer-readable storage medium and an electronic device.

[0029] The technical solutions in the embodiments of the present invention will be described clearly and completely below with reference to the accompanying drawings in the embodiments of the present invention.

[0030] The method embodiments provided in the embodiments of the present application can be executed in a mobile terminal, a computer terminal or a similar computing device. Taking running on a mobile terminal as an example, Figure 1 FIG. 1 is a hardware structure diagram of a mobile terminal of a data migration test method according to an embodiment of the present invention. Figure 1 As shown, the mobile terminal may include one or more ( Figure 1 Only one is shown) a processor 102 (the processor 102 may include but is not limited to a microprocessor MCU or a programmable logic device FPGA and other processing devices) and a memory 104 for storing data, wherein the mobile terminal may also include a transmission device 106 and an input and output device 108 for communication functions. It will be understood by those skilled in the art that Figure 1 The structure shown is only for illustration and does not limit the structure of the mobile terminal. Figure 1 More or fewer components than shown, or with Figure 1 Different configurations shown.

[0031] The memory 104 can be used to store computer programs, such as software programs and modules of application software, such as the computer program corresponding to the device information display method in the embodiment of the present invention. The processor 102 executes various functional applications and data processing by running the computer program stored in the memory 104, that is, implementing the above-mentioned method. The memory 104 may include a high-speed random access memory and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory 104 may further include a memory remotely located relative to the processor 102, and these remote memories can be connected to the mobile terminal via a network. Examples of the above-mentioned networks include but are not limited to the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof. The transmission device 106 is used to receive or send data via a network. Specific examples of the above-mentioned network may include a wireless network provided by the mobile terminal's communication provider. In one example, the transmission device 106 includes a network adapter (Network Interface Controller, abbreviated as NIC), which can be connected to other network devices via a base station to communicate with the Internet. In one example, the transmission device 106 may be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0032] In this embodiment, a data migration testing method running on a mobile terminal, a computer terminal or a similar computing device is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0033] Figure 2 FIG. 1 is a flow chart of a data migration test method according to an embodiment of the present application. Figure 2 As shown, the method includes the following steps:

[0034] Step S201: Extract source tables from the old system database to an intermediate database, and convert the source tables in the intermediate database using preset conversion rules to obtain target tables that are compatible with the structure of the new system database. The intermediate database is a temporary database established during the data migration process from the old system to the new system.

[0035] Specifically, extracting source tables into a staging database occurs during the initial stages of data migration. This involves extracting data from the legacy database and transferring it to a temporary database specifically established for the migration, known as the staging database. The staging database serves as a transitional area, allowing data to be converted and initially validated without directly impacting the new system's database state. During the data extraction process, pre-written SQL statements are executed to select the data to be migrated from the legacy database according to specific rules. This involves data selection, filtering, backup, or direct copying. After the data is extracted into the staging database, it is converted according to pre-defined transformation rules, also executed through pre-written SQL statements. Transformation rules are designed based on the differences between the old and new database structures to adapt the source data format to a format compatible with the new system's database. These include adjusting data types, field mapping, data encoding conversion, and data length adjustments to ensure seamless data integration with the new system's database structure. The design of these transformation rules must precisely account for the differences in the data models between the old and new systems to ensure that the data accurately represents the same information in both environments.

[0036] After conversion, the source data is transformed into a target table in the intermediate database. The target table's structure and format have been adjusted to suit the characteristics of the new system's database. This process ensures that the data can be correctly identified and used in the new system, avoiding import failures or data errors caused by data structure mismatches.

[0037] The intermediate database is a key link in the data migration testing process, and it mainly plays the following two roles: First, it serves as a data conversion platform, allowing various preprocessing and conversion of data without affecting the direct operation of the new system; second, it serves as a place for data verification. Before the data is officially imported into the new system, preliminary and detailed comparison and inspection can be performed in the intermediate database to ensure the correctness and consistency of the data.

[0038] Through step S201, the data migration testing method effectively addresses differences in data structure and format, ensuring accurate data conversion during the migration process. This provides important technical support for new system deployment and a smooth data transition. Compared to traditional one-time data migrations, this approach is more conducive to discovering and resolving potential issues, ensuring high-quality data migration completion.

[0039] Step S202: Perform a first comparison operation on the source table and the target table in the intermediate database to obtain a first comparison result. The first comparison operation includes comparing the total amount, details, primary keys, and data quality of the source table and the target table.

[0040] Specifically, the first comparison operation is implemented through SQL statements. The main purpose of the total amount comparison between the source table and the target table is to confirm whether the number of data records in the source table is equal to the number of data records in the target table after conversion. This process is achieved by comparing the number of data rows in the source table and the target table. It aims to quickly check whether there are missing or redundant data rows during the data migration process to ensure data integrity. The detailed comparison between the source table and the target table is a detailed check of each data record in the source table and the target table to verify the consistency of the data before and after the conversion. It involves field-level comparison, that is, checking whether each field value in the source table and the target table is exactly the same to ensure that the specific content of the data has not changed during the migration process. For tables with primary key constraints, primary key comparison is a very critical test. The primary key is a field or field combination in the table used to uniquely identify each record. By comparing the primary keys of the source and target tables, it can be ensured that the value of the primary key field has not changed during the data migration process, avoiding data duplication or loss, and ensuring the uniqueness and relevance of the data. The data quality comparison between the source table and the target table focuses on the validity and compliance of the data, involving checks on data format, field integrity, data range, and data type, etc., to ensure that the data in the target table meets the specifications and standards of the new system database. For example, checks are made to see whether the date field format is correct, whether numeric fields contain non-numeric characters, whether fields are empty, etc., to avoid situations where the new system cannot operate normally due to data quality issues.

[0041] By combining these multiple comparisons, we can obtain a preliminary comparison result, which provides a foundation for subsequent testing steps and effectively ensures data migration quality control. Automating these comparisons significantly reduces human error, improves the efficiency and accuracy of data migration testing, and ensures a smooth transition between the old and new systems. In practice, the preliminary comparison result can be obtained by executing the corresponding SQL query statements and data processing scripts.

[0042] Step S203: performing a second comparison operation on the data in the target table and the data in the new system database to obtain a second comparison result. The second comparison operation includes a total amount comparison and a detailed comparison between the data in the target table and the data in the new system database.

[0043] Specifically, after the first comparison operation confirms that the total amount of data in the source and target tables is consistent, the second comparison operation further verifies that, after the target table data is imported into the new system database, the total amount of data in the corresponding table in the new system database is equal to that in the target table. This comparison process primarily verifies the completeness of the data import process and whether there was any data loss or excessive data duplication during the import process. A total amount comparison is a quick and effective preliminary test step that can promptly identify obvious problems with the data import. A detailed comparison of the target table data with the data in the new system database is a detailed verification of the data in the new system database, ensuring that every field of every data entry corresponds to the data in the target table. This covers field-level data consistency verification and can identify subtle errors that may be overlooked in a total amount comparison, such as inconsistent field values ​​and errors in data types. A detailed comparison helps confirm that the specific content of the data has been correctly preserved after migration to the new system.

[0044] The importance of the second comparison operation lies in re-verifying the effectiveness of the data migration from the perspective of the new system database, ensuring that the data maintains its accuracy and integrity after conversion from the source table to the target table and then importing it into the new system database. This process is particularly critical because data storage and processing in the new system may differ from the old system. Especially when migrating from a centralized to a distributed database, data fragmentation and distribution may affect data integrity and consistency.

[0045] Automating the second comparison not only improves testing efficiency but also reduces errors and omissions in manual testing, ensuring the reliability and accuracy of data migration testing. In practice, the second comparison results, along with the first, serve as a key component of comprehensive analysis of the success of the data migration test, providing a solid foundation for the smooth rollout of the new system.

[0046] Step S204: Send an interface request message to the new system, obtain a response message, and perform a third comparison operation on the fields in the response message with the corresponding fields in the old system database to obtain a third comparison result;

[0047] Specifically, interface request messages are constructed according to the interface specifications defined by the new system to simulate real-world business requests. These messages can contain data from various business scenarios, such as querying account information, processing transaction requests, and updating customer profiles. These messages are sent to the new system using automated testing tools or custom scripts, simulating actual user or external system operations to verify the proper functioning of the new system's interface functions and the accuracy of its data processing logic. When the new system receives an interface request message, it responds according to its internal business logic and data processing rules, returning one or more response messages. The response message contains the results of the new system's processing of the request, such as query results, transaction status, and update feedback. These response messages are the key basis for the third comparison operation.

[0048] Compare the field values ​​in the response message received from the new system with the corresponding field values ​​in the old system database. This step verifies that the new system's data processing and returned results are consistent with the old system, ensuring that after the data migration, the new system can correctly understand and process the data and return accurate results. This comparison involves comparing single fields, comparing complex fields (such as calculated fields), and verifying the logical relationships between fields.

[0049] The dynamic verification process described above yields a third comparison result, which reflects the functional correctness and data consistency of the new system when processing the migrated data. If the third comparison result matches expectations, it indicates that the new system is processing the data correctly and that the data migration has not caused any functional issues. Conversely, if there are discrepancies, further investigation is required to determine the source of the problem, which could be a data migration issue or a functional or logical issue with the new system.

[0050] By simulating actual business scenarios, dynamic validation can uncover deeper data migration issues, such as errors in data processing logic, abnormal interface calls, and data misinterpretation in the new system. This is beyond the capabilities of static data comparison. By combining the aforementioned comparisons of total volume, details, primary keys, and data quality, dynamic validation provides a comprehensive data migration testing solution, ensuring that data is not only correct in quantity and quality, but also correctly processed in real-world business scenarios, thereby improving the stability and reliability of the new system launch.

[0051] Step S205 , analyzing the first comparison result, the second comparison result, and the third comparison result to obtain a data migration test result.

[0052] Specifically, the first comparison result, derived from a comparison of the source and target tables in the intermediate database, includes information on data volume, details, primary keys, and data quality. This result is analyzed to verify the integrity and consistency of the data during the conversion process, as well as whether it meets pre-set data quality standards. If the first comparison result indicates that all data passes the comparison, it indicates that there were no issues with the data conversion and preprocessing phases, and that the data structure and content match the requirements of the new system. The second comparison result, obtained by comparing the target table data with the new system database, verifies the accuracy of the data import phase. This analysis focuses on confirming that the total volume of data in the new system database after data import is consistent with the target table, as well as whether data details, such as field values, fully match. If both the total volume and details are consistent, it can be assumed that no data was lost or added during the data import, ensuring data accuracy and consistency. The third comparison result is derived from feedback from the new system's dynamic functional testing, which involves testing actual business scenarios through interface request and response messages. These results are analyzed to confirm that the new system correctly responds to business operations after data migration and that the returned results are consistent with those of the legacy system. This step verifies the actual application effect of the data in the new system and ensures that the data migration will not affect the business functions of the system.

[0053] After analyzing the three comparison results above, they are combined to form a comprehensive data migration test report. If all comparison results show no data anomalies, consistency and integrity are maintained, and the new system can correctly process the data and return expected results, then the data migration test can be considered successful and the new system can be launched. Anomalies in the comparison results at any stage indicate potential issues with the data migration, requiring appropriate corrections or optimizations based on the specific circumstances of the anomaly. In actual operations, the analysis and synthesis of comparison results can be facilitated by automated testing tools or scripts, which can efficiently process the comparison results of large amounts of data, reduce manual intervention, and improve the accuracy and efficiency of testing. In addition, the analysis process also involves analyzing logs and error messages during the test execution to more comprehensively understand the status and effectiveness of the data migration, providing a reference for subsequent system optimization and maintenance.

[0054] In summary, by analyzing the comparison results at each stage, we can comprehensively evaluate the overall effectiveness of data migration and quickly identify and locate any potential problems, thereby greatly improving the accuracy, efficiency, and coverage of data migration testing, reducing labor costs, and ensuring the stable operation and data security of the new system after it goes online, providing strong technical support and guarantee for large-scale data migration projects.

[0055] Through this embodiment, an intermediate library is constructed as a bridge to achieve automated testing of data migration from the old system database to the new system database. First, by converting the old system data in the intermediate library, a target table that conforms to the new system database structure is generated, ensuring seamless connection of data format and compatibility; then, by performing a static first comparison operation in the intermediate library, the source table and the target table are carefully compared from multiple dimensions such as total amount, details, primary key and data quality, effectively verifying the accuracy of data conversion; then, a second comparison operation is performed on the target table data and the new system database data to further verify the integrity and consistency of the data after importing into the new system, which is a direct verification of the data migration results; finally, through a dynamic third comparison operation, that is, sending an interface request to the new system and comparing the response data with the old system database, the accuracy of the new system in processing the migrated data is comprehensively evaluated from the perspective of actual business functions, ensuring that the data migration is not only correct at the technical level, but also maintains consistency and effectiveness in terms of business logic. Combining the comparison results at the above three levels can provide comprehensive and in-depth data migration test results, significantly improving the quality, efficiency and credibility of data migration testing, ensuring the smooth launch and stable operation of the new system, and thus solving the limitations of data migration testing methods, including low accuracy, low efficiency and incomplete testing.

[0056] During the specific implementation process, a first comparison operation is performed on the above-mentioned source table and the above-mentioned target table in the above-mentioned intermediate library to obtain a first comparison result. The above-mentioned first comparison operation includes a total amount comparison, a detailed comparison, a primary key comparison and a data quality comparison between the above-mentioned source table and the above-mentioned target table, including: obtaining the table mapping relationship file of the above-mentioned new system database and the above-mentioned old system database, the above-mentioned table mapping relationship file includes the correspondence between the tables in the above-mentioned new system database and the above-mentioned old system database, as well as the matching status of the fields; generating a first comparison script according to the above-mentioned table mapping relationship file; performing a configuration operation on the data to be verified, the above-mentioned configuration operation includes full data verification and partial data verification; executing the above-mentioned first comparison script based on the above-mentioned configuration operation, performing the above-mentioned first comparison operation on the above-mentioned source table and the above-mentioned target table to obtain the above-mentioned first comparison result, the above-mentioned first comparison result includes a first total amount comparison result, a first detailed comparison result, a primary key comparison result and a data quality comparison result.

[0057] Specifically, before data migration, you need to prepare a detailed table mapping relationship file, which records in detail the correspondence between tables in the old system database (source system) and the new system database (target system), as well as the matching of fields. This is the basis for data migration and the key basis for generating comparison scripts to ensure that the data can be accurately aligned during the conversion process to meet the data structure requirements of the new system. Based on the above table mapping relationship file, the automated tool or script generates the first comparison script, which contains specific instructions for performing total comparison, detailed comparison, primary key comparison and data quality comparison. These instructions are designed for the source table and the target table, and can accurately compare the data in the two tables to check for any inconsistencies or quality issues during the data conversion process.

[0058] Before performing the first comparison, you need to configure the data to be verified. You can choose to verify the data in full or in part. Full verification compares all data, while partial verification allows you to specify a specific business primary key (such as a customer number) for data verification. This is suitable for quickly locating issues or testing with limited resources.

[0059] Based on the configured data to be verified, the first comparison script is executed to perform a detailed comparison of the source and target tables. The first total comparison result confirms the consistency of the total data volume in the source and target tables; the first detailed comparison result verifies the consistency of the field values ​​in each record; the primary key comparison result ensures the correctness of key business identifiers during the conversion process; and the data quality comparison result verifies whether the data meets established quality standards, such as rationality and validity.

[0060] Through the first comparison operation, the integrity and quality of data conversion can be automatically detected, providing a solid foundation for subsequent steps of data migration testing, while also ensuring that the data conversion from the source system to the target system is accurate.

[0061] In some embodiments of the present application, a second comparison operation is performed on the data of the above-mentioned target table and the data of the above-mentioned new system database to obtain a second comparison result. The above-mentioned second comparison operation includes a total comparison and a detailed comparison between the data of the above-mentioned target table and the data of the above-mentioned new system database, including: obtaining the new system database table structure, the above-mentioned new system database table structure is obtained from the table mapping relationship file of the above-mentioned new system database and the above-mentioned old system database; according to the above-mentioned new system database table structure, using the preset sharding calculation rules and verification algorithm, generating a second comparison script for verifying the correctness of the data sharding; configuring the connection information of the above-mentioned new system database corresponding to each shard, the above-mentioned connection information includes the host address, database name, port number, user name and password; performing a configuration operation on the data to be verified, the above-mentioned configuration operation includes full data verification and partial data verification; based on the above-mentioned connection information and the above-mentioned configuration operation, the above-mentioned second comparison operation is performed on the data of the above-mentioned target table and the data of the above-mentioned new system database, executing the above-mentioned second comparison script, and obtaining the above-mentioned second comparison result, the above-mentioned second comparison result includes a second total comparison result and a second detailed comparison result.

[0062] Specifically, before performing the second comparison, the new system database table structure must be accurately obtained. This can be directly retrieved from the table mapping files between the new and old system databases. This ensures an understanding of the data organization and distribution logic within the new system database, which serves as the foundation for generating the second comparison script. Automated comparison scripts are developed using pre-defined sharding calculation rules and verification algorithms. The sharding calculation rules determine how data is allocated to different shards (databases) based on business primary keys (such as customer numbers), while the verification algorithm verifies that the data distribution in the new system database is as expected, specifically that data with the same business primary key is correctly allocated to the same shard. This ensures logical correctness and data consistency within the shards after the data migration. To enable direct access and querying of the new system database, database connection information for each shard must be configured, including the host address, database name, port number, username, and password. This information ensures that automated testing tools or comparison scripts can successfully connect to each shard of the new system database and perform data comparison operations.

[0063] Similar to the first comparison operation, the second comparison operation also supports full data validation and partial data validation. Users can configure the verification to validate the entire dataset or only the data under a specific business primary key. This provides flexibility and targeted testing, which is particularly effective when resources are limited or only specific data issues need to be quickly identified.

[0064] Based on the new system database's connection information and the configured data to be verified, a second comparison script is executed to compare the target table's data with the new system database's data. The second total comparison confirms whether the total amount of data in each shard after the data migration is consistent with the target table. The second detailed comparison verifies whether the field values ​​of each record in the shard match the target table data, ensuring the integrity of the data migration and the accuracy of the sharding logic.

[0065] This second comparison not only verifies the consistency of the total amount and details of the data after migration to the new distributed or centralized system database, but more importantly, ensures the correct sharding of the data in the new system, which directly impacts system performance and data access efficiency. Furthermore, this second comparison further verifies the reliability of the data migration, providing strong data assurance for the smooth launch and stable operation of the new system.

[0066] Furthermore, based on the table structure of the above-mentioned new system database, using the preset sharding calculation rules and verification algorithm, a second comparison script for verifying the correctness of data sharding is generated, including: based on the above-mentioned preset sharding calculation rules, parsing the records of each business table in the above-mentioned new system database, calculating the corresponding sharding value, and obtaining the sharding value calculation result; using the above-mentioned verification algorithm, checking the correlation between each of the above-mentioned business tables in the above-mentioned new system database, and obtaining the correlation check result; generating the above-mentioned second comparison script based on the above-mentioned sharding value calculation result and the above-mentioned correlation check result.

[0067] Specifically, the preset shard calculation rules are a set of predefined algorithms used to determine which shard (database) each record should be assigned to, based on the characteristics of the business data and the design of the new system's database architecture. These rules consider factors such as the business primary key (such as the customer ID), geographic location, and data type to achieve load balancing, optimize data access performance, or meet regulatory compliance requirements. By reading records from each business table in the new system database and applying the preset shard calculation rules, the shard value for each record is calculated. The shard value is a key indicator that determines the shard to which a record should be assigned and serves as the basis for subsequent data sharding correctness verification. The verification algorithm is specifically designed to verify data relevance and consistency. In a distributed database environment, data under the same business primary key should be distributed across the same shard to ensure efficient and correct cross-table queries and transactions. By querying and comparing data with the same business primary key across different business tables, the verification algorithm verifies that all data has been correctly assigned to the same shard. This verification ensures that even when data is distributed across multiple shards, the consistency and integrity of related data is maintained.

[0068] The shard value calculation results and the relevance check results are integrated to form the core basis for verifying the correctness of data sharding. Using this integrated result, a second comparison script is generated. This second comparison script contains specific SQL statements or other database query languages ​​to perform total and detailed comparisons, verifying that the actual shard values ​​are consistent with the expected values ​​and that the shard allocation of data under the business primary key complies with the relevance check results. The script is designed with efficiency, coverage, and readability in mind, making the comparison process automated and efficient.

[0069] Through the above process, a second comparison script is generated specifically for verifying the correctness of data sharding. This script not only checks the consistency of the total amount and details of the data, but also focuses on the correct implementation of the data sharding logic and the effective maintenance of business data relevance in the new system database. This not only enhances the depth and breadth of data migration testing but also provides additional security for the launch of the new system, ensuring the correct distribution and efficient management of data in a distributed database environment.

[0070] In other embodiments of the present application, an interface request message is sent to the above-mentioned new system, a response message is obtained, and a third comparison operation is performed on the fields in the above-mentioned response message with the corresponding fields in the above-mentioned old system database to obtain a third comparison result, including: sending the above-mentioned interface request message to the above-mentioned new system, the above-mentioned interface request message includes multiple business scenarios to cover the interface functions of the above-mentioned new system; collecting the above-mentioned response message made by the above-mentioned new system to the above-mentioned interface request message, the above-mentioned response message includes the data field returned after executing specific business logic; generating a third comparison script according to the above-mentioned response message; executing the above-mentioned third comparison script, performing the above-mentioned third comparison operation on the fields in the above-mentioned response message with the corresponding fields in the above-mentioned old system database to obtain the above-mentioned third comparison result.

[0071] Specifically, the phase of sending interface request messages to the new system involves constructing request messages that cover the various interface functions of the new system. When designing these messages, the diversity of business scenarios must be fully considered to ensure coverage of key interfaces in the new system. The purpose of sending interface request messages is to trigger the business logic of the new system, causing it to execute the appropriate processing and return a response message, thereby dynamically verifying the correctness of the new system's behavior when handling the migrated data. Upon receiving the interface request message, the new system generates a response message based on its internal business logic and data processing rules. The response message contains the data fields returned after executing the specific business logic. These fields directly reflect the processing results of the new system and are used to compare them with the corresponding fields in the legacy system database. After collecting the response messages, a third comparison script is generated based on the field information in these response messages. This third comparison script details the comparison process, including how to extract the corresponding field values ​​from the legacy system database and how to match them with the field values ​​in the response message. When generating the script, ensure the accuracy of the comparison logic to avoid false positives or false negatives.

[0072] When the third comparison script is executed, it retrieves the field data related to the interface request from the old system database and then performs a detailed comparison with the corresponding fields in the response message. This field-level comparison drills down to the specific data values, verifying that the new system maintains consistency with the old system when processing data. If the comparison results show an exact match in field values, it indicates that the new system's business logic and data processing capabilities remain reliable after the data migration. Conversely, if any discrepancies are found, it indicates that the new system has problems handling certain business scenarios, requiring further analysis and correction.

[0073] This third comparison not only verifies the new system's interface functionality but also thoroughly checks the correctness of its data processing logic and business consistency. This approach is not only applicable to functional testing of new systems but also for rapid business verification before system launch, ensuring that the migrated system can operate stably in real-world business environments and that its data processing capabilities meet design requirements. This significantly improves the efficiency and accuracy of data migration testing and reduces risks associated with system launch.

[0074] Furthermore, before sending the above-mentioned interface request message to the above-mentioned new system, the above-mentioned method also includes: obtaining the mapping relationship between the fields in the new system interface response message and the fields in the old system database table through data combing and system analysis; constructing a field comparison table based on the above-mentioned mapping relationship, and the above-mentioned field comparison table is used to guide the comparison of the fields in the above-mentioned response message with the corresponding fields in the above-mentioned old system database.

[0075] Specifically, before sending an interface request message to the new system, data combing and system analysis must first be performed. This step aims to gain an in-depth understanding of the structure of the old system database and the interface functions of the new system, so as to accurately identify the meaning and function of the fields in the interface response message, as well as the corresponding relationship with the fields in the old system database. Data combing includes analysis of field type, length, and data format, while system analysis focuses on understanding business logic and the direction of data flow. The result of data combing and system analysis is a detailed field mapping relationship list, which records in detail the mapping relationship between each field in the new system interface response message and the corresponding field in the old system database, including field name, data type, data length, and possible conversion rules. The field mapping relationship is the basis for constructing a field comparison table, ensuring that the data of the new system and the old system can be correctly aligned during the comparison process, avoiding test failures due to misunderstanding of fields.

[0076] Based on the obtained field mapping relationships, a field comparison table is constructed. This is a structured data table used in the third comparison operation to guide how to match and compare the fields in the new system's interface response message with the corresponding fields in the old system database. The field comparison table contains the field name, its location in the old system database, the field conversion rule (if applicable), and the expected comparison result.

[0077] After sending an interface request message and receiving a response message, the third comparison script performs a field-level comparison using the field comparison table. Based on the field mappings and conversion rules recorded in the comparison table, the third comparison script extracts the corresponding field data from the legacy system database and compares it with the corresponding field values ​​in the response message. The comparison results detail any field value discrepancies, providing a basis for subsequent analysis and problem location.

[0078] This process ensures accurate and efficient verification of the consistency of the new system interface's output data with the legacy system's database data during dynamic testing (the third comparison operation), thereby comprehensively evaluating the business logic processing capabilities after data migration. This not only improves test accuracy and coverage, but also simplifies the testing process, reduces test error rates, and provides strong support for data migration testing before the new system goes live.

[0079] In other embodiments of the present application, the first comparison result, the second comparison result and the third comparison result are analyzed to obtain the data migration test result, including: analyzing the first comparison result, the second comparison result and the third comparison result; when there is no abnormality in the first comparison result, the second comparison result and the third comparison result, determining that the data migration test result is correct; when there is an abnormality in any comparison result among the first comparison result, the second comparison result and the third comparison result, locating the abnormality according to a preset error code mapping rule, and determining that there is an abnormality in the data migration test result.

[0080] Specifically, the first comparison result, the second comparison result, and the third comparison result are parsed. Each comparison result contains the test data comparison information of each stage, such as the comparison status of the total amount of data, detailed data, field values, etc. The parsing operation is intended to convert these original comparison data into an understandable and analyzable form for subsequent analysis and judgment. After all the comparison results are parsed, the comparison results of these three stages are combined to determine the final data migration test results. If all the comparison results (the first comparison result, the second comparison result, and the third comparison result) show that there are no abnormalities in the data comparison, that is, the total amount, details, and field values ​​of the data during the conversion, migration, and interface interaction process are consistent with expectations, then the overall data migration test results will be judged to be correct.

[0081] However, if an anomaly is found in the comparison results at any stage (such as data volume mismatch, field value difference, etc.), it is necessary to locate the specific cause of the anomaly based on the preset error code mapping rules. The error code mapping rule is a pre-defined mechanism that associates possible comparison anomalies with specific error codes. These error codes can intuitively indicate whether there is an error in the data conversion program, the data sharding import program, or the new system business logic. By locating the anomaly, the system can quickly identify the fault link and provide direction for fixing the problem and optimizing the migration process. After discovering the anomaly and locating the specific source of the error, it is determined that there is an anomaly in the data migration test results. This means that at least one link in the data migration process has not met the expected standards and requires further investigation and correction. After determining the abnormal test results, the testing team can take targeted measures to solve the corresponding problems based on the error code and anomaly details to ensure the quality and success of the data migration project.

[0082] In short, comprehensive analysis and judgment of the three comparison results provide a comprehensive and accurate data migration test result. This not only confirms the correctness of the data migration, but also promptly identifies and locates potential anomalies, providing strong support for subsequent data migration project management and system rollout. This comprehensive judgment mechanism ensures the comprehensiveness and reliability of data migration testing, helps reduce system rollout risks, and improves the success rate of data migration.

[0083] In other embodiments of the present application, an adaptive data extraction and conversion algorithm can be introduced, which can automatically analyze the table structure and field attributes of the old system and the new system database, intelligently generate data extraction and conversion scripts, significantly reduce manual intervention, and improve the efficiency and accuracy of the preliminary preparation of data migration. Through deep learning model analysis of historical data migration cases, a set of algorithms that can automatically identify and extract key fields, handle data type conversion, and identify indexes and constraints are trained. In actual applications, the adaptive data extraction and conversion algorithm will automatically detect the structural differences between the source database and the target database, generate corresponding data conversion scripts, and ensure that the data remains consistent and complete before and after migration. The adaptive data extraction and conversion algorithm reduces the workload of testers, reduces the possibility of human error, can quickly adapt to changes in various database architectures, and improve the flexibility and testing speed of data migration projects.

[0084] In order to enable those skilled in the art to more clearly understand the technical solution of the present application, the implementation process of the data migration testing method of the present application will be described in detail below with reference to specific embodiments.

[0085] This embodiment relates to a specific data migration test method, which performs three comparison tests through a scheduling module. The scheduling module can realize the scheduling of serial and parallel tasks, flexibly configure the comparison program, and verify the correctness, consistency and integrity of the data at the database layer and the interface layer. The data migration and data verification flow chart is shown in FIG. Figure 3 As shown. By performing total quantity verification, primary key verification, detail verification, single-table multi-field association verification, multi-table association verification, and data quality verification on the source table and the target table after data conversion in the intermediate library, the correctness of the data is verified to obtain the first comparison result; by performing total quantity verification and detail verification on the target table in the intermediate library and the new system database, the consistency and integrity of the data are verified to obtain the second comparison result; by sending an interface request message to the new system, the obtained response message is compared with the old system database at the field level to verify the correctness of the data and obtain the third comparison result; by analyzing the first comparison result, the second comparison result, and the third comparison result, the correctness, consistency, and integrity of the migration result are verified. The method includes comparison module 1, comparison module 2, comparison module 3, and result verification module. The module schematic diagram is shown as follows. Figure 4 shown.

[0086] 1. Comparison module 1:

[0087] This module is divided into 2 submodules:

[0088] 1) The detail verification submodule automatically generates a comparison script based on the mapping relationship between the new and old system database table structures, and simultaneously queries the intermediate library source table and target table to obtain the first comparison result;

[0089] 2) The module for configuring data to be verified supports two modes: partial verification and full verification. The default mode is full verification. When configuring data (business primary key, such as customer number), the module uses partial verification mode.

[0090] 2. Comparison module 2:

[0091] This module is divided into 4 submodules:

[0092] 1) The shard value calculation and verification submodule, based on the sharding logic of the migration program, writes shard value calculation and verification functions to determine whether the migration results are consistent with the calculated sharding results. It also uses inter-table associations to check whether all business tables are inserted into the same shard (database) in the new system, achieving consistency and integrity verification.

[0093] 2) Database configuration submodule, configures the new system database connection information corresponding to each shard (library), including host address, database name, port, user name and password;

[0094] 3) The data configuration module to be verified supports two modes: partial verification and full verification. The default mode is full verification. When configuring data (business primary key, such as customer number), it is partial verification mode;

[0095] 4) Detailed verification submodule: Since the intermediate database target table and the new system database table structure are completely consistent, it is only necessary to compare the value of each field in each table of the new system with the data in the intermediate database target table to obtain the second comparison result;

[0096] 3. Comparison module 3:

[0097] This module is divided into 2 submodules:

[0098] 1) The data configuration module to be verified supports two modes: partial verification and full verification. The default mode is full verification. When configuring data (business primary key, such as customer number), the partial verification mode is used;

[0099] 2) The detail checking submodule sends an interface request message to the new system and queries the old system database by sorting out the correspondence between the fields of the interface response message and the fields of the old system database table to obtain the third comparison result.

[0100] 4. Result verification module: By analyzing the comparison results, determine whether each link in the migration process is correct. At the same time, by establishing error code mapping rules, the source of the error can be quickly known when an error occurs, and the erroneous module can be quickly determined. If the first comparison result is wrong, it is judged that an error occurred in the data conversion program, corresponding to error code A+table name; if the first comparison result is correct and the second comparison result is wrong, it is judged that an error occurred in the data sharding or import program, corresponding to error code B+table name; if the first comparison result and the second comparison result are correct and the third comparison result is wrong, it is judged that there is an error in the business logic of the new system, corresponding to error code C+scenario name; if the first comparison result, the second comparison result and the third comparison result are all correct, then the migration result is judged to be correct.

[0101] The specific implementation flow diagram of this embodiment is as follows Figure 5 Specifically:

[0102] S0: Task scheduling module, supports serial and parallel scheduling of comparison tasks of each module.

[0103] S1: First comparison.

[0104] Specifically: S1-1: Obtain the table mapping relationship file for the new and old system databases. S1-2: Automatically generate comparison script 1 (the first comparison script) based on the table mapping relationship. S1-3: Configure the data to be verified. S1-4: Based on the generated comparison script 1, call and execute comparison script 1. S1-5: Obtain the comparison results and analyze the test results.

[0105] S2: Second comparison.

[0106] Specifically: S2-1: Obtain the new system database table structure. S2-2: Automatically generate comparison script 2 (the second comparison script) based on the new system database table structure in the mapping file. S2-3: Configure the data to be verified and the distributed database connection information of the new system, including the host address, database name, port, user name, and password. S2-4: Execute comparison script 2 based on the configured data to be verified, verify the old value results, and check the total amount and details of each business table. S2-5: Obtain the comparison results and analyze the test results.

[0107] S3: The third comparison.

[0108] Specifically: S3-1: Obtain the correspondence between the sorted new system interface response message and the fields in the old system database table. S3-2: Obtain the interface message in production. S3-3: Automatically generate an interface script based on the interface message, namely comparison script 3 (the third comparison script). S3-4: Configure the data to be verified. S3-5: Execute comparison script 3 and perform a one-to-one comparison based on the correspondence between the interface response message and the fields in the old system database table. S3-6: Obtain the comparison results and analyze the test results.

[0109] This embodiment provides a comparison method between a centralized database and a distributed database, which can support data migration comparison and verification between centralized systems, between distributed systems, and between centralized and distributed systems. By developing a shard value calculation and verification module, it is verified whether the shard value of the business table is correct when the data is imported from the intermediate library target table into the new system data, and whether the data of each business table under the same business primary key (such as customer number) can be allocated to the same sub-library, thereby ensuring the consistency and integrity of the data after migration and further improving the quality of migration data testing. The static + dynamic verification method automatically sends an interface request message to the new system, compares the response message with the old system database field, and achieves an end-to-end verification effect. It can quickly verify the correctness of the new system logic and the migration data, ensure the quality of migration data testing and improve testing efficiency, and save data migration test verification time. This embodiment provides a general data migration test method, which automatically generates a test script and selects full or partial verification according to the usage scenario. It is suitable for both functional test verification before production and rapid business verification during production, saving labor costs and improving testing efficiency. This embodiment also provides a method for accurately locating errors in data migration programs. By mapping predefined comparison scripts to error codes, the specific error link can be quickly determined and located. This allows for rapid problem resolution before production and during the verification process, improving problem-solving efficiency. This embodiment also provides a method for scheduling migration test comparison tasks, which supports flexible scheduling of each comparison task. It can use serial scheduling tasks to focus on issues in a comparison module based on actual usage scenarios, and parallel scheduling tasks to select the comparison module to be executed. Each task is independent of the test results of the previous link, saving testing time and improving testing efficiency.

[0110] During the data migration test process, especially the dynamic test phase, it is crucial to predict and warn of possible abnormal behaviors in advance, which can avoid the waste of time and resources caused by discovering serious problems in the middle and late stages of the test. This embodiment introduces an abnormal behavior prediction and warning system based on artificial intelligence. By analyzing historical data migration test results and logs, it learns and builds a model to predict abnormalities that may occur in the future data migration process, issues warnings in time, and guides testers to intervene and adjust in advance. Machine learning algorithms (such as neural networks, decision trees, etc.) are used to analyze past data migration test records to identify which field combinations, which data sharding strategies, or interface interaction modes are more likely to cause errors or abnormalities. Based on these models, potential risk points can be predicted before the new data migration test begins, triggering an early warning mechanism. The abnormal behavior prediction and warning system based on artificial intelligence makes data migration testing more proactive and preventive, can significantly reduce the impact of sudden problems, shorten the time for problem investigation and resolution, and enhance the overall stability of the data migration project. In addition, as the model continues to learn and optimize, the accuracy and timeliness of the warning will continue to improve, further improving the quality and efficiency of testing.

[0111] The embodiments of the present application also provide a data migration testing device. It should be noted that the data migration testing device of the embodiments of the present application can be used to execute the data migration testing method provided by the embodiments of the present application. The device is used to implement the above-mentioned embodiments and preferred implementation modes, and the details that have been described will not be repeated here. As used below, the term "module" can implement a combination of software and / or hardware for a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceived.

[0112] The following introduces the data migration testing device provided in the embodiment of the present application.

[0113] Figure 6 : is a structural block diagram of a data migration test device according to an embodiment of the present application. Figure 6As shown, the device includes a conversion unit 10, a first comparison unit 20, a second comparison unit 30, a third comparison unit 40 and a parsing unit 50. The conversion unit is used to extract the source table from the old system database to the intermediate database, and convert the source table in the intermediate database using a preset conversion rule to obtain a target table that is compatible with the structure of the new system database. The intermediate database is a temporary database established during the data migration process from the old system to the new system. The first comparison unit is used to perform a first comparison operation on the source table and the target table in the intermediate database to obtain a first comparison result. The first comparison operation includes a total amount comparison, a detailed comparison, a primary key comparison and a data quality comparison between the source table and the target table. The second comparison unit is used to convert A second comparison operation is performed on the data of the target table and the data of the new system database to obtain a second comparison result. The second comparison operation includes a total comparison and a detailed comparison of the data of the target table and the data of the new system database. The third comparison unit is used to send an interface request message to the new system, obtain a response message, and perform a third comparison operation on the fields in the response message with the corresponding fields in the old system database to obtain a third comparison result. The parsing unit is used to parse the first comparison result, the second comparison result and the third comparison result to obtain a data migration test result.

[0114] Through this embodiment, an intermediate library is constructed as a bridge to achieve automated testing of data migration from the old system database to the new system database. First, by converting the old system data in the intermediate library, a target table that conforms to the new system database structure is generated, ensuring seamless connection of data format and compatibility; then, by performing a static first comparison operation in the intermediate library, the source table and the target table are carefully compared from multiple dimensions such as total amount, details, primary key and data quality, effectively verifying the accuracy of data conversion; then, a second comparison operation is performed on the target table data and the new system database data to further verify the integrity and consistency of the data after importing into the new system, which is a direct verification of the data migration results; finally, through a dynamic third comparison operation, that is, sending an interface request to the new system and comparing the response data with the old system database, the accuracy of the new system in processing the migrated data is comprehensively evaluated from the perspective of actual business functions, ensuring that the data migration is not only correct at the technical level, but also maintains consistency and effectiveness in terms of business logic. Combining the comparison results at the above three levels can provide comprehensive and in-depth data migration test results, significantly improving the quality, efficiency and credibility of data migration testing, ensuring the smooth launch and stable operation of the new system, and thus solving the limitations of data migration testing methods, including low accuracy, low efficiency and incomplete testing.

[0115] In the specific implementation process, the first comparison unit includes a first acquisition module, a first generation module, a first configuration module and a first comparison module. Among them, the first acquisition module is used to obtain the table mapping relationship file of the new system database and the old system database, and the table mapping relationship file includes the correspondence between the tables in the new system database and the old system database, as well as the matching between the fields; the first generation module is used to generate a first comparison script according to the table mapping relationship file; the first configuration module is used to perform configuration operations on the data to be verified, and the configuration operations include full data verification and partial data verification; the first comparison module is used to execute the first comparison script based on the configuration operations, perform the first comparison operation on the source table and the target table, and obtain the first comparison result, which includes the first total comparison result, the first detailed comparison result, the primary key comparison result and the data quality comparison result.

[0116] The first total amount comparison confirms the consistency of the total amount of data in the source and target tables; the first detail comparison verifies the consistency of field values ​​in each record; the primary key comparison ensures the correctness of key business identifiers during the conversion process; and the data quality comparison verifies whether the data meets established quality standards, such as rationality and validity. This first comparison automatically verifies the integrity and quality of the data conversion, providing a solid foundation for subsequent data migration testing steps and ensuring that the data conversion from the source system to the target system is accurate.

[0117] In some embodiments of the present application, the second comparison unit includes a second acquisition module, a second generation module, a second configuration module, a third configuration module, and a second comparison module. The second acquisition module is used to obtain the table structure of the new system database, which is obtained from the table mapping relationship file between the new system database and the old system database; the second generation module is used to generate a second comparison script for verifying the correctness of the data sharding based on the table structure of the new system database and the preset sharding calculation rules and verification algorithm; the second configuration module is used to configure the connection information of the new system database corresponding to each shard, and the connection information includes the host address, database name, port number, user name, and password; the third configuration module is used to perform configuration operations on the data to be verified, and the configuration operations include full data verification and partial data verification; the second comparison module is used to perform the second comparison operation on the data of the target table and the data of the new system database based on the connection information and the configuration operations, execute the second comparison script, and obtain the second comparison result, which includes the second total comparison result and the second detailed comparison result.

[0118] The second total amount comparison result confirms whether the total amount of data in each shard after data migration is consistent with the target table; the second detailed comparison result checks whether the field value of each record in the shard matches the target table data, ensuring the integrity of the data migration and the accuracy of the sharding logic. Through this second comparison operation, not only is the consistency of the total amount of data and data details after the data is migrated to the distributed or centralized new system database verified, but more importantly, it ensures that the sharding results of the data in the new system are correct, which directly affects the system's performance and data access efficiency. In addition, the second comparison operation further verifies the reliability of the data migration, providing strong data support for the smooth launch and stable operation of the new system.

[0119] Furthermore, the second generation module includes a parsing submodule, a checking submodule, and a generation submodule. The parsing submodule is configured to parse the records of each business table in the new system database based on the preset sharding calculation rule, calculate the corresponding sharding value, and obtain a sharding value calculation result; the checking submodule is configured to use the verification algorithm to check the associations between the business tables in the new system database and obtain a correlation check result; and the generation submodule is configured to generate the second comparison script based on the sharding value calculation result and the correlation check result.

[0120] Through the above process, a second comparison script is generated specifically for verifying the correctness of data sharding. This script not only checks the consistency of the total amount and details of the data, but also focuses on the correct implementation of the data sharding logic and the effective maintenance of business data relevance in the new system database. This not only enhances the depth and breadth of data migration testing but also provides additional security for the launch of the new system, ensuring the correct distribution and efficient management of data in a distributed database environment.

[0121] In some other embodiments of the present application, the third comparison unit includes a sending module, a collecting module, a third generating module, and a third comparison module. The sending module is used to send the interface request message to the new system, the interface request message includes multiple business scenarios to cover the interface functions of the new system; the collecting module is used to collect the response message made by the new system to the interface request message, the response message includes the data field returned after executing specific business logic; the third generating module is used to generate a third comparison script based on the response message; the third comparison module is used to execute the third comparison script, perform the third comparison operation on the fields in the response message with the corresponding fields in the old system database, and obtain the third comparison result.

[0122] The above process not only verified the new system's interface functionality but also thoroughly examined the correctness of its data processing logic and business consistency. This approach is not only applicable to functional testing of new systems but also plays a role in rapid business verification before the system goes live, ensuring that the system, after data migration, can operate stably in the actual business environment and that its data processing capabilities meet design requirements. This significantly improves the efficiency and accuracy of data migration testing and reduces the risks of system launch.

[0123] Furthermore, the apparatus further includes an analysis unit and a construction unit. The analysis unit is configured to, before sending the interface request message to the new system, obtain a mapping relationship between fields in the new system interface response message and fields in the old system database table through data combing and system analysis; the construction unit is configured to construct a field comparison table based on the mapping relationship, and the field comparison table is used to guide the comparison of fields in the response message with corresponding fields in the old system database.

[0124] This process ensures accurate and efficient verification of the consistency of the new system interface's output data with the legacy system's database data during dynamic testing (the third comparison operation), thereby comprehensively evaluating the business logic processing capabilities after data migration. This not only improves test accuracy and coverage, but also simplifies the testing process, reduces test error rates, and provides strong support for data migration testing before the new system goes live.

[0125] In some other embodiments of the present application, the parsing unit includes a parsing module, a first determination module, and a second determination module. The parsing module is configured to parse the first comparison result, the second comparison result, and the third comparison result; the first determination module is configured to determine that the data migration test result is correct if none of the first comparison result, the second comparison result, and the third comparison result are abnormal; and the second determination module is configured to locate the abnormality according to a preset error code mapping rule and determine that the data migration test result is abnormal if any of the first comparison result, the second comparison result, and the third comparison result are abnormal.

[0126] By comprehensively analyzing and assessing the three comparison results, we can provide a comprehensive and accurate data migration test result. This not only confirms the correctness of the data migration, but also promptly identifies and locates potential anomalies, providing strong support for subsequent data migration project management and system rollout. This comprehensive assessment mechanism ensures the comprehensiveness and reliability of data migration testing, helping to reduce system rollout risks and increase the success rate of data migration.

[0127] The data migration testing device includes a processor and memory. The conversion unit, first comparison unit, second comparison unit, third comparison unit, parsing unit, etc. are stored as program units in the memory. The processor executes the program units stored in the memory to implement the corresponding functions. The modules are all located in the same processor; alternatively, the modules can be located in different processors in any combination.

[0128] The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM, and the memory includes at least one memory chip.

[0129] An embodiment of the present invention provides a computer-readable storage medium, which includes a stored program. When the program is executed, the device where the computer-readable storage medium is located is controlled to execute the data migration test method.

[0130] An embodiment of the present invention provides an electronic device comprising a processor, a memory, and a program stored in the memory and executable on the processor. When the processor executes the program, the steps of the aforementioned data migration testing method are implemented. The device herein may be a server, a PC, a PAD, a mobile phone, or the like.

[0131] The present application also provides a computer program product, which, when executed on a data processing device, is suitable for executing a program that initializes the steps of the above-mentioned data migration test method.

[0132] Obviously, those skilled in the art will appreciate that the various modules or steps of the present invention described above can be implemented using a general-purpose computing device, can be centralized on a single computing device, or can be distributed across a network of multiple computing devices. They can be implemented using program code executable by the computing device, and thus, can be stored in a storage device and executed by the computing device. In some cases, the steps shown or described herein can be performed in a different order than that shown, or can be fabricated as separate integrated circuit modules, or multiple modules or steps can be fabricated as a single integrated circuit module. Thus, the present invention is not limited to any particular combination of hardware and software.

[0133] Those skilled in the art will appreciate that the embodiments of the present application can be provided as methods, systems, or computer program products. Therefore, the present application can adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application can adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0134] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the steps in the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0135] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0136] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0137] In a typical configuration, a computing device includes one or more processors (CPUs), input / output interfaces, network interfaces, and memory.

[0138] The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0139] Computer-readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. The information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable media does not include transitory media such as modulated data signals and carrier waves.

[0140] It should also be noted that the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, commodity, or apparatus that includes a series of elements includes not only those elements but also other elements not explicitly listed, or includes elements inherent to such process, method, commodity, or apparatus. In the absence of further limitations, an element defined by the phrase "comprises a ..." does not exclude the presence of other identical elements in the process, method, commodity, or apparatus that includes the element.

[0141] The above description is merely a preferred embodiment of the present application and is not intended to limit the present application. Various modifications and variations are possible for those skilled in the art. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of the present application shall be included within the scope of protection of the present application.

Claims

1. A data migration testing method, characterized in that: include: Extract source tables from the old system database to an intermediate database, and convert the source tables in the intermediate database using preset conversion rules to obtain target tables that are compatible with the structure of the new system database. The intermediate database is a temporary database set up during the data migration process from the old system to the new system. performing a first comparison operation on the source table and the target table in the intermediate database to obtain a first comparison result, wherein the first comparison operation includes a total amount comparison, a detail comparison, a primary key comparison, and a data quality comparison between the source table and the target table; performing a second comparison operation on the data of the target table and the data of the new system database to obtain a second comparison result, wherein the second comparison operation includes a total amount comparison and a detailed comparison between the data of the target table and the data of the new system database; Sending an interface request message to the new system, obtaining a response message, and performing a third comparison operation on fields in the response message with corresponding fields in the old system database to obtain a third comparison result; The first comparison result, the second comparison result, and the third comparison result are analyzed to obtain a data migration test result.

2. The method according to claim 1, characterized in that A first comparison operation is performed on the source table and the target table in the intermediate database to obtain a first comparison result. The first comparison operation includes a total amount comparison, a detail comparison, a primary key comparison, and a data quality comparison between the source table and the target table, including: Obtaining a table mapping relationship file of the new system database and the old system database, wherein the table mapping relationship file includes a correspondence between tables in the new system database and the old system database, and matching conditions between fields; Generate a first comparison script according to the table mapping relationship file; Perform configuration operations on the data to be verified, including full data verification and partial data verification; The first comparison script is executed based on the configuration operation, and the first comparison operation is performed on the source table and the target table to obtain the first comparison result, which includes a first total comparison result, a first detailed comparison result, a primary key comparison result, and a data quality comparison result.

3. The method according to claim 1, characterized in that Performing a second comparison operation on the data in the target table and the data in the new system database to obtain a second comparison result, wherein the second comparison operation includes a total amount comparison and a detailed comparison between the data in the target table and the data in the new system database, including: Acquire a new system database table structure, where the new system database table structure is acquired from a table mapping relationship file between the new system database and the old system database; According to the new system database table structure, using the preset sharding calculation rules and verification algorithm, a second comparison script for verifying the correctness of the data sharding is generated; Configure the connection information of the new system database corresponding to each shard, the connection information including the host address, database name, port number, user name and password; Perform configuration operations on the data to be verified, including full data verification and partial data verification; Based on the connection information and the configuration operation, the second comparison operation is performed on the data of the target table and the data of the new system database, and the second comparison script is executed to obtain the second comparison result, which includes a second total comparison result and a second detailed comparison result.

4. The method according to claim 3, characterized in that According to the new system database table structure, using the preset sharding calculation rules and verification algorithm, a second comparison script for verifying the correctness of the data sharding is generated, including: Based on the preset sharding calculation rule, the records of each business table in the new system database are parsed, the corresponding sharding value is calculated, and the sharding value calculation result is obtained; Using the verification algorithm, checking the correlation between the business tables in the new system database to obtain a correlation checking result; The second comparison script is generated according to the fragment value calculation result and the correlation check result.

5. The method according to claim 1, wherein Sending an interface request message to the new system, obtaining a response message, and performing a third comparison operation on fields in the response message with corresponding fields in the old system database to obtain a third comparison result, including: Sending the interface request message to the new system, where the interface request message includes multiple business scenarios to cover the interface functions of the new system; Collecting the response message made by the new system to the interface request message, wherein the response message includes a data field returned after executing specific business logic; Generate a third comparison script according to the response message; The third comparison script is executed to perform the third comparison operation on the fields in the response message and the corresponding fields in the old system database to obtain the third comparison result.

6. The method according to claim 5, characterized in that Before sending the interface request message to the new system, the method further includes: Through data combing and system analysis, obtain the mapping relationship between the fields in the new system interface response message and the fields in the old system database table; A field comparison table is constructed according to the mapping relationship, and the field comparison table is used to guide the comparison between the fields in the response message and the corresponding fields in the old system database.

7. The method according to claim 1, characterized in that Parsing the first comparison result, the second comparison result, and the third comparison result to obtain a data migration test result includes: Analyzing the first comparison result, the second comparison result, and the third comparison result; If there is no abnormality in the first comparison result, the second comparison result, and the third comparison result, determining that the data migration test result is correct; When any one of the first comparison result, the second comparison result, and the third comparison result is abnormal, the abnormality is located according to a preset error code mapping rule, and it is determined that the data migration test result is abnormal.

8. A data migration testing device, characterized in that: include: A conversion unit is configured to extract source tables from the old system database to an intermediate database, and convert the source tables in the intermediate database using preset conversion rules to obtain target tables that are compatible with the structure of the new system database. The intermediate database is a temporary database established during the data migration process from the old system to the new system. a first comparison unit, configured to perform a first comparison operation on the source table and the target table in the intermediate database to obtain a first comparison result, wherein the first comparison operation includes a comparison of the total amount, details, primary keys, and data quality between the source table and the target table; a second comparison unit, configured to perform a second comparison operation on the data in the target table and the data in the new system database to obtain a second comparison result, wherein the second comparison operation includes a total amount comparison and a detailed comparison between the data in the target table and the data in the new system database; a third comparison unit, configured to send an interface request message to the new system, obtain a response message, and perform a third comparison operation on fields in the response message with corresponding fields in the old system database to obtain a third comparison result; The parsing unit is configured to parse the first comparison result, the second comparison result, and the third comparison result to obtain a data migration test result.

9. A computer-readable storage medium, characterized in that The computer-readable storage medium includes a stored program, wherein when the program is executed, the device where the computer-readable storage medium is located is controlled to execute the data migration testing method according to any one of claims 1 to 7.

10. An electronic device, characterized in that: include: One or more processors, a memory, and one or more programs, wherein the one or more programs are stored in the memory and are configured to be executed by the one or more processors, and the one or more programs include a method for executing the data migration testing method described in any one of claims 1 to 7.

Images