Data Migration Testing Method, Device, Equipment, Storage Medium and Program Product

Abstract

The present application provides a data migration testing method, apparatus, device, storage medium and program product. It relates to the technical field of data processing. Obtain a field mapping table from a source system to a target system, and obtain a table-level mapping relationship model according to the field mapping table, wherein the table-level mapping relationship model contains data migration conversion relationships; obtain source data in the source system and target data located in the target system after migration; perform migration testing on the source data and the target data with reference to the field mapping table and the table-level mapping relationship model. By obtaining the table-level mapping relationship model containing data migration conversion relationships, even when the structures of the source table and the target table change, the source data can be processed according to the data migration transfer relationships in the table-level mapping relationship model and then perform migration testing with the target data, thereby improving the efficiency and accuracy of data migration testing.

Description

Technical Field

[0001] The present application relates to the technical field of data processing, and particularly relates to a data migration testing method, device, equipment, storage medium and program product. Background Art

[0002] During the process of database processing, data migration is usually involved. To ensure the accuracy of the migrated data, it is usually necessary to test the migrated data. Currently, the commonly used testing method is to directly compare the original system table data with the target system table data to verify the consistency of the data before and after migration.

[0003] However, the above method of directly comparing data is only applicable when the structures of the source system table and the target system table remain unchanged. For data migration in the scenario of system reconstruction and testing in the scenario of reconstructing and transforming new system batch processing jobs, it is usually difficult to implement, requires high professionalism of testers, and is time-consuming and laborious for manual testing of data migration with incomplete coverage. Summary of the Invention

[0004] The present application provides a data migration testing method, device, equipment, storage medium and program product to solve the technical problem of inaccurate live detection in the prior art.

[0005] In a first aspect, the present application provides a data migration testing method, including: obtaining a field mapping table from a source system to a target system, and obtaining a table-level mapping relationship model according to the field mapping table, where the table-level mapping relationship model includes a data migration conversion relationship;

[0006] obtaining source data in the source system and target data located in the target system after migration;

[0007] performing migration testing on the source data and the target data with reference to the field mapping table and the table-level mapping relationship model.

[0008] In a second aspect, the present application provides a data migration testing device, including:

[0009] a table-level mapping relationship model obtaining module, configured to obtain a field mapping table from a source system to a target system, and obtain a table-level mapping relationship model according to the field mapping table, where the table-level mapping relationship model includes a data migration conversion relationship;

[0010] a data obtaining module, configured to obtain source data in the source system and target data located in the target system after migration;

[0011] a migration testing module, configured to perform migration testing on the source data and the target data with reference to the field mapping table and the table-level mapping relationship model.

[0012] In a third aspect, the present application provides an electronic device, including: a processor, and a memory communicatively connected to the processor;

[0013] The memory stores computer-executable instructions;

[0014] The processor executes the computer-executable instructions stored in the memory to implement the method as described in the present application.

[0015] In a fourth aspect, the present application provides a computer-readable storage medium, in which computer-executable instructions are stored, and when the computer-executable instructions are executed by a processor, they are used to implement the method as described in the present application.

[0016] In a fifth aspect, the present application provides a computer program product, including a computer program, and when the computer program is executed by a processor, it implements the method as described in the present application.

[0017] The data migration test method, device, equipment, storage medium and program product provided by the present application obtain a table-level mapping relationship model containing data migration conversion relationships, so that even when the structures of the source table and the target table change, the source data can be processed according to the data migration transfer relationship in the table-level mapping relationship model and then migrated and tested with the target data, thereby improving the efficiency and accuracy of data migration testing. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The accompanying drawings herein are incorporated into the specification and constitute a part of the specification, showing embodiments consistent with the present application, and are used together with the specification to explain the principles of the present application.

[0019] Figure 1 is the flowchart of the data migration method provided by the embodiment of the present application Figure 1 ;

[0020] Figure 2 is the flowchart of the data migration method provided by the embodiment of the present application Figure 2 ;

[0021] Figure 3 is the schematic structural diagram of the data migration device provided by the embodiment of the present application;

[0022] Figure 4 is the schematic structural diagram of the electronic device provided by the embodiment of the present application.

[0023] Through the above accompanying drawings, specific embodiments of the present application have been shown, and there will be more detailed descriptions hereinafter. These drawings and textual descriptions are not intended to limit the scope of the concept of the present application in any way, but to illustrate the concept of the present application to those skilled in the art by referring to specific embodiments. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0024] Exemplary embodiments will be described in detail herein, and examples thereof are shown in the accompanying drawings. When the following description refers to the accompanying drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present application. On the contrary, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the appended claims.

[0025] The technical solution of the present application and how the technical solution of the present application solves the above technical problems will be described in detail below with specific embodiments. These several specific embodiments below can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present application will be described below with reference to the accompanying drawings. The acquisition, storage, use, processing, etc. of data in the technical solution of the present application all comply with the relevant provisions of national laws and regulations.

[0026] Figure 1 This is the first process of the data migration test method provided for the embodiments of the present application. As Figure 1 shown, the living body detection method includes the following steps:

[0027] Step S101, obtain the field mapping table from the source system to the target system, and obtain the table-level mapping relationship model according to the field mapping table.

[0028] Specifically, the embodiments of the present application are applicable to data migration testing in the system reconstruction scenario. Of course, it is not limited to this scenario. The source system refers to the system before migration, and the target system refers to the system after migration. For the system reconstruction scenario, due to changes in the database table design of the source system and the target system, the table names, field names, field type lengths, precisions, and even some fields involve conversions and spicy system code values in the new and old systems. The specific content of the system scenario is not limited in this embodiment. Among them, the field mapping table is the basis for data mapping and processing from the source system, and the field mapping table includes the source table name, source table fields, processing methods, processing statements, filtering conditions, target table name, target table fields, primary key identification, and field index relationship. The field mapping table can specifically be uploaded to the test platform by testers and the database connection information of the source system and the target system is configured.

[0029] Optionally, obtain the table-level mapping relationship model according to the field mapping table, including: determining the fields to be compared from the source table fields, and generating the expression of each field to be compared in the source table according to the processing method, processing statement and filtering condition of the fields to be compared; concatenating the expressions of each field to be compared in the source table to generate the concatenated generation formula of the fields to be compared in the source table before and after; determining the fields to be compared from the target table fields, and concatenating the fields to be compared in the target table to generate the concatenated generation formula of the fields to be compared in the target table before and after; and constructing the table-level mapping relationship model according to the source table name, target table name, source table primary key field, target table primary key field, the concatenated generation formula of the fields to be compared in the source table before and after, and the concatenated generation formula of the fields to be compared in the target table before and after, where the concatenated generation formula of the fields to be compared in the source table before and after and the concatenated generation formula of the fields to be compared in the target table before and after are used as the data migration conversion relationship.

[0030] In a specific implementation, determine the fields to be compared from the source table fields. For example, the fields are age and address (add). The processing method or filtering condition for age is to convert the calculation of years to months, and the processing method or filtering condition for add is to retain the content in the address to the city level address and delete the community or village level address. Therefore, the field expression for age is age * 12; the field expression for add is del community or village from add. After obtaining the expressions of each field to be compared in the source table, concatenate the expressions of each field to be compared in the source table to generate the concatenated generation formula of the fields to be compared in the source table before and after: (age * 12) AND del (community or village from add). At the same time, determine the fields to be compared age and address (ADD) from the target table fields, and concatenate the fields to be compared in the target table to generate the concatenated generation formula of the fields to be compared in the target table before and after: (age) AND ADD. After generating the concatenated generation formula of the fields to be compared in the source table before and after, and the concatenated generation formula of the fields to be compared in the target table before and after, extract information such as the source table name, target table name, source table primary key field, and target table primary key field from the field mapping table, and combine the obtained generation formulas and the information from the field mapping table to construct the table-level mapping relationship model.

[0031] It should be noted that in this embodiment, the concatenated generation formula of the fields to be compared in the source table before and after, and the concatenated generation formula of the fields to be compared in the target table before and after can be used as the data migration conversion relationship. Since the table-level mapping relationship model also includes relevant information such as the source table name and target table name, during subsequent migration testing, the source data can be processed according to the relationship included in the expression and then compared with the target data, thus avoiding the problem of inaccurate testing caused by direct data comparison in system reconstruction.

[0032] Step S102, obtain the source data in the source system and the target data in the target system after migration.

[0033] Among them, after obtaining the table-level mapping relationship model, source data can be extracted from the source table names in the source system and target data can be extracted from the target table names in the target system according to information such as the source table name and target table name in the table-level mapping relationship model.

[0034] Step S103, perform a migration test on the source data and the target data with reference to the field mapping table and the table-level mapping relationship model.

[0035] Optionally, performing a migration test on the source data and the target data with reference to the field mapping table and the table-level mapping relationship model includes: performing a test on the migration content of the source data and the target data with reference to the table-level mapping relationship model to obtain a migration content test result; performing a migration index constraint test on the source data and the target data with reference to the field mapping table to obtain a migration index constraint test result; performing a test on the migration sequence range of the source data and the target data with reference to the table-level mapping relationship model to obtain a migration sequence range test result.

[0036] Optionally, the method further includes: when it is determined that the fields to be compared include code value fields, obtain the new and old system code value mapping table records, where the new and old system code value mapping table records include the conversion relationship of the code value fields in the new and old systems; obtain the source code values in the code value fields in the source table names in the source system and the target code values in the target table names in the target system; determine whether the source code values and the target code values conform to the conversion relationship, and if so, determine that the code value conversion test passes, otherwise, determine that the code value conversion test fails; where the code value fields include industry code value fields, area unit code fields or measurement unit fields.

[0037] Specifically, in this embodiment, a migration test will be performed on the source data and the target data with reference to the table-level mapping relationship model and the field mapping table. The test items of the migration test are multi-faceted, including migration content, migration index constraint and migration sequence range. The comparison of the migration content can include the comparison of the data volume, sampling comparison and full volume comparison. This embodiment does not limit the comparison of the migration content. And when the fields to be compared include code value fields, a separate test will be performed on the code value fields, and the code value fields do not participate in the comparison of the migration content. The following will specifically analyze and explain each test item.

[0038] Optionally, test the migration content of the source data and the target data with reference to the table-level mapping relationship model, and obtain the test results of the migration content, including: compare the data volumes of the source data and the target data with reference to the table-level mapping relationship model to obtain the data volume comparison result; sample and compare the source data and the target data with reference to the table-level mapping relationship model to obtain the sampling comparison result; when both the data volume comparison result and the sampling comparison result pass the test, compare the full volumes of the source data and the target data with reference to the table-level mapping relationship model to obtain the full volume comparison result.

[0039] Optionally, compare the data volumes of the source data and the target data with reference to the table-level mapping relationship model to obtain the data volume comparison result, including: extract the source table name and the target table name from the table-level mapping relationship model; obtain the first quantity of the source data in the source table name in the source system, and obtain the second quantity of the target data in the target table name in the target system; determine whether the first quantity is the same as the second quantity, and if they are the same, determine that the data volume comparison test passes, otherwise determine that the data volume comparison test fails.

[0040] Optionally, sample and compare the source data and the target data with reference to the table-level mapping relationship model to obtain the sampling comparison result, including: extract the source table name and the target table name from the table-level mapping relationship model; randomly obtain sampled source data in the source table name in the source system, and obtain sampled target data corresponding to the sampled source data in the target table name in the target system; process the sampled source data according to the expression before and after splicing the fields to be compared in the source table to obtain the processed result of the sampled source data; determine whether the processed result of the sampled source data is the same as the sampled target data, and if they are the same, determine that the sampling comparison test passes, otherwise determine that the sampling comparison test fails.

[0041] Optionally, the method further includes: when it is determined that the sampling comparison test fails, obtain the difference fields in which the contents of the fields to be compared are different, and the different contents in the difference fields; display the difference fields and the different contents.

[0042] Specifically, during the data migration test comparison process in the refactoring scenario, due to changes in the field design in the refactoring scenario, such changes themselves will cause data differences. Therefore, in the refactoring scenario, there are problems such as a large gap between the source table and the target table structures and the inability to directly compare the processing statements. Therefore, a sampling comparison of the data volume and data content should be carried out before the full comparison. This stage can quickly verify the problem of incorrect migrated data. Since the table-level mapping relationship model contains the source table name and the target table name, and the information such as the fields to be compared is also included in the splicing generation formula, the first data volume of the source data corresponding to the fields to be compared is obtained from the source table name, and the second data volume of the target data corresponding to the fields to be compared is obtained from the target table name. When the two are consistent, it indicates that no data loss occurred during the migration process. In addition, the sampling comparison is to map and process the source table data based on the table-level mapping relationship model and then perform a sampling comparison with the target table data, solving the problems of a large gap between the source table and the target table structures and the inability to directly compare the processing statements in the refactoring scenario. And in the sampling comparison link, the analysis of the differential fields is added, intuitively showing the field-level difference content of the specific differential records before and after migration, greatly improving the efficiency of developers to locate specific incorrect data, and also facilitating designers to analyze problems such as the accuracy problem, space problem, and the impact of the processing statement on the source data of the fields before and after migration from the data-driven perspective, which is beneficial to the design correctness.

[0043] Optionally, perform a full comparison of the source data and the target data with reference to the table-level mapping relationship model to obtain the full comparison result, including: extracting the source table name and the target table name from the table-level mapping relationship model; obtaining the source data of each record in the source table name in the source system, and obtaining the target data corresponding to each record in the target table name in the target system; processing the source data of each record according to the source table field expression before and after splicing generation formula to obtain the processing result of the source data of each record, calculating the first MD5 value of the processing result of the source data of each record, and the second MD5 value of the target data corresponding to each record; determining whether the first MD5 value of the source data of each record is the same as the second MD5 value of the target data. If they are the same, it is determined that the full comparison test passes; otherwise, it is determined that the full comparison test fails.

[0044] Among them, in the embodiments of the present application, a full comparison is added on the basis of the comparison of the data volume and the sampling comparison of the data content. After mapping and processing the source table data, the data content of all fields to be compared in the source table is spliced for each record to calculate the MD5, and the data content of all fields to be compared is also spliced before and after for the fields of the target table in the target system to calculate the MD5. Among them, the splicing order of the target table fields should correspond to the splicing order of the source table. By comparing the MD5 generated by each record in the source system with the MD5 generated by each record in the target system, it is verified whether all records in the source table and the target table are consistent under the condition of meeting the mapping processing rules, solving the problem of incomplete coverage of manual testing.

[0045] Optionally, the method further includes: when it is determined that the fields to be compared include code value fields, obtaining the records of the new-old system code value mapping table, where the records of the new-old system code value mapping table include the conversion relationships of the code value fields in the new and old systems; obtaining the source code values in the code value fields of the source table name in the source system and the target code values in the target table name in the target system; determining whether the source code values and the target code values conform to the conversion relationship, and if so, determining that the code value conversion test passes, otherwise, determining that the code value conversion test fails.

[0046] Specifically, the code value fields in this embodiment include industry code value fields, area unit code fields, or measurement unit fields. These code values are not completely consistent in the new and old systems and are usually recorded in a separate new-old system code mapping table. The fields involving code value conversion in the new and old systems do not directly participate in the above sampling comparison and full-scale comparison. The code value ranges of the fields involving code value conversion in the new and old systems are tested separately to verify whether the data code value ranges after the migration of the new system meet the requirements of the new system. Then, through the field mapping table from the source system to the target system and the association of the code value types in the new-old system code value mapping table, the converted values of the relevant code value fields in the source table in the new system are confirmed. By verifying whether the values of the code value fields in the target table are equal to the converted values, it is determined whether the code value fields are migrated correctly.

[0047] Optionally, perform a migration index constraint test on the source data and the target data with reference to the field mapping table, and obtain the migration index constraint test result, including: obtaining the source fields with index constraint relationships and the target fields with index constraint relationships according to the field index relationships in the field mapping table; determining whether the source fields with index constraint relationships are consistent with the target fields with index constraint relationships, and if so, determining that the migration index constraint test passes, otherwise, determining that the migration index constraint test fails.

[0048] Optionally, perform a test on the migration sequence range of the source data and the target data with reference to the table-level mapping relationship model, and obtain the migration sequence range test result, including: obtaining the primary key fields of the source table and the primary key fields of the target table according to the table-level mapping relationship model; determining the source table sequence range according to the primary key fields of the source table and determining the target table sequence range according to the primary key fields of the target table; determining whether the source table sequence range is the same as the target table sequence range, and if the same, determining the migration sequence range test, otherwise, determining that the migration sequence range test fails.

[0049] Specifically, sequences are usually used as the primary key field values of database tables. The primary key values generated by sequences are unique and can uniquely identify a record. For the scenario of migrating from the source system to the target system, designers usually impose restrictions on the sequence range so that the target system sequence has a certain redundancy compared to the source system. For non-stop migrations, this can ensure that the incremental business data generated by the source system during the migration process is properly merged into the target system.

[0050] In this embodiment, by obtaining a table-level mapping relationship model containing data migration and conversion relationships, even when the structures of the source table and the target table change, the source data can be processed according to the data migration and transfer relationships in the table-level mapping relationship model and then subjected to migration testing with the target data, thereby improving the efficiency and accuracy of data migration testing.

[0051] Figure 2 is the flow of the data migration testing method provided by the embodiment of the present application Figure 2 。On the basis of the above embodiment, this embodiment adds the situation of detecting the availability of the migrated data, such as Figure 2 As shown, this data migration testing method includes the following steps:

[0052] Step S201: Obtain a field mapping table from the source system to the target system, and obtain a table-level mapping relationship model according to the field mapping table.

[0053] Step S202: Obtain the source data in the source system and the target data located in the target system after migration.

[0054] Step S203: Conduct migration testing on the source data and the target data with reference to the field mapping table and the table-level mapping relationship model.

[0055] Step S204: When it is determined that the migration testing passes, run a specified batch job on the source system based on the source data at a specified time point to obtain the first batch of processing operation results.

[0056] Among them, in this implementation manner, based on the fact that the new and old systems have data at the same time point after migration and run the same batch of jobs at the same time, the business logic of the batch processing job transformation of the new system is verified by verifying the result tables after batch processing of the new and old systems. The source system data has been migrated to the target system, and when the content, code value conversion, index constraint, and sequence range of the data all pass the test, the data of the source system and the target system at the same time point provides a prerequisite for the batch comparison of the new and old systems. Therefore, run a specified batch job on the source data at a specified time point first to obtain the first batch of processing operation results.

[0057] Step S205: Run a specified batch job on the target system based on the target data at a specified time point to obtain the second batch of processing operation results.

[0058] Similarly, the target system will also run a specified batch job on the target data at a specified time point, that is, the target system runs the same batch of jobs as the source system at the same time based on the target data, and then obtains the second batch of processing operation results. The second batch of processing operation results in this implementation manner may be the same as or different from the first batch of processing operation results.

[0059] In step S206, it is determined whether the operation results of the first batch of processing and the second batch of processing are the same. If so, step S207 is executed; otherwise, step S208 is executed.

[0060] In step S207, it is determined that the migrated data is available.

[0061] Among them, when the operation results of the first batch of processing and the second batch of processing are the same, it indicates that the source data is accurately migrated to the target system, and the target data migrated to the target system is available, that is, it can operate normally.

[0062] In step S208, it is determined that the migrated data is unavailable.

[0063] Among them, when the operation results of the first batch of processing and the second batch of processing are different, it indicates that although the source data is accurately migrated to the target system, the target data migrated to the target system is unavailable, that is, it cannot operate normally. The reason for the above may be that the format of the target data does not match the target system, so it may be necessary to adjust the field mapping table.

[0064] It should be noted that when it is determined that the migrated data is unavailable, in this embodiment, an alarm prompt will also be issued to prompt the tester to timely remind the user to make corresponding adjustments to further improve the efficiency and accuracy of data migration testing.

[0065] In this embodiment, by obtaining the table-level mapping relationship model including the data migration conversion relationship, even when the structures of the source table and the target table change, the source data can be processed according to the data migration transfer relationship in the table-level mapping relationship model and then migrated and tested with the target data, thereby improving the efficiency and accuracy of data migration testing. By testing the availability of the migrated data, a comprehensive test of the migrated data is realized, ensuring the comprehensiveness of the migrated data test.

[0066] Figure 3 It is a schematic structural diagram of the data migration testing device provided by the embodiment of the present application. As Figure 3 shown, the living body detection device includes: a table-level mapping relationship model acquisition module 310, a data acquisition module 320, and a migration testing module 330.

[0067] The table-level mapping relationship model acquisition module 310 is used to obtain the field mapping table from the source system to the target system, and obtain the table-level mapping relationship model according to the field mapping table, where the table-level mapping relationship model includes the data migration conversion relationship;

[0068] The data acquisition module 320 is used to obtain the source data in the source system and the target data located in the target system after migration;

[0069] The migration test module 330 is used to perform a migration test on the source data and the target data with reference to the field mapping table and the table-level mapping relationship model.

[0070] Optionally, the field mapping table includes a source table name, source table fields, a processing method, a processing statement, a filtering condition, a target table name, target table fields, a primary key identifier, and a field index relationship.

[0071] Optionally, the table-level mapping relationship model acquisition module is used to determine the fields to be compared from the source table fields, and generate source table expressions for each field to be compared according to the processing method, processing statement, and filtering condition of the fields to be compared;

[0072] Concatenate the source table expressions for each field to be compared to generate a concatenated expression for the fields to be compared in the source table;

[0073] Determine the fields to be compared from the target table fields, and concatenate the target table fields to be compared to generate a concatenated expression for the fields to be compared in the target table;

[0074] And construct a table-level mapping relationship model according to the source table name, target table name, source table primary key field, target table primary key field, the concatenated expression for the fields to be compared in the source table, and the concatenated expression for the fields to be compared in the target table,

[0075] wherein, the concatenated expression for the fields to be compared in the source table and the concatenated expression for the fields to be compared in the target table are used as the data migration conversion relationship.

[0076] Optionally, the migration test module includes:

[0077] The migration content test result acquisition sub-module is used to perform a test on the migration content of the source data and the target data with reference to the table-level mapping relationship model, and obtain the migration content test result;

[0078] The migration index constraint test result acquisition sub-module is used to perform a migration index constraint test on the source data and the target data with reference to the field mapping table, and obtain the migration index constraint test result;

[0079] The migration sequence range test result acquisition sub-module is used to perform a migration sequence range test on the source data and the target data with reference to the table-level mapping relationship model, and obtain the migration sequence range test result.

[0080] Optionally, the migration content test result acquisition sub-module includes:

[0081] The data volume comparison result sub-unit is used to perform a data volume comparison on the source data and the target data with reference to the table-level mapping relationship model, and obtain the data volume comparison result;

[0082] The sampling comparison result acquisition subunit is used to perform sampling comparison on the source data and the target data with reference to the table-level mapping relationship model, and obtain the sampling comparison result;

[0083] The full-volume comparison result acquisition subunit is used to perform full-volume comparison on the source data and the target data with reference to the table-level mapping relationship model when both the data volume comparison result and the sampling comparison result pass the test, and obtain the full-volume comparison result.

[0084] Optionally, the data volume comparison result subunit is used to extract the source table name and the target table name from the table-level mapping relationship model;

[0085] Obtain the first quantity of the source data in the source table name in the source system, and obtain the second quantity of the target data in the target table name in the target system;

[0086] Judge whether the first quantity is the same as the second quantity. If they are the same, it is determined that the data volume comparison test passes; otherwise, it is determined that the data volume comparison test fails.

[0087] Optionally, the sampling comparison result acquisition subunit is used to extract the source table name and the target table name from the table-level mapping relationship model;

[0088] Randomly obtain the sampling source data in the source table name in the source system, and obtain the sampling target data corresponding to the sampling source data in the target table name in the target system;

[0089] Process the sampling source data according to the generative formula concatenated before and after the source table field to be compared expression, and obtain the processed result of the sampling source data;

[0090] Judge whether the processed result of the sampling source data is the same as the sampling target data. If they are the same, it is determined that the sampling comparison test passes; otherwise, it is determined that the sampling comparison test fails.

[0091] Optionally, the device further includes a difference content and distinction content display module, which is used to obtain the difference fields where the content in the fields to be compared is different, and the distinction content in the difference fields when it is determined that the sampling comparison test fails;

[0092] Display the difference fields and the distinction content.

[0093] Optionally, the full-volume comparison result acquisition subunit is used to extract the source table name and the target table name from the table-level mapping relationship model;

[0094] Obtain the source data of each record in the source table name in the source system, and obtain the target data corresponding to each record in the target table name in the target system;

[0095] Process the source data of each record according to the generative formula concatenated before and after the source table field to be compared expression, and obtain the processed result of the source data of each record

[0096] Calculate the first MD5 value of the processing result of the source data of each record and the second MD5 value of the target data corresponding to each record;

[0097] Determine whether the first MD5 value of the source data of each record is the same as the second MD5 value of the target data. If they are the same, it is determined that the full - volume comparison test passes; otherwise, it is determined that the full - volume comparison test fails.

[0098] Optionally, a migration index constraint test result acquisition sub - module is used to obtain the source fields with index constraint relationships and the target fields with index constraint relationships according to the field index relationships in the field mapping table;

[0099] Determine whether the source fields with index constraint relationships are consistent with the target fields with index constraint relationships. If so, it is determined that the migration index constraint test passes; otherwise, it is determined that the migration index constraint test fails.

[0100] Optionally, a migration sequence range test result acquisition sub - module is used to obtain the primary key fields of the source table and the primary key fields of the target table according to the table - level mapping relationship model;

[0101] Determine the source table sequence range according to the primary key fields of the source table and determine the target table sequence range according to the primary key fields of the target table;

[0102] Determine whether the source table sequence range is the same as the target table sequence range. If they are the same, it is determined that the migration sequence range test passes; otherwise, it is determined that the migration sequence range test fails.

[0103] Optionally, the device further includes a migration data availability test module, which is used to, after determining that the migration test passes, run a specified batch of jobs on the source system based on the source data at a specified time point to obtain the first batch of processing operation results;

[0104] Run a specified batch of jobs on the target system based on the target data at a specified time point to obtain the second batch of processing operation results;

[0105] Determine whether the first batch of processing operation results is the same as the second batch of processing operation results. If so, it is determined that the migration data is available; otherwise, it is determined that the migration data is unavailable.

[0106] The data migration test device provided by the embodiments of the present application can be used to execute the technical solutions of the data migration test method in the above - mentioned embodiments. Its implementation principle and technical effects are similar and will not be elaborated here.

[0107] It should be noted that the division of each module of the above device is only a division of logical functions. In actual implementation, it can be fully or partially integrated into a physical entity, or physically separated. And these modules can all be implemented in the form of software called by processing elements; they can also all be implemented in the form of hardware; some modules can also be implemented in the form of software called by processing elements, and some modules can be implemented in the form of hardware. In addition, all or part of these modules can be integrated together or can be independently implemented. The processing element here can be an integrated circuit with signal processing capabilities. In the implementation process, each step of the above method or each of the above modules can be completed by the integrated logic circuit in the processor element or the instructions in the form of software.

[0108] Figure 4 FIG. shows a schematic structural diagram of an electronic device 10 that can be used to implement an embodiment of the present invention. The electronic device is intended to represent various forms of digital computers, such as, laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as, personal digital processing, cellular phones, smart phones, wearable devices (such as helmets, glasses, watches, etc.) and other similar computing devices. The components shown herein, their connections and relationships, and their functions are only examples and are not intended to limit the implementation of the present invention described and / or claimed herein.

[0109] As Figure 4 shown, the electronic device 10 includes at least one processor 11, and a memory communicatively connected to the at least one processor 11, such as a read-only memory (ROM) 12, a random access memory (RAM) 13, etc. Among them, the memory stores a computer program executable by the at least one processor. The processor 11 can execute various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 12 or the computer program loaded from the storage unit 18 into the random access memory (RAM) 13. In the RAM 13, various programs and data required for the operation of the electronic device 10 can also be stored. The processor 11, the ROM 12, and the RAM 13 are connected to each other through a bus 14. The input / output (I / O) interface 15 is also connected to the bus 14.

[0110] Multiple components in the electronic device 10 are connected to the I / O interface 15, including: an input unit 16, such as a keyboard, a mouse, etc.; an output unit 17, such as various types of displays, speakers, etc.; a storage unit 18, such as a disk, an optical disc, etc.; and a communication unit 19, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information / data with other devices via a computer network such as the Internet and / or various telecommunication networks.

[0111] The processor 11 can be various general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The processor 11 executes the various methods and processes described above, such as the living body detection method.

[0112] In some embodiments, the living body detection method can be implemented as a computer program, which is tangibly contained in a computer-readable storage medium, such as the storage unit 18. In some embodiments, part or all of the computer program can be loaded and / or installed onto the electronic device 10 via the ROM 12 and / or the communication unit 19. When the computer program is loaded into the RAM 13 and executed by the processor 11, one or more steps of the living body detection method described above can be executed. Alternatively, in other embodiments, the processor 11 can be configured to execute the living body detection method by any other suitable means (e.g., by means of firmware).

[0113] The various embodiments of the systems and technologies described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include: being implemented in one or more computer programs, which can be executed and / or interpreted on a programmable system including at least one programmable processor, the programmable processor can be a special or general-purpose programmable processor, can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit the data and instructions to the storage system, the at least one input device, and the at least one output device.

[0114] A computer program for implementing the method of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general purpose computer, a special purpose computer, or other programmable data processing apparatus, such that the computer programs, when executed by the processor, cause the functions / operations specified in the flowchart and / or block diagram to be implemented. The computer program may be executed entirely on the machine, partly on the machine, as a stand-alone software package partly on the machine and partly on a remote machine or entirely on the remote machine or server.

[0115] In the context of the present invention, a computer-readable storage medium may be a tangible medium that can contain or store a computer program for use by or in connection with an instruction execution system, apparatus, or device. The computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. Alternatively, the computer-readable storage medium may be a machine-readable signal medium. More specific examples of the machine-readable storage medium would include an electrical connection based on one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0116] In order to provide interaction with a user, the systems and techniques described herein may be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or a trackball) by which the user can provide input to the electronic device. Other kinds of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form (including acoustic input, voice input, or tactile input).

[0117] The systems and techniques described herein can be implemented in a computing system including backend components (e.g., as a data server), or a computing system including middleware components (e.g., an application server), or a computing system including frontend components (e.g., a user computer having a graphical user interface or a web browser through which a user can interact with an implementation of the systems and techniques described herein), or a computing system including any combination of such backend components, middleware components, or frontend components. The components of the system can be interconnected to each other by digital data communication in any form or medium (e.g., a communication network). Examples of communication networks include: local area network (LAN), wide area network (WAN), blockchain network, and the Internet.

[0118] The computing system can include a client and a server. The client and the server are generally remote from each other and typically interact through a communication network. The client-server relationship is created by computer programs running on respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or a cloud host, which is a host product in the cloud computing service system, and solves the defects of difficult management and weak business scalability existing in traditional physical hosts and VPS services.

[0119] An embodiment of the present invention also provides a computer program product, including a computer program which, when executed by a processor, implements the migration data testing method provided in any embodiment of the present application.

[0120] In the process of implementing the computer program product, computer program code for performing the operations of the present invention can be written in one or more programming languages or combinations thereof. The programming languages include object-oriented programming languages such as Java, Smalltalk, C++, and also include conventional procedural programming languages such as the "C" language or similar programming languages. The program code can be executed entirely on the user computer, partially on the user computer, executed as an independent software package, partially on the user computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer can be connected to the user computer through any type of network - including a local area network (LAN) or a wide area network (WAN) - or can be connected to an external computer (e.g., by connecting through the Internet using an Internet service provider).

[0121] Note that the above is only a preferred embodiment of the present invention and the technical principles applied. Those skilled in the art will understand that the present invention is not limited to the specific embodiments described herein. Various obvious changes, re-adjustments, and substitutions can be made by those skilled in the art without departing from the protection scope of the present invention. Therefore, although the present invention has been described in more detail through the above embodiments, the present invention is not limited to the above embodiments. Without departing from the concept of the present invention, it can also include more other equivalent embodiments, and the scope of the present invention is determined by the scope of the appended claims.

Claims

1. A data migration test method, characterized in that, Including: Obtain a field mapping table from the source system to the target system, and obtain a table-level mapping relationship model according to the field mapping table, where the table-level mapping relationship model contains data migration conversion relationships; Obtain the source data in the source system and the target data located in the target system after migration; Perform a migration test on the source data and the target data with reference to the field mapping table and the table-level mapping relationship model; Wherein, the field mapping table contains the source table name, source table fields, processing method, processing statements, filtering conditions, target table name, target table fields, primary key identifier, and field index relationship; Wherein, obtaining the table-level mapping relationship model according to the field mapping table includes: Determine the fields to be compared from the source table fields, and generate source table expression for each field to be compared according to the processing method, processing statements, and filtering conditions of the fields to be compared; Concatenate the source table expression for each field to be compared to generate a concatenated generated expression of the source table fields to be compared before and after; Determine the fields to be compared from the target table fields, and concatenate each field to be compared in the target table to generate a concatenated generated expression of the target table fields to be compared before and after; And construct the table-level mapping relationship model according to the source table name, the target table name, the source table primary key field, the target table primary key field, the concatenated generated expression of the source table fields to be compared before and after, and the concatenated generated expression of the target table fields to be compared before and after; Wherein, the concatenated generated expression of the source table fields to be compared before and after and the concatenated generated expression of the target table fields to be compared before and after are used as the data migration conversion relationships.

2. The method according to claim 1, wherein The performing a migration test on the source data and the target data with reference to the field mapping table and the table-level mapping relationship model includes: Perform a test on the migration content of the source data and the target data with reference to the table-level mapping relationship model to obtain a migration content test result; Perform a migration index constraint test on the source data and the target data with reference to the field mapping table to obtain a migration index constraint test result; Perform a test on the migration sequence range of the source data and the target data with reference to the table-level mapping relationship model to obtain a migration sequence range test result.

3. The method according to claim 2, characterized in that, The performing a test on the migration content of the source data and the target data with reference to the table-level mapping relationship model to obtain a migration content test result includes: Perform a data volume comparison on the source data and the target data with reference to the table-level mapping relationship model to obtain a data volume comparison result; Perform a sampling comparison on the source data and the target data with reference to the table-level mapping relationship model to obtain a sampling comparison result; When both the data volume comparison result and the sampling comparison result pass the test, perform a full volume comparison on the source data and the target data with reference to the table-level mapping relationship model to obtain a full volume comparison result.

4. The method according to claim 3, wherein The performing a data volume comparison on the source data and the target data with reference to the table-level mapping relationship model to obtain a data volume comparison result includes: Extract the source table name and the target table name from the table-level mapping relationship model; Obtain the first quantity of the source data from the source table name in the source system, and obtain the second quantity of the target data from the target table name in the target system; Determine whether the first quantity is the same as the second quantity. If they are the same, it is determined that the data volume comparison test passes; otherwise, it is determined that the data volume comparison test fails.

5. The method according to claim 3, wherein Sampling comparison is performed on the source data and the target data with reference to the table-level mapping relationship model to obtain the sampling comparison result, including: Extract the source table name and the target table name from the table-level mapping relationship model; Randomly obtain sampled source data from the source table name in the source system, and obtain the corresponding sampled target data from the target table name in the target system; Process the sampled source data according to the front and back splicing generation formula of the source table fields to be compared to obtain the processed result of the sampled source data; Determine whether the processed result of the sampled source data is the same as the sampled target data. If they are the same, it is determined that the sampling comparison test passes; otherwise, it is determined that the sampling comparison test fails.

6. The method according to claim 5, characterized in that, The method further includes: When it is determined that the sampling comparison test fails, obtain the difference fields in which the content in the fields to be compared is different, and the difference content in the difference fields; Display the difference fields and the difference content.

7. The method according to claim 3, wherein Full-volume comparison is performed on the source data and the target data with reference to the table-level mapping relationship model to obtain the full-volume comparison result, including: Extract the source table name and the target table name from the table-level mapping relationship model; Obtain the source data of each record from the source table name in the source system, and obtain the corresponding target data of each record from the target table name in the target system; Process the source data of each record according to the front and back splicing generation formula of the source table fields to be compared to obtain the processed result of the source data of each record; Calculate the first MD5 value of the processed result of the source data of each record and the second MD5 value of the corresponding target data of each record; Determine whether the first MD5 value of the source data of each record is the same as the second MD5 value of the target data. If they are the same, it is determined that the full-volume comparison test passes; otherwise, it is determined that the full-volume comparison test fails.

8. The method according to claim 2, characterized in that, The method further includes: When it is determined that the fields to be compared include code value fields, obtain the records of the new and old system code value mapping table, where the records of the new and old system code value mapping table include the conversion relationship of the code value fields in the new and old systems; Obtain the source code value in the code value field of the source table name in the source system, and the target code value in the target table name in the target system; Determine whether the source code value and the target code value conform to the conversion relationship. If so, it is determined that the code value conversion test passes; otherwise, it is determined that the code value conversion test fails.

9. The method according to claim 2, wherein Migration index constraint test is performed on the source data and the target data with reference to the field mapping table to obtain the migration index constraint test result, including: Obtain the source fields with index constraint relationships and the target fields with index constraint relationships according to the field index relationships in the field mapping table; Determine whether the source field with the index constraint relationship is consistent with the target field with the index constraint relationship. If so, determine that the migration index constraint test passes; otherwise, determine that the migration index constraint test fails.

10. The method according to claim 2, characterized in that, Testing the migration sequence range for the source data and the target data with reference to the table-level mapping relationship model, and obtaining the migration sequence range test result, including: Obtain the primary key field of the source table and the primary key field of the target table according to the table-level mapping relationship model; Determine the source table sequence range according to the primary key field of the source table, and determine the target table sequence range according to the primary key field of the target table; Determine whether the source table sequence range is the same as the target table sequence range. If they are the same, determine that the migration sequence range test passes; otherwise, determine that the migration sequence range test fails.

11. The method according to claim 1, wherein After testing the migration of the source data and the target data with reference to the field mapping table and the table-level mapping relationship model, it further includes: When it is determined that the migration test passes, run the specified batch job on the source system based on the source data at the specified time point to obtain the first batch of processing operation results; Run the specified batch job on the target system based on the target data at the specified time point to obtain the second batch of processing operation results; Determine whether the first batch of processing operation results is the same as the second batch of processing operation results. If so, determine that the migrated data is available; otherwise, determine that the migrated data is unavailable.

12. A data migration test device, characterized in that, It includes: A table-level mapping relationship model acquisition module, configured to obtain a field mapping table from the source system to the target system, and obtain a table-level mapping relationship model according to the field mapping table, where the table-level mapping relationship model includes a data migration conversion relationship; A data acquisition module, configured to acquire the source data in the source system and the target data located in the target system after migration; A migration test module, configured to test the migration of the source data and the target data with reference to the field mapping table and the table-level mapping relationship model; Wherein, the field mapping table includes the source table name, source table fields, processing method, processing statement, filtering condition, target table name, target table fields, primary key identifier, and field index relationship; Among them, the table-level mapping relationship model acquisition module is specifically configured to determine the fields to be compared from the source table fields, and generate source table expressions for each field to be compared according to the processing method, processing statement, and filtering condition of the fields to be compared; splice the source table expressions for each field to be compared to generate a spliced expression before and after the source table fields to be compared; determine the fields to be compared from the target table fields, and splice the target table fields to be compared to generate a spliced expression before and after the target table fields to be compared; and construct the table-level mapping relationship model according to the source table name, the target table name, the source table primary key field, the target table primary key field, the spliced expression before and after the source table fields to be compared, and the spliced expression before and after the target table fields to be compared; wherein, the spliced expression before and after the source table fields to be compared and the spliced expression before and after the target table fields to be compared are used as the data migration conversion relationship.

13. An electronic device, characterized in that, It includes: A processor, and a memory communicatively connected to the processor; The memory stores computer-executable instructions; The processor executes the computer-executable instructions stored in the memory to implement the method according to any one of claims 1-11.

14. A computer-readable storage medium, characterized in that, Computer-executable instructions are stored in the computer-readable storage medium, and when the computer-executable instructions are executed by a processor, they are used to implement the method according to any one of claims 1-11.

15. A computer program product, characterized in that, Including a computer program which, when executed by a processor, implements the method according to any one of claims 1-11.

Images