Consistency verification method and device for data migration, equipment and medium

Abstract

The invention relates to the technical field of big data, and discloses a data migration consistency verification method and device, equipment and a medium, and the method comprises the steps: in response to a verification instruction in a data migration process, determining a granularity level of field verification in the data migration process; determining at least one target dimension field suitable for consistency verification according to the granularity level; querying table data containing the target dimension field in a source database and a target database respectively; and taking the table data containing the target dimension field as to-be-checked data, and performing at least one time of difference check on the to-be-checked data to obtain a check result. It can be guaranteed that consistency verification in the data migration process has higher flexibility, a large amount of development and test manpower does not need to be input, and the data migration efficiency is improved.

Description

Technical Field

[0001] The present invention relates to the fields of big data technology and data processing technology, and in particular to a consistency verification method, device, equipment and medium for data migration. Background Art

[0002] In today's digital age, data migration is becoming more and more common. It can migrate data from one database to another, or from one server to another. During the data migration process, data consistency is the process of ensuring that data remains consistent between different systems. If the data is inconsistent, it may cause system erroneous behavior or even data loss or damage. Therefore, data consistency verification can ensure the accuracy, integrity and reliability of the data.

[0003] During the data migration process, the relevant technology can perform consistency verification on the data by comparing the number of rows and the total number of records in the relevant tables in the source database and the target database. Since the comparison process requires manual data verification, the consistency verification process of the corresponding data migration requires a large amount of development and testing manpower, which affects the efficiency of data migration. Summary of the invention

[0004] The present invention provides a consistency verification method, device, computer equipment and medium for data migration to solve the technical problem that the consistency verification process of data migration in the related art requires a large amount of development and testing manpower, which affects the efficiency of data migration.

[0005] In a first aspect, a consistency verification method for data migration is provided, comprising:

[0006] In response to a verification instruction during the data migration process, determine a granularity level of field verification during the data migration process;

[0007] Determining at least one target dimension field suitable for consistency checking according to the granularity level;

[0008] Query the table data containing the target dimension field in the source database and the target database respectively;

[0009] The table data containing the target dimension field is used as the data to be checked, and at least one difference check is performed on the data to be checked to obtain a check result.

[0010] In a second aspect, a consistency check device for data migration is provided, comprising:

[0011] A first determination module, configured to determine a granularity level of field verification during the data migration process in response to a verification instruction during the data migration process;

[0012] A second determination module, configured to determine at least one target dimension field suitable for consistency checking according to the granularity level;

[0013] A query module, used to query table data containing target dimension fields in the source database and the target database respectively;

[0014] The verification module is used to take the table data containing the target dimension field as the data to be verified, perform at least one difference verification on the data to be verified, and obtain a verification result.

[0015] In a third aspect, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above-mentioned data migration consistency verification method when executing the computer program.

[0016] In a fourth aspect, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the consistency verification method for data migration are implemented.

[0017] In the scheme implemented by the consistency verification method, device, computer equipment and storage medium of the above-mentioned data migration, the client can respond to the verification instruction in the data migration process to determine the granularity level of field verification in the data migration process, determine at least one target dimension field suitable for consistency verification according to the granularity level, query the table data containing the target dimension field in the source database and the target database respectively, use the table data containing the target dimension field as the data to be verified, perform at least one difference verification on the data to be verified, obtain the verification result, and return the verification result to the client. In the present invention, by determining the granularity level of field verification in the data migration process, the dimension field suitable for consistency verification is adaptively adjusted, and further performing at least one difference verification on the table data containing the corresponding dimension field, so that the consistency verification of the data migration process has higher flexibility, without the need to invest a large amount of development and testing manpower, thereby improving the efficiency of data migration. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative labor.

[0019] Figure 1 It is a schematic diagram of an application environment of a consistency verification method for data migration in one embodiment of the present invention;

[0020] Figure 2It is a flow chart of a consistency verification method for data migration in one embodiment of the present invention;

[0021] Figure 3 yes Figure 1 A schematic flow chart of a specific implementation of step S10;

[0022] Figure 4 is a flow chart of a consistency verification method for data migration in another embodiment of the present invention;

[0023] Figure 5 yes Figure 1 A schematic flow chart of a specific implementation of step S40;

[0024] Figure 6 is a flow chart of a consistency verification method for data migration in another embodiment of the present invention;

[0025] Figure 7 It is a structural schematic diagram of a consistency verification device for data migration in one embodiment of the present invention;

[0026] Figure 8 is a schematic diagram of a structure of a computer device in one embodiment of the present invention;

[0027] Fig. 9 It is another structural schematic diagram of a computer device in one embodiment of the present invention. DETAILED DESCRIPTION

[0028] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0029] The data migration consistency verification method provided by the embodiment of the present invention can be applied in the following aspects: Figure 1In an application environment, wherein the client responds to the verification instruction in the data migration process, determines the granularity level of the field verification in the data migration process, determines at least one target dimension field suitable for consistency verification according to the granularity level, queries the table data containing the target dimension field in the source database and the target database respectively, uses the table data containing the target dimension field as the data to be verified, performs at least one difference verification on the data to be verified, obtains the verification result, and returns the verification result to the client. In the present invention, by determining the granularity level of the field verification in the data migration process, the dimension field suitable for consistency verification is adaptively adjusted, and further performs at least one difference verification on the table data containing the corresponding dimension field, so that the consistency verification of the data migration process has higher flexibility, without investing a lot of development and testing manpower, and improving the efficiency of data migration. Among them, the client can be but is not limited to various personal computers, laptops, smart phones, tablet computers and portable wearable devices. The server can be implemented with an independent server or a server cluster consisting of multiple servers. The present invention is described in detail below through specific embodiments.

[0030] See also Figure 2 As shown, Figure 2 A flow chart of a consistency verification method for data migration provided by an embodiment of the present invention includes the following steps:

[0031] S10: In response to a verification instruction during the data migration process, determine a granularity level of field verification during the data migration process.

[0032] The consistency verification method of data migration provided by the present invention can be applied to data processing platforms in various application scenarios or business platforms with data migration requirements. Specifically, in medical application scenarios, with the continuous development of medical technology and the improvement of the informationization level of medical institutions, the amount of medical data has increased dramatically. These data include basic patient information (such as name, gender, date of birth, contact information, etc.), medical record information (symptom description, diagnosis results, treatment process, etc.), inspection and test reports (X-ray, CT, blood test, etc.), and medical order information (drug prescriptions, treatment plans, etc.). At the same time, medical institutions may update information systems, such as upgrading from old hospital management systems (HIS) to more advanced electronic medical record systems (EMR), or integrating data centers, which requires data migration. The accuracy and integrity of medical data are directly related to the diagnosis and treatment of patients. Incorrect data migration may lead to medical errors, and the security and compliance of data during the migration process must be guaranteed.

[0033] It is understandable that in order to ensure the accuracy of the data migration process, consistency testing is required when converting data from one storage system or format to another. If there is no consistency testing, data may be wrong in the extraction, conversion and loading stages. In one application scenario, due to the possible differences in the data type definitions of the source database and the target database, field errors may occur in the data migration. For example, the date field in the source database is in the "YYYY-MM-DD" format, and the target database requires the "MM / DD / YYYY" format. If consistency testing is not performed, the date data may be stored incorrectly, affecting the accuracy of the decision. In another application scenario, since data usually has an inherent structure and association, the association between fields in the data migration will be destroyed. For example, in an e-commerce system, the user ID field in the order table is associated with the user ID in the user table, and the product ID field is associated with the product ID in the product information table. If consistency testing is not performed, these associations may be destroyed, affecting the normal operation of the business process.

[0034] In this embodiment, field verification is used as a consistency check method before and after data migration. The granularity level can be divided according to the data migration requirements. When the verification instruction in the data migration process is triggered, the granularity level of the field verification in the data migration process can be determined according to user requirements or the default method.

[0035] In an implementable scenario, field checking can be divided into granular levels according to the needs of anomaly location during data migration. Field checking can also be divided into granular levels according to effective resource utilization and time requirements during data migration, so as to divide field checking into field level, record level, and table level or data set level.

[0036] In the process of dividing the granularity of field verification according to the needs of locating anomalies during data migration, if the granularity is divided into field level, when data migration is abnormal, field level verification can help quickly locate the inconsistent field position in a large amount of verification data. For example, if the date data in a report is found to be abnormal after data migration, the data format, type and value range of the date field can be directly checked through field level verification. If the date field is in the "YYYY-MM-DD" format in the source system, and the target system requires the "MM / DD / YYYY" format, and it is not converted correctly during the migration process, this granularity division can focus the consistency detection in the migration process on the specific format conversion link. Here, field level verification can check the data type of each field in detail, which helps to fix the problem in time. If the granularity is divided into record level, when data migration is abnormal, record level verification helps to find problems in data association and record integrity. For example, in a database system containing multiple related tables, such as the order table, user table, and product table in an e-commerce system, record-level validation can check whether the association between order records and user records and product records is correct. If the user finds that the corresponding user information cannot be found through the order record, record-level validation can be used to check whether the association between the user ID field in the order table and the user ID field in the user table is incorrect during the migration process. For the case where the granularity level is divided into table level or data set level, when there are problems with the data volume or overall business logic, table-level validation division becomes very important. For example, if the amount of data in the inventory table of the inventory management system is greatly reduced after data migration, table-level validation can quickly determine whether there is a problem with the data migration process of the entire table, or whether some data is lost because the data cleaning rules are too strict.

[0037] In the process of dividing the granularity of field verification according to the effective resource utilization and time requirements in the data migration process, for the case where the granularity is divided into field level, field level verification can be used as a quick check method in the initial verification and testing stage of data migration. Because it focuses on the details of each field, some obvious problems such as data format and type can be discovered at an early stage. For example, in the data migration test of a small database, field level verification can quickly discover some simple problems that may affect the subsequent process, such as garbled characters caused by character encoding errors, and it is relatively simple to fix these problems, without reprocessing the entire data structure or a large number of records, saving time and resources. For the case where the granularity is divided into record level, record level verification balances the accuracy and efficiency of troubleshooting in ensuring the correct data association and record integrity. In the data migration process, checking the association relationship and integrity between records does not require in-depth details of each field like field level verification, but it is more specific than table level verification. For example, in the data migration of a customer relationship management system, checking whether the association between customer records and contact records is correct through record level verification can avoid customer information confusion caused by association errors. For situations where the granularity level is divided into table level or data set level, table-level validation is very useful in the later stages of data migration or in large-scale data migration scenarios. When the amount of data is huge, performing table-level validation first can quickly understand the overall situation of data migration. For example, when migrating a data warehouse containing millions of records from one platform to another, first perform table-level data volume and business logic validation to determine whether the data migration is basically successful.

[0038] In a specific implementation scenario, the granularity level of the field check can be determined by the text description information provided by the user. Figure 3 As shown, in step S10, that is, in response to the verification instruction in the data migration process, determining the granularity level of the field verification in the data migration process includes the following steps:

[0039] S11: In response to a verification instruction during the data migration process, receiving text description information of field verification during the data migration process.

[0040] S12: Convert the text description information of the field verification into text blocks with different verification functions through a large language model.

[0041] S13: Determine the granularity level of field verification during data migration according to the verification function corresponding to the text block.

[0042] For steps S11-S13, considering the different consistency requirements of different users in the data migration process, the text description information of the field verification in the data migration process can be received by user-defined input. The text description information can be the text description information for field format verification. For data migration involving multiple source systems and field docking with the target system, in terms of format verification, the format specification of each field needs to be strictly reviewed. For character fields, such as name, address, etc., the source system may have different case rules or character length restrictions. It is necessary to ensure that it complies with its established format after migration to the target system. For example, the name field in the source system allows all uppercase input, while the target system requires the first letter to be uppercase and the rest to be lowercase. At this time, the format of the name field should be checked one by one according to the conversion rules. The text description information can also be the text description information for data type verification. Data type verification is a key link in field verification. Fields defined as integer types in the source system, such as employee numbers, order numbers, etc., must still remain integer types in the target system after data migration. If there is a data type mismatch, it may cause data storage errors or exceptions when the program reads data. For example, if the original integer order number is converted to a character type by mistake, it may cause sorting errors or matching failures in the order query and processing process. The text description information can also be text description information for value range verification, which is intended to ensure that the data values ​​in the source system can be reasonably mapped to the allowed value range of the target system. Taking the score field as an example, the source system score range may be 0-100, while the target system may divide the scores into grades, such as "excellent" (90-100), "good" (80-89), etc. During data migration, the value of each score field needs to be checked according to the set conversion rules to ensure that it is correctly converted to the grade value in the target system.

[0043] It is understandable that, since the text description information of the field check is not a standard field expression form, the text description information of the field check can be converted into text blocks with different check functions through the large language model, and then the granularity level of the field check during the data migration process is determined according to the check function corresponding to the text block. Specifically, the check function corresponding to the text block can be matched with different check blocks to obtain the check blocks that match the text blocks; the granularity level of the field check during the data migration process is determined according to the check blocks that match the text blocks.

[0044] In this embodiment, different check blocks may include but are not limited to format check blocks, data type check blocks, value range check blocks, field relevance check blocks, and field integrity check blocks.

[0045] For the format check block, data migration involves the connection of multiple fields between the source system and the target system. In terms of the corresponding check function, the format specifications of each field need to be strictly reviewed. For character fields, such as name, address, etc., the source system may have different case rules or character length restrictions. It is necessary to ensure that it complies with the established format after migration to the target system. For example, the name field in the source system allows all uppercase input, while the target system requires the first letter to be uppercase and the rest to be lowercase. At this time, the format check block is equivalent to a block that checks the format one by one according to the format conversion rules.

[0046] For the data type check block, data type consistency is the key link in field verification. Fields defined as integer types in the source system must remain integer types in the target system after data migration. If data types do not match, data storage errors or exceptions when the program reads data may occur. For example, if the original integer order number is converted to a character type by mistake, sorting errors or matching failures may occur in the order query and processing process. For Boolean fields, the logic of true or false should remain unchanged before and after migration. Boolean fields that indicate whether the user is activated in the source system should be correctly identified as Boolean types in the target system after data migration. At this time, the format check block is equivalent to a block that checks the data types one by one according to the data type conversion rules.

[0047] For the range check word block, the range check of each field is to ensure that the data value in the source system can be reasonably mapped to the allowed range of the target system. Taking the grade field as an example, the source system grade range may be 0-100, while the target system may divide the grades into grades, such as "excellent" (90-100), "good" (80-89), etc. During migration, the value of each grade field needs to be checked according to the set conversion rules to ensure that it is correctly converted to the grade value in the target system. For enumeration type fields, such as gender (male, female) and order status (submitted, processed, canceled, etc.), it is necessary to confirm that the enumeration value in the source system can accurately correspond to the enumeration set of the target system. If the order status in the source system has "pending review" and the target system does not have this status, it needs to be converted to a similar status in the target system according to the predetermined business logic, and the accuracy of the conversion is checked to prevent data from being processed normally or business rule execution errors due to value range mismatch. At this time, the range check word block is equivalent to a word block that checks the range of the value range one by one according to the range conversion rules.

[0048] For the field association check block, field association check is particularly important in a database structure involving multi-table association. For example, in an order management system, the user ID field in the order table is associated with the user ID field in the user table. For each migrated order record, it is necessary to verify whether the associated user record is correct. By checking whether the foreign key constraint is valid, it is ensured that the order information can be accurately linked to the corresponding user information. In a many-to-many relationship scenario, such as in a student course selection system, the student table and the course table are associated through the course selection record table. It is necessary to check each record in the course selection record table to confirm that the combination of student ID and course ID still correctly reflects the course selection relationship between students and courses after migration. Any error in the associated field may cause interruption or erroneous execution of data query, statistics and business processes. At this time, the field association check block is equivalent to a block that checks the field association one by one according to the association between the fields.

[0049] For the field integrity check block, the field integrity check focuses on whether all fields of each record are completely migrated to the target system. In the user information table, fields such as user name, password, email, contact information, etc. should all be completely present in the migrated record, especially the required fields, such as the name field when the user registers, which must not have blank values ​​after migration. For complex fields containing multiple subfields, such as the address field (including subfields such as province, city, district, and detailed address), check whether each subfield is completely migrated and there is no data loss or incorrect merging. If there are missing or incorrect fields, it may affect business operations based on complete data, such as user information query, order delivery address confirmation, etc. At this time, the field integrity check block is equivalent to a block that checks the integrity of the fields one by one according to the field integrity rules.

[0050] S20: Determine at least one target dimension field suitable for consistency verification according to the granularity level.

[0051] It is understandable that the granularity level is a verification dimension used to compare fields for consistency during data migration. Accordingly, the verification fields applicable to different verification dimensions are also different.

[0052] For the fine granularity level, from the perspective of data format, the data dictionary of the system and the target system can be compared in detail to find out the fields with obvious differences in format. For example, some fields with different character encoding, length restrictions, and upper and lower case rules in text fields are the target dimension fields for verification. For the medium granularity level, all foreign key fields representing the association between tables can be found in the database relationship diagram or data model. For example, in an e-commerce system, the order table is associated with the user table through the user ID field, and the product table is associated with the product ID field. Then the user ID and product ID fields in the order table are the target dimension fields for verification. For each order record, it is necessary to ensure that these fields can be correctly associated with the corresponding user and product records. For the coarse granularity level, before and after data migration, compare the number of records in the source table and the target table. For example, in the migration of a customer information table, the change in the number of records can intuitively reflect whether the data is completely migrated. If the source table has 1,000 customer records, the target table should also be close to this number. Then the field that counts the data volume of each table is the target dimension field for verification.

[0053] S30: Query the table data containing the target dimension field in the source database and the target database respectively.

[0054] In this embodiment, a database query language may be used for querying. For a relational database, a basic query statement may be used, and then conditional query and associated query may be added. For a non-relational database, a basic query operation may be used, and then associated query may be added.

[0055] Considering the query process of the data to be checked, further, after step S20, as Figure 4 As shown, the consistency verification method for data migration also includes the following steps:

[0056] S50: splicing the at least one target dimension field to generate a field query statement according to the spliced ​​fields.

[0057] Correspondingly, step S30: querying the table data containing the target dimension field in the source database and the target database respectively by executing the field query statement.

[0058] It should be noted that different types of databases have different ways of splicing fields in query statements. In relational databases, this includes basic query statements and field splicing when multiple tables are associated. Field splicing in basic query statements needs to be separated by punctuation marks. Field splicing in multiple table associated queries usually uses a join operation to associate tables, and then splices fields from different tables in the query statement.

[0059] S40: taking the table data containing the target dimension field as the data to be checked, performing at least one difference check on the data to be checked to obtain a check result.

[0060] In a specific implementation scenario, in order to accurately locate the difference problem, the process of performing multiple difference checks on the data to be checked needs to consider the previous check result. If the previous check result is inconsistent, a second difference check can be performed on the inconsistent data to be checked. Specifically, Figure 5 As shown, in step S40, the table data containing the target dimension field is used as the data to be checked, and at least one difference check is performed on the data to be checked to obtain the check result, including the following steps:

[0061] S41: taking the table data containing the target dimension field as the data to be checked, and performing a first difference check on the data to be checked to determine the table difference of the data to be checked.

[0062] S42: If there are table differences in the data to be checked, a second difference check is performed on the data to be checked with the table differences to obtain a check result.

[0063] In this embodiment, the first difference check on the data to be checked is equivalent to a coarse-grained field check process. Once the data to be checked is inconsistent after the coarse-grained check, it is necessary to further determine which fields may be related to the difference, and a second difference check is required for the inconsistent data part in the data to be checked, that is, a fine-grained field check. After the fine-grained field check, the cause of the difference can be analyzed. For example, in the sales order scenario, if after the first coarse-grained difference check, it is found that there is a difference between the total number of orders and the total sales, a second fine-grained difference check is performed on the total number of orders and the total sales. Here, the fine-grained fields include the order date, order status, customer ID, product ID and other fields in the order table, because these fields may affect the order statistics and sales calculation.

[0064] Specifically, the tabular data containing the target dimension fields can be used as the data to be verified, and the consistency of the data volume of the data to be verified can be compared; if the data volume of the data to be verified is inconsistent or there are differences in the fields of the data to be verified on the basis of the consistency of the data volume of the data to be verified, it is determined that there are tabular differences in the data to be verified; otherwise, it is determined that there are no tabular differences in the data to be verified.

[0065] In the above implementation, by first performing an overall check using coarse-grained fields, and then performing a secondary check on the relevant parts using fine-grained fields after anomalies are found, consistency issues in table data during migration or other operations can be checked more comprehensively and in-depth, which helps to promptly discover and resolve data inconsistencies.

[0066] Considering the abnormal reason investigation of the verification result, further, after step S40, as Figure 6 As shown, the consistency verification method for data migration also includes the following steps:

[0067] S60: If the verification result is that the data to be checked has consistency differences, a pre-trained difference recognition model is used to perform problem analysis on the data to be checked with consistency differences to obtain the causes of the differences.

[0068] S70: According to the cause of the difference, corresponding parameters of the source database and / or the target database are adjusted during the data migration process.

[0069] Among them, the difference recognition model is obtained by training using the verification results with consistency differences in the historical verification process as sample data. During the training process, the problem scope and data source can be determined first, and then the possible causes of the consistency problem are analyzed based on the problem scope and data source. The corresponding verification results are marked according to the causes of the consistency problem, and the sample data is trained according to the marked results to obtain the difference recognition model. After the verification results are subsequently input into the difference recognition model, the difference recognition model can output the consistency reasons accordingly. For example, inconsistent data types of source data and target data will lead to inconsistent data formats, incorrect mapping rules or complex data conversion problems during data migration will lead to data conversion errors, foreign key relationship errors or improper handling of many-to-many relationships will lead to data association problems, incomplete data extraction or data loading problems will lead to data loss or duplication.

[0070] It is understandable that after determining the cause of the difference, considering the impact of the difference cause on the data migration process, it is necessary to adjust the corresponding parameters of the source database and / or the target database. Different difference causes are usually handled in different ways. For the difference cause of inconsistent data format, if the data type is found to be inconsistent, such as the source data is an integer type, and the target data is incorrectly converted to a character type, it is necessary to perform a data type conversion operation. For a relational database, a data type conversion function can be used. For the difference problem of data conversion error, when the data mapping rule error is found, review and modify the mapping document of the data migration. For example, if the order status in the source system and the order status in the target system are mapped incorrectly, it is necessary to redefine the mapping relationship according to the correct business logic, and then write a data conversion script to reconvert the migrated data according to the new mapping rule. For the difference cause of data association problems, the foreign key relationship error can first determine the correct foreign key association rule. In a relational database, the foreign key constraint can be updated by modifying the table structure or using a database management tool, and then the value of the associated field can be checked and updated according to the correct association rule.

[0071] It should be understood that the order of execution of the steps in the above embodiment does not necessarily mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiment of the present invention.

[0072] In one embodiment, a consistency check device for data migration is provided, and the consistency check device for data migration corresponds one-to-one to the consistency check method for data migration in the above embodiment. Figure 7 As shown, the consistency verification device for data migration includes a first determination module 101, a second determination module 102, a query module 103 and a verification module 104. The detailed description of each functional module is as follows:

[0073] A first determination module 101, for determining a granularity level of field verification during data migration in response to a verification instruction during data migration;

[0074] A second determination module 102, configured to determine at least one target dimension field suitable for consistency checking according to the granularity level;

[0075] A query module 103, used to query the table data containing the target dimension field in the source database and the target database respectively;

[0076] The verification module 104 is used to take the table data containing the target dimension field as the data to be verified, perform at least one difference verification on the data to be verified, and obtain a verification result.

[0077] In one embodiment, the apparatus further comprises:

[0078] a splicing module, configured to splice the at least one target dimension field suitable for consistency verification after determining the at least one target dimension field suitable for consistency verification according to the granularity level of the field verification, so as to generate a field query statement according to the spliced ​​fields;

[0079] Accordingly, the query module 103 is specifically used for:

[0080] The table data containing the target dimension field is queried in the source database and the target database respectively by executing the field query statement.

[0081] In one embodiment, the first determining module 101 is specifically configured to:

[0082] In response to a verification instruction during the data migration process, receiving text description information of field verification during the data migration process;

[0083] Converting the text description information of the field verification into text blocks with different verification functions through a large language model;

[0084] According to the checking function corresponding to the text block, the granularity level of the field checking during the data migration process is determined.

[0085] In one embodiment, the first determining module 101 is further configured to:

[0086] Matching the checking function corresponding to the text block with different checking blocks to obtain a checking block that matches the text block;

[0087] The granularity level of field verification during data migration is determined according to the verification word blocks that match the text word blocks.

[0088] In one embodiment, the verification module 103 is specifically configured to:

[0089] The table data containing the target dimension field is used as the data to be checked, and a first difference check is performed on the data to be checked to determine the table difference of the data to be checked;

[0090] If there are table differences in the data to be checked, a second difference check is performed on the data to be checked with the table differences to obtain a check result.

[0091] In one embodiment, the verification module 102 is further configured to:

[0092] The table data containing the target dimension field is used as the data to be checked, and the consistency of the data volume of the data to be checked is compared;

[0093] If the data volume of the data to be checked is inconsistent or there are differences in the fields of the data to be checked based on the consistency of the data volume of the data to be checked, it is determined that there are table differences in the data to be checked; otherwise, it is determined that there are no table differences in the data to be checked.

[0094] In one embodiment, the apparatus further comprises:

[0095] A third determination module is used for performing at least one difference check on the data to be checked after taking the table data containing the target dimension field as the data to be checked, and obtaining the check result, if the check result is that the data to be checked has a consistency difference, using a pre-trained difference recognition model to perform problem analysis on the data to be checked with the consistency difference to obtain the cause of the difference, wherein the difference recognition model is obtained by training using the check results with consistency differences in the historical check process as sample data;

[0096] The adjustment module is used to adjust corresponding parameters of the source database and / or the target database during the data migration process according to the cause of the difference.

[0097] The present invention provides a consistency verification device for data migration, which adaptively adjusts the dimensional fields suitable for consistency verification by determining the granularity level of field verification during the data migration process, and further performs at least one difference verification on the table data containing the corresponding dimensional fields, so that the consistency verification of the data migration process has higher flexibility, does not require a large amount of development and testing manpower, and improves the data migration efficiency.

[0098] For the specific definition of the consistency verification device for data migration, please refer to the definition of the intelligent question and answer method above, which will not be repeated here. Each module in the above-mentioned consistency verification device for data migration can be implemented in whole or in part by software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0099] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Figure 8As shown. The computer device includes a processor, a memory, a network interface and a database connected via a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile and / or volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external client via a network connection. When the computer program is executed by the processor, it implements the functions or steps of a consistency verification method for data migration on the server side.

[0100] In one embodiment, a computer device is provided. The computer device may be a client, and its internal structure diagram may be as follows: Fig. 9 As shown. The computer device includes a processor, a memory, a network interface, a display screen and an input device connected via a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external server via a network connection. When the computer program is executed by the processor, it implements the functions or steps of a client side of a consistency verification method for data migration.

[0101] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the following steps when executing the computer program:

[0102] In response to a verification instruction during the data migration process, determine a granularity level of field verification during the data migration process;

[0103] Determining at least one target dimension field suitable for consistency checking according to the granularity level;

[0104] Query the table data containing the target dimension field in the source database and the target database respectively;

[0105] The table data containing the target dimension field is used as the data to be checked, and at least one difference check is performed on the data to be checked to obtain a check result.

[0106] In one embodiment, a computer readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the following steps are implemented:

[0107] In response to a verification instruction during the data migration process, determine a granularity level of field verification during the data migration process;

[0108] Determining at least one target dimension field suitable for consistency checking according to the granularity level;

[0109] Query the table data containing the target dimension field in the source database and the target database respectively;

[0110] The table data containing the target dimension field is used as the data to be checked, and at least one difference check is performed on the data to be checked to obtain a check result.

[0111] It should be noted that the above functions or steps that can be implemented by the computer-readable storage medium or computer device can refer to the relevant descriptions on the server side and the client side in the aforementioned method embodiment. To avoid repetition, they will not be described one by one here.

[0112] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0113] Those skilled in the art can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example. In actual applications, the above-mentioned functions can be distributed and completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0114] The embodiments described above are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included in the protection scope of the present invention.

Claims

1. A consistency verification method for data migration, characterized in that: include: In response to a verification instruction during the data migration process, determine a granularity level of field verification during the data migration process; Determining at least one target dimension field suitable for consistency checking according to the granularity level; Query the table data containing the target dimension field in the source database and the target database respectively; The table data containing the target dimension field is used as the data to be checked, and at least one difference check is performed on the data to be checked to obtain a check result.

2. The consistency verification method for data migration according to claim 1, characterized in that: After determining at least one target dimension field suitable for consistency checking according to the granularity level of the field check, the method further includes: splicing the at least one target dimension field to generate a field query statement according to the spliced ​​fields; Accordingly, querying the table data containing the target dimension field in the source database and the target database respectively includes: The table data containing the target dimension field is queried in the source database and the target database respectively by executing the field query statement.

3. The consistency verification method for data migration according to claim 1, characterized in that: The step of determining the granularity level of field verification during the data migration process in response to the verification instruction during the data migration process includes: In response to a verification instruction during the data migration process, receiving text description information of field verification during the data migration process; Converting the text description information of the field verification into text blocks with different verification functions through a large language model; According to the checking function corresponding to the text block, the granularity level of the field checking during the data migration process is determined.

4. The consistency verification method for data migration according to claim 3, characterized in that: Determining the granularity level of field verification during data migration according to the verification function corresponding to the text block includes: Matching the checking function corresponding to the text block with different checking blocks to obtain a checking block that matches the text block; The granularity level of field verification during data migration is determined according to the verification word blocks that match the text word blocks.

5. The consistency verification method for data migration according to claim 1, characterized in that: The step of taking the table data containing the target dimension field as the data to be checked, performing at least one difference check on the data to be checked, and obtaining a check result includes: The table data containing the target dimension field is used as the data to be checked, and a first difference check is performed on the data to be checked to determine the table difference of the data to be checked; If there are table differences in the data to be checked, a second difference check is performed on the data to be checked with the table differences to obtain a check result.

6. The consistency verification method for data migration according to claim 5, characterized in that: The step of taking the table data containing the target dimension field as the data to be checked and performing a first difference comparison on the data to be checked to determine the table difference of the data to be checked includes: The table data containing the target dimension field is used as the data to be checked, and the consistency of the data volume of the data to be checked is compared; If the data volume of the data to be checked is inconsistent or there are differences in the fields of the data to be checked based on the consistency of the data volume of the data to be checked, it is determined that there are table differences in the data to be checked; otherwise, it is determined that there are no table differences in the data to be checked.

7. The consistency verification method for data migration according to any one of claims 1 to 6, characterized in that: After taking the table data containing the target dimension field as the data to be checked, performing at least one difference check on the data to be checked, and obtaining the check result, the method further includes: If the verification result shows that the data to be verified has consistency differences, a pre-trained difference recognition model is used to analyze the problem of the data to be verified with consistency differences to obtain the cause of the difference. The difference recognition model is trained by using the verification results with consistency differences in the historical verification process as sample data; According to the cause of the difference, corresponding parameters of the source database and / or the target database are adjusted during the data migration process.

8. A consistency check device for data migration, characterized in that: include: A first determination module, configured to determine a granularity level of field verification during the data migration process in response to a verification instruction during the data migration process; A second determination module, configured to determine at least one target dimension field suitable for consistency checking according to the granularity level; A query module, used to query table data containing target dimension fields in the source database and the target database respectively; The verification module is used to take the table data containing the target dimension field as the data to be verified, perform at least one difference verification on the data to be verified, and obtain a verification result.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the consistency verification method for data migration as described in any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the consistency verification method for data migration as claimed in any one of claims 1 to 7 are implemented.

Images