Migration data verification method and device, equipment and storage medium

Abstract

The invention discloses a migration data verification method and device, equipment and a storage medium, and relates to the technical field of data processing, and the migration data verification method comprises the steps: obtaining a data verification range, a data type and a data association condition corresponding to original data and migration data; determining a migration data verification rule between the original data and the migration data based on the data association condition; and on the basis of the data type and the migration data verification rule, carrying out migration data verification on the original data and the migration data in the data verification range. According to the method, the personalized data verification requirement in the data migration process is met by constructing the data verification rule and the verification method with the flexible data verification range, and then the consistency and integrity of the data in the data migration process are verified, so that the accuracy, flexibility and verification efficiency of data verification in the data migration process are effectively improved; and the user satisfaction is improved.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a migration data verification method, device, equipment and storage medium. Background Art

[0002] In the data migration process of some enterprises, such as banks, data verification is a key link to ensure the consistency of data between the old and new models. Although there has been some progress in the current industry, there are still some significant defects.

[0003] For example, defects include insufficient verification flexibility, that is, the data migration process often involves reconstruction modes such as field type changes, field splitting and field merging. Most existing tools are confined to single-dimensional field matching and cannot flexibly respond to diversified data architecture changes, making it difficult to meet the diverse business needs of the data migration process; efficiency and cost dilemma, that is, the data migration process involves processing the entire amount of data, and the use of traditional verification methods for data verification will face the dilemma of extremely long verification cycles, which will not only affect the cycle of the entire migration project, but also consume a lot of time, manpower and computing resources; the lag in regulatory rule verification, that is, although regulatory norms continue to evolve, at the data verification level, the adaptability verification of historical data is not perfect enough. A large amount of historical data will be involved in data migration. Existing data verification tools often only focus on consistency verification of new and old data, and lack an effective mechanism to ensure consistency verification of historical data with the latest regulatory guidelines, which poses potential data quality risks. Summary of the invention

[0004] The main purpose of this application is to provide a migration data verification method, device, equipment and storage medium, aiming to meet the personalized data verification needs in the data migration process, and then verify the consistency and integrity of the data in the data migration process, so as to effectively improve the accuracy, flexibility and verification efficiency of data verification in the data migration process, and improve user satisfaction.

[0005] To achieve the above objectives, the present application proposes a migration data verification method, the method comprising:

[0006] Obtain data verification scope, data type, and data association conditions corresponding to the original data and the migrated data;

[0007] Determining migration data verification rules between the original data and the migration data based on the data association condition;

[0008] Based on the data type and the migration data verification rule, migration data verification is performed on the original data and the migration data within the data verification range.

[0009] In one embodiment, the determining, based on the data association condition, a migration data verification rule between the original data and the migration data includes:

[0010] Acquire data migration logic between the original data and the migrated data;

[0011] Based on the data migration logic and the data association condition, a migration data verification rule is obtained; and / or,

[0012] The data migration logic and the data association conditions are input into a verification rule matching model to obtain a migration data verification rule output by the verification rule matching model, wherein the verification rule matching model is obtained by iteratively training the historical data migration logic and the historical data association conditions.

[0013] In one embodiment, the training process of the verification rule matching model includes:

[0014] Obtain historical data migration logic and historical data association conditions;

[0015] Based on the preset verification rule database, knowledge is injected into the initial verification rule matching model;

[0016] The historical data migration logic and the historical data association conditions are input into the initial verification rule matching model after knowledge injection for iterative training to obtain a verification rule matching model.

[0017] In one embodiment, the performing migration data verification on the original data and the migration data within the data verification range based on the data type and the migration data verification rule includes:

[0018] Based on the data type, data division is performed on the original data and the migration data within the data verification range to obtain several groups of migration verification data;

[0019] Based on the migration data verification rules, performing data integrity verification and data consistency verification on each migration verification data;

[0020] Based on the migration data verification rule, data relevance verification is performed on the original data and the migration data within the data verification range.

[0021] In one embodiment, the data relevance verification of the original data and the migration data within the data verification scope based on the migration data verification rule includes:

[0022] Determine a first data association condition of the original data within the data verification range and a second data association condition of the migration data;

[0023] Based on the migration data verification rule, data relevance verification is performed on the first data association condition and the second data association condition.

[0024] In one embodiment, after performing migration data verification on the original data and the migration data within the data verification range based on the data type and the migration data verification rule, the method further includes:

[0025] Get the migration data verification results;

[0026] Performing data analysis on the migration data verification result to generate a migration data verification report;

[0027] The migration data verification report is pushed to the target user for the target user to view.

[0028] In one embodiment, the migration data verification result includes data comparison results, data comparison rules, and historical comparison records;

[0029] The performing data analysis on the migration data verification result to generate a migration data verification report includes:

[0030] Performing data integration analysis on the data comparison result, the data comparison rule and the historical comparison record to obtain a data analysis result;

[0031] Performing data visualization on the data analysis results to obtain visualized chart information;

[0032] The data analysis results and the visual chart information are associated and combined to generate a migration data verification report.

[0033] In addition, to achieve the above purpose, the present application also proposes a migration data verification device, the migration data verification device comprising:

[0034] An acquisition module, used to acquire data verification ranges, data types, and data association conditions corresponding to original data and migrated data;

[0035] a determination module, configured to determine a migration data verification rule between the original data and the migration data based on the data association condition;

[0036] The verification module is used to perform migration data verification on the original data and the migration data within the data verification range based on the data type and the migration data verification rule.

[0037] In addition, to achieve the above-mentioned purpose, the present application also proposes a migration data verification device, which includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, and the computer program is configured to implement the steps of the migration data verification method described above.

[0038] In addition, to achieve the above-mentioned purpose, the present application also proposes a storage medium, which is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by the processor, the steps of the migration data verification method described above are implemented.

[0039] In addition, to achieve the above-mentioned purpose, the present application also provides a computer program product, which includes a computer program, and when the computer program is executed by a processor, the steps of the migration data verification method described above are implemented.

[0040] The present application provides a migration data verification method, apparatus, device and storage medium. The migration data verification method obtains a data verification range, a data type and a data association condition corresponding to the original data and the migration data, and then determines the migration data verification rules between the original data and the migration data based on the data association condition, thereby performing migration data verification on the original data and the migration data within the data verification range based on the data type and the migration data verification rule, thereby meeting the personalized data verification requirements during the data migration process, and then verifying the consistency and integrity of the data during the data migration process, thereby effectively improving the accuracy, flexibility and verification efficiency of data verification during the data migration process, and improving user satisfaction. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0042] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0043] Figure 1 A flowchart of the first embodiment of the method for verifying the migration data of the present application is provided;

[0044] Figure 2 A flowchart of the second embodiment of the migration data verification method of this application is provided;

[0045] Figure 3A flowchart of the third embodiment of the migration data verification method of this application is provided;

[0046] Figure 4 A flowchart of the fourth embodiment of the migration data verification method of the present application is provided;

[0047] Figure 5 This is a schematic diagram of the module structure of the migration data verification device according to an embodiment of the present application;

[0048] Figure 6 A schematic diagram of the device structure of the hardware operating environment involved in the migration data verification method in the embodiment of the present application.

[0049] The purpose, features and advantages of this application will be further described in conjunction with the embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0050] It should be understood that the specific embodiments described herein are only used to explain the technical solutions of the present application and are not used to limit the present application.

[0051] In order to better understand the technical solution of the present application, a detailed description will be given below in conjunction with the accompanying drawings and specific implementation methods.

[0052] It should be noted that the execution subject of this embodiment can be a computing service device with data processing, network communication and program running functions, such as a tablet computer, a personal computer, a mobile phone, etc., or an electronic device capable of realizing the above functions, a big data service platform, a migration data verification system, etc. The following takes the migration data verification system as an example to illustrate this embodiment and the following embodiments.

[0053] Based on this, the present application embodiment provides a migration data verification method, referring to Figure 1 , Figure 1 A flowchart of the first embodiment of the migration data verification method of the present application is provided.

[0054] In this embodiment, the migration data verification method includes steps S11 to S13:

[0055] Step S11, obtaining data verification range, data type and data association condition corresponding to the original data and the migrated data;

[0056] It should be noted that the original data refers to the data existing in the original system before migration. These data are the starting point of the migration process and usually include various records and information in the business operation process, such as daily business flow in banking enterprises. The migrated data refers to the data stored in the target system after the data migration process. These data are the result of extracting, converting and loading the original data into the target system.

[0057] It should be further explained that the data verification scope refers to the data range involved in data verification, which can be a specific part of the data set, such as a table, a field, or key information in the data set, so as to achieve custom data range verification, and then allow users to adjust the cycle of the entire migration project by themselves, to prevent the migration verification of the full data from consuming a lot of time, manpower and computing resources.

[0058] Furthermore, the data type refers to the type of data, such as numeric value, text, date, data table and field, etc., so as to ensure that the data is correctly migrated and converted according to the data type. The data association condition refers to the association or dependency relationship between data, so as to ensure the consistency and integrity of data through the data association condition, for example, to ensure the consistency between the order data and customer data migrated from the original system.

[0059] Specifically, the data verification range can be obtained through target user configuration, and the data types and data association conditions corresponding to the original data and the migrated data can be obtained through data detection and other methods. There is no restriction on the acquisition method, which can be set according to actual conditions.

[0060] Step S12, determining a migration data verification rule between the original data and the migration data based on the data association condition;

[0061] It should be noted that the migration data validation rules refer to a series of rules and standards formulated in the process of data migration to ensure the accuracy and integrity of data, and are used to verify whether the migrated data meets the predetermined quality requirements, which include but are not limited to the consistency, accuracy, integrity, validity and reliability of data. The migration data validation rules may include data consistency rules (ensuring that data is consistent between the source system and the target system), data accuracy rules (verifying the accuracy of data), data integrity rules (ensuring that all necessary data is migrated and no data is lost), data validity rules (verifying the validity of data, for example, data in certain fields must meet specific business logic or range restrictions, such as the age field must be greater than 18), data reliability rules (ensuring that the source of data is reliable, for example, the migrated data should come from an authoritative data source, or there is a clear description of the data source), data format rules (verifying whether the format of data meets the standards, business logic rules (ensuring that data follows business rules and processes), data security rules (ensuring that data is not tampered with or leaked during the migration process), data association rules (verifying the relationship between data, for example, whether the association between order records and customer records is correct) and performance rules (ensuring that the process of data migration and verification meets performance requirements, for example, the speed and efficiency of data migration), etc.

[0062] It should be further explained that the presentation form of the migration data validation rule may include SQL (Structured Query Language) and data dictionary (Data Dictionary), etc., which is not limited here and can be set according to actual conditions.

[0063] Specifically, the data migration logic between the original data and the migration data is obtained, and then based on the data migration logic and the data association conditions, the migration data verification rules are obtained. In addition, the data migration logic and the data association conditions are input into the verification rule matching model to obtain the migration data verification rules output by the verification rule matching model, wherein the verification rule matching model is obtained by iterative training of historical data migration logic and historical data association conditions.

[0064] Step S13: performing migration data verification on the original data and the migration data within the data verification range based on the data type and the migration data verification rule.

[0065] Specifically,

[0066] This embodiment obtains the data verification range, data type and data association conditions corresponding to the original data and the migrated data, and then determines the migration data verification rules between the original data and the migrated data based on the data association conditions, so as to perform migration data verification on the original data and the migrated data within the data verification range based on the data type and the migration data verification rules, thereby meeting the personalized data verification needs in the data migration process, lowering the user's usage threshold, and then verifying the consistency and integrity of the data in the data migration process, thereby effectively improving the accuracy, flexibility and verification efficiency of data verification in the data migration process, and improving user satisfaction.

[0067] Based on this, the present application embodiment provides a migration data verification method, referring to Figure 2 , Figure 2 A flowchart of the second embodiment of the migration data verification method of this application is provided.

[0068] In a feasible implementation manner, determining the migration data verification rule between the original data and the migration data based on the data association condition includes:

[0069] Step S21, obtaining data migration logic between the original data and the migration data;

[0070] It should be noted that the data migration logic refers to the rules and steps for converting and migrating data from the original system to the target system during the data migration process. It is used to determine how data is converted from one system format to another system format, how to handle data type differences, and how to establish associations between source data and target data. For example, the data migration logic for table reconstruction includes one-to-one reconstruction, column splitting, row splitting, row splitting + column splitting, etc.; the data migration logic for field reconstruction includes field abandonment, direct field mapping, field type change, field content change, and new field addition, etc.

[0071] Specifically, the data migration logic between the original data and the migrated data may be obtained by configuring according to the target user or reading the historical data migration logic corresponding to the original data and the migrated data. This is not limited here and may be set according to actual conditions.

[0072] Step S22, obtaining migration data verification rules based on the data migration logic and the data association conditions; and / or,

[0073] Specifically, the data migration logic and the data association conditions are matched with the preset verification rule database for similarity to obtain the migration data verification rules. In addition, a secondary check can be performed to determine whether the migration data verification rules comply with the data migration logic and the data association conditions to ensure the accuracy of the migration data verification rules and prevent verification rule errors.

[0074] Step S23, inputting the data migration logic and the data association condition into the verification rule matching model to obtain the migration data verification rule output by the verification rule matching model, wherein the verification rule matching model is obtained by iteratively training the historical data migration logic and the historical data association condition.

[0075] It should be noted that the verification rule matching model is used to output verification rules between the original data that currently needs to be verified for data migration and the migration data. In addition, the generated migration data verification rules can be reused for future data migration tasks, reducing workload and saving computing resources.

[0076] This embodiment obtains the data migration logic between the original data and the migration data, and then obtains the migration data verification rules based on the data migration logic and the data association conditions. In addition, the data migration logic and the data association conditions are input into the verification rule matching model to obtain the migration data verification rules output by the verification rule matching model, wherein the verification rule matching model is obtained by iteratively training the historical data migration logic and the historical data association conditions, thereby ensuring the accuracy and consistency of the data during the verification migration process, and ensuring that the verification data can maintain its internal relationship during the migration process. At the same time, the migration data verification rules can be dynamically adjusted according to different data types and association conditions, so as to adapt to complex data structures, thereby improving the accuracy and efficiency of data verification.

[0077] In a feasible implementation manner, the training process of the verification rule matching model includes:

[0078] Step S31, obtaining historical data migration logic and historical data association conditions;

[0079] It should be noted that the historical data migration logic refers to the data migration logic generated when the data migration verification was performed in the past, and the historical data association condition refers to the data association condition generated when the data migration verification was performed in the past.

[0080] Specifically, the historical data migration logic and the historical data association conditions may be obtained by reading a historical data migration verification database or reading a historical migration verification log. The acquisition method is not limited and may be set according to actual conditions.

[0081] Step S32, based on the preset verification rule database, knowledge injection is performed on the initial verification rule matching model;

[0082] It should be noted that the preset verification rule database refers to a pre-set database storing migration data verification rules, and the preset verification rule database can be continuously adjusted and updated to maintain the accuracy and timeliness of the rules.

[0083] It should be further explained that the initial verification rule matching model refers to an initial template framework that has not been trained, and is used to obtain a verification rule matching model after iterative training.

[0084] Specifically, the preset verification rule database is injected into the structured data knowledge base into the initial verification rule matching model through a supervised learning algorithm, wherein prompt words can be generated based on the knowledge, and prompt words and answer pairs can be designed to be fed into the large model. There is no limitation here, and it can be set according to actual conditions, so as to integrate specific domain knowledge or external knowledge into the pre-trained model, so that the model can achieve excellent training results in specific tasks or fields, thereby enhancing the model's understanding of data migration rule terms, concepts and relationships, thereby improving the generalization ability and robustness of model training.

[0085] Step S33, inputting the historical data migration logic and the historical data association conditions into the initial verification rule matching model after knowledge injection for iterative training to obtain a verification rule matching model.

[0086] Specifically, the historical data migration logic, the historical data association conditions, and the sample labels corresponding to the historical data migration logic and the historical data association conditions are subjected to data preprocessing, such as data cleaning and normalization, so that the historical data migration logic, the historical data association conditions, and the sample labels corresponding to the historical data migration logic and the historical data association conditions are input into the initial verification rule matching model after knowledge injection, and the predicted value output by the initial verification rule matching model after knowledge injection is obtained, and then based on the predicted value and the sample label, the model loss value is calculated using the loss function. In this embodiment, the loss function can be set according to actual needs and is not specifically limited here. After the model loss value is calculated, the training process ends, and the error back propagation algorithm is used to update the model parameters in the initial verification rule matching model after knowledge injection, and then the next training is carried out. During the training process, determine whether the updated initial verification rule matching model after knowledge injection meets the preset training end conditions. If so, the updated initial verification rule matching model after knowledge injection is used as the verification rule matching model. If not, continue to train the model, wherein the preset training end conditions include loss convergence and reaching the maximum number of iterations threshold, etc.

[0087] This embodiment obtains historical data migration logic and historical data association conditions, and then based on a preset verification rule database, performs knowledge injection into the initial verification rule matching model, thereby inputting the historical data migration logic and the historical data association conditions into the initial verification rule matching model after knowledge injection for iterative training to obtain a verification rule matching model, thereby significantly improving the quality, accuracy and efficiency of generated verification rules, and through knowledge injection, the model can learn the characteristics and writing styles of different types of verification rules, so as to better adapt to the specific needs in different scenarios, generate verification rules that better meet user needs, and effectively improve data migration verification efficiency.

[0088] Based on this, the present application embodiment provides a migration data verification method, referring to Figure 3 , Figure 3 A flowchart of the third embodiment of the migration data verification method of this application is provided.

[0089] In a feasible implementation manner, the performing migration data verification on the original data and the migration data within the data verification range based on the data type and the migration data verification rule includes:

[0090] Step S41, based on the data type, dividing the original data and the migration data within the data verification range to obtain several groups of migration verification data;

[0091] It should be noted that the migration verification data refers to a set of original data and migration data of the same data type, so as to perform detailed data migration verification, and enable users to quickly identify and focus on the problem, greatly improving the accuracy and efficiency of problem location.

[0092] Specifically, the data types corresponding to the original data and the migration data within the data verification range are determined, and then the original data and the migration data within the data verification range are divided according to different data types to obtain several groups of migration verification data.

[0093] Step S42: performing data integrity verification and data consistency verification on each migration verification data based on the migration data verification rule;

[0094] Specifically, according to the verification rules for different data types in the migration data verification rules, data integrity verification and data consistency verification are performed on each migration verification data. For example, the data integrity verification and data consistency verification include table-level record number comparison, row-level data value comparison, field statistical value comparison, missing type verification and format specification type verification, etc. Among them, the field statistical value comparison includes vacancy rate comparison, maximum / minimum / average verification, enumeration value verification, etc., which are not limited here and can be set according to actual conditions.

[0095] In one embodiment, the data integrity verification and data consistency verification include format specification type-dictionary value verification, that is, verifying the enumeration value of the data to ensure that the data complies with the predefined dictionary standard.

[0096] Step S43: Based on the migration data verification rule, data relevance verification is performed on the original data and the migration data within the data verification range.

[0097] Specifically, a first data association condition of the original data within the data verification range and a second data association condition of the migration data are determined, and then based on the migration data verification rule, data association verification is performed on the first data association condition and the second data association condition.

[0098] This embodiment divides the original data and the migration data within the data verification scope based on the data type to obtain several groups of migration verification data, and then performs data integrity verification on each migration verification data based on the migration data verification rule, and then performs data consistency verification on each migration verification data based on the migration data verification rule, and then performs data correlation verification on the original data and the migration data within the data verification scope based on the migration data verification rule, thereby achieving refined data migration verification by distinguishing different data types, helping to ensure the correctness of the data migration process, thereby reducing risks after data migration, and avoiding business interruptions or errors due to data migration errors.

[0099] In one possible implementation,

[0100] Step S51, determining a first data association condition of the original data within the data verification range and a second data association condition of the migration data;

[0101] It should be noted that the first data association condition refers to the data association condition of the original data, that is, the original data association condition before data migration, and the second data association condition refers to the data association condition of the migrated data, that is, the data association condition after data migration.

[0102] Specifically, the first data association condition of the original data within the data verification range and the second data association condition of the migration data can be determined by field association information in the database, which is not limited here and can be set according to actual conditions.

[0103] Step S52: Based on the migration data verification rule, data correlation verification is performed on the first data correlation condition and the second data correlation condition.

[0104] Specifically, through the inter-table association class-association verification rule in the migration data verification rule, the first data association condition and the second data association condition are verified for data association, that is, the association relationship between tables is checked to ensure that the data maintains the correct business logic.

[0105] This embodiment determines the first data association condition of the original data within the data verification scope and the second data association condition of the migration data, and then based on the migration data verification rule, performs data association verification on the first data association condition and the second data association condition, thereby ensuring that the data maintains the correct business logic, reducing the risk of migration errors generated after data migration, and thus improving user satisfaction.

[0106] In a feasible implementation manner, after performing migration data verification on the original data and the migration data within the data verification range based on the data type and the migration data verification rule, the method further includes:

[0107] Step S61, obtaining the migration data verification result;

[0108] It should be noted that the migration data verification result refers to the verification result obtained after performing migration data verification on the original data and migration data within the data verification scope, including verification pass results and verification fail results, etc. In one embodiment, when the migration data verification result is a verification pass result, the user can close the verification test and generate a migration data verification report; when the migration data verification result is a verification fail result, the user can choose to modify and re-execute the migration data verification rules, or choose to generate a migration data verification report to read the explanation of the verification failure in the migration data verification report, thereby optimizing the data migration process.

[0109] Specifically, the migration data verification result is obtained through the data consistency verification and data integrity verification process in the data migration verification process, which is not limited here.

[0110] Step S62, performing data analysis on the migration data verification result to generate a migration data verification report;

[0111] It should be noted that the migration data verification report refers to an analysis report generated for the results of the migration data verification. It supports viewing the verification results at the table level and field level, supports viewing the fields with inconsistent table-level comparisons, supports viewing the number of inconsistent records at the field level, supports viewing and downloading comparison SQL, etc. There is no restriction here, so that users can understand the overall situation and final results of the migration data verification.

[0112] Specifically, data integration analysis is performed on the data comparison results, the data comparison rules and the historical comparison records to obtain data analysis results, and then data visualization is performed on the data analysis results to obtain visualization chart information, so as to associate and combine the data analysis results and the visualization chart information to generate a migration data verification report.

[0113] Step S63: Push the migration data verification report to the target user for the target user to view.

[0114] Specifically, the migration data verification report is pushed to the target user for the target user to view. The push can be performed through APP, mini-programs, email, and QR code scanning. In addition, the target user is also supported to share the migration data verification report, such as sharing it with other users through file links or downloaded files, so as to improve the convenience of user sharing.

[0115] This embodiment obtains the migration data verification result, performs data analysis on the migration data verification result, generates a migration data verification report, and pushes the migration data verification report to the target user for the target user to view. The migration data verification report is generated to help discover possible problems in the data migration process, thereby optimizing the migration strategy and process, while ensuring the accuracy, reliability and traceability of data verification, helping users to quickly understand and process verification results, and improving work efficiency.

[0116] Based on this, the present application embodiment provides a migration data verification method, referring to Figure 4 , Figure 4 A flowchart of the fourth embodiment of the migration data verification method of the present application is provided.

[0117] In a feasible implementation manner, the migration data verification result includes a data comparison result, a data comparison rule, and a historical comparison record; and the data analysis of the migration data verification result to generate a migration data verification report includes:

[0118] Step S71, performing data integration analysis on the data comparison result, the data comparison rule and the historical comparison record to obtain a data analysis result;

[0119] It should be noted that the data comparison result refers to the data comparison result between the original data and the migrated data, including the comparison result of the number of records between the new and old tables and the field comparison result of the new and old fields, etc. The data comparison rule refers to the comparison rule used to compare the original data with the migrated data, such as the record number comparison SQL and the field comparison SQL, etc. The historical comparison record refers to the comparison record of the data for previous data migration verification.

[0120] Specifically, the migration data verification result may also include data information such as the number of inconsistent field records, the number of consistent fields in the table, etc., which is not limited here. Therefore, data integration and analysis are performed on the data comparison results, the data comparison rules, the historical comparison records, and the number of inconsistent field records to obtain data analysis results, among which, integration and analysis can be performed through statistical tools, etc., which is not limited here and can be set according to actual conditions.

[0121] Step S72, visualizing the data analysis results to obtain visualized chart information;

[0122] It should be noted that the visual chart information refers to a graphical representation created through data analysis and visualization tools, which is used to reflect the various information and trends shown by data verification analysis within a certain period of time.

[0123] Specifically, the data analysis results are visualized through a preset visualization tool, wherein a suitable visualization tool and visualization chart type can be selected according to the type of data to be displayed and the analysis target, so as to obtain visualization chart information output by the preset visualization tool, wherein the visualization tools include Tableau, Power BI, QlikView, Google Charts, D3.js, Matplotlib (Python), Seaborn (Python) and Plotly and other tools, and the visualization chart types include bar charts, line graphs, pie charts, scatter plots, bubble charts, heat maps, maps, radar charts, tree diagrams, chord diagrams, etc., without limitation here.

[0124] In addition, after the preset visualization tool outputs the visualization chart information, the visualization chart information can be pushed to the target user so that the target user can adjust the visualization chart information, such as selecting colors, setting coordinate axes, adding legends, adjusting layouts, setting interactive elements, etc., and then after determining the final chart, the visualization chart information can be checked to determine whether the chart is clear and easy to understand, whether it can effectively convey information, etc., and at the same time, optimization can be performed based on the verification results, such as adjusting colors, adding labels, modifying axis titles, etc.

[0125] Step S73: Associating and combining the data analysis results and the visual chart information to generate a migration data verification report.

[0126] Specifically, the data analysis results and the visual chart information are associated and combined to generate a migration data verification report, wherein the associated and combined data analysis results and the visual chart information can be adapted to a preset report template to generate a migration data verification report, and the preset report template can be selected by the target user, or the system can automatically adapt the template with the highest matching degree in the template library, which is not limited here and can be set according to actual conditions.

[0127] This embodiment obtains data analysis results by performing data integration analysis on the data comparison results, the data comparison rules and the historical comparison records, and then performs data visualization on the data analysis results to obtain visual chart information, thereby associating and combining the data analysis results and the visual chart information to generate a migration data verification report, thereby ensuring the accuracy, reliability and traceability of data verification, helping users to quickly understand and process verification results, and improving work efficiency.

[0128] It should be understood that the order of execution of the steps in the above embodiment does not necessarily mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiment of the present invention.

[0129] This application also provides a migration data verification device, please refer to Figure 5 , the migration data verification device comprises:

[0130] An acquisition module 51 is used to acquire a data verification range, a data type, and a data association condition corresponding to the original data and the migrated data;

[0131] A determination module 52, configured to determine a migration data verification rule between the original data and the migration data based on the data association condition;

[0132] The verification module 53 is used to perform migration data verification on the original data and the migration data within the data verification range based on the data type and the migration data verification rule.

[0133] The migration data verification device is also used for:

[0134] Acquire data migration logic between the original data and the migrated data;

[0135] Based on the data migration logic and the data association condition, a migration data verification rule is obtained; and / or,

[0136] The data migration logic and the data association conditions are input into a verification rule matching model to obtain a migration data verification rule output by the verification rule matching model, wherein the verification rule matching model is obtained by iteratively training the historical data migration logic and the historical data association conditions.

[0137] The migration data verification device is also used for:

[0138] Obtain historical data migration logic and historical data association conditions;

[0139] Based on the preset verification rule database, knowledge is injected into the initial verification rule matching model;

[0140] The historical data migration logic and the historical data association conditions are input into the initial verification rule matching model after knowledge injection for iterative training to obtain a verification rule matching model.

[0141] The migration data verification device is also used for:

[0142] Based on the data type, data division is performed on the original data and the migration data within the data verification range to obtain several groups of migration verification data;

[0143] Based on the migration data verification rules, performing data integrity verification and data consistency verification on each migration verification data;

[0144] Based on the migration data verification rule, data relevance verification is performed on the original data and the migration data within the data verification range.

[0145] The migration data verification device is also used for:

[0146] Determine a first data association condition of the original data within the data verification range and a second data association condition of the migration data;

[0147] Based on the migration data verification rule, data relevance verification is performed on the first data association condition and the second data association condition.

[0148] The migration data verification device is also used for:

[0149] Get the migration data verification results;

[0150] Performing data analysis on the migration data verification result to generate a migration data verification report;

[0151] The migration data verification report is pushed to the target user for the target user to view.

[0152] The migration data verification device is also used for:

[0153] Performing data integration analysis on the data comparison result, the data comparison rule and the historical comparison record to obtain a data analysis result;

[0154] Performing data visualization on the data analysis results to obtain visualized chart information;

[0155] The data analysis results and the visual chart information are associated and combined to generate a migration data verification report.

[0156] The migration data verification device provided by the present application adopts the migration data verification method in the above embodiment, and can solve the technical problems in the background technology. Compared with the prior art, the beneficial effects of the migration data verification device provided by the present application are the same as the beneficial effects of the migration data verification method provided by the above embodiment, and the other technical features in the migration data verification device are the same as the features disclosed in the above embodiment method, which will not be repeated here.

[0157] The present application provides a migration data verification device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the migration data verification method in the above-mentioned embodiment one.

[0158] Reference below Figure 6 , which shows a schematic diagram of the structure of a migration data verification device suitable for implementing the embodiment of the present application. The migration data verification device in the embodiment of the present application may include but is not limited to mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Descriptions), PMPs (Portable Media Players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Figure 6 The migration data verification device shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0159] like Figure 6As shown, the migration data verification device may include a processing device 1001 (e.g., a central processing unit, a graphics processor, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM: Read Only Memory) 1002 or a program loaded from a storage device 1003 to a random access memory (RAM: Random Access Memory) 1004. In RAM1004, various programs and data required for the operation of the migration data verification device are also stored. The processing device 1001, ROM1002, and RAM1004 are connected to each other through a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Generally, the following systems can be connected to the I / O interface 1006: an input device 1007 including, for example, a touch screen, a touch pad, a keyboard, a mouse, an image sensor, a microphone, an accelerometer, a gyroscope, etc.; an output device 1008 including, for example, a liquid crystal display (LCD: Liquid Crystal Display), a speaker, a vibrator, etc.; a storage device 1003 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 1009. The communication device 1009 can allow the migration data verification device to communicate wirelessly or wired with other devices to exchange data. Although the migration data verification device with various systems is shown in the figure, it should be understood that it is not required to implement or have all the systems shown. More or fewer systems can be implemented or have alternatively.

[0160] In particular, according to the embodiments disclosed in the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present application include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device, or installed from a storage device 1003, or installed from a ROM 1002. When the computer program is executed by the processing device 1001, the above-mentioned functions defined in the method of the embodiment disclosed in the present application are executed.

[0161] The migration data verification device provided by the present application adopts the migration data verification method in the above embodiment, which can solve the technical problems in the background technology. Compared with the prior art, the beneficial effects of the migration data verification device provided by the present application are the same as the beneficial effects of the migration data verification method provided by the above embodiment, and the other technical features in the migration data verification device are the same as the features disclosed in the method of the previous embodiment, which will not be repeated here.

[0162] It should be understood that the various parts disclosed in this application can be implemented by hardware, software, firmware or a combination thereof. In the description of the above embodiments, specific features, structures, materials or characteristics can be combined in any one or more embodiments or examples in a suitable manner.

[0163] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

[0164] The present application provides a computer-readable storage medium having computer-readable program instructions (ie, computer programs) stored thereon, and the computer-readable program instructions are used to execute the migration data verification method in the above-mentioned embodiment.

[0165] The computer-readable storage medium provided in the present application may be, for example, a USB flash drive, but is not limited to electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, systems or devices, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, system or device. The program code contained on the computer-readable storage medium may be transmitted using any appropriate medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination of the above.

[0166] The computer-readable storage medium may be included in the migration data verification device; or may exist independently without being assembled into the migration data verification device.

[0167] The computer-readable storage medium carries one or more programs. When the one or more programs are executed by the migration data verification device, the migration data verification device:

[0168] Obtain data verification scope, data type, and data association conditions corresponding to the original data and the migrated data;

[0169] Determining migration data verification rules between the original data and the migration data based on the data association condition;

[0170] Based on the data type and the migration data verification rule, migration data verification is performed on the original data and the migration data within the data verification range.

[0171] Computer program code for performing the operations of the present application may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0172] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present application. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0173] The modules involved in the embodiments described in this application may be implemented by software or hardware, wherein the name of the module does not constitute a limitation on the unit itself in some cases.

[0174] The readable storage medium provided in the present application is a computer-readable storage medium, which stores computer-readable program instructions (i.e., computer programs) for executing the above-mentioned migration data verification method, and can solve the technical problems in the background technology. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in the present application are the same as the beneficial effects of the migration data verification method provided in the above-mentioned embodiment, and will not be repeated here.

[0175] An embodiment of the present application provides a computer program product, including a computer program, which implements the steps of the migration data verification method as described above when executed by a processor.

[0176] The computer program product provided in this application can solve the technical problems in the background technology. Compared with the prior art, the beneficial effects of the computer program product provided in the embodiment of this application are the same as the beneficial effects of the migration data verification method provided in the above embodiment, which will not be repeated here.

[0177] The above descriptions are only some embodiments of the present application, and are not intended to limit the patent scope of the present application. All equivalent structural changes made using the contents of the present application specification and drawings under the technical concept of the present application, or direct / indirect applications in other related technical fields are included in the patent protection scope of the present application.

Claims

1. A migration data verification method, characterized in that: include: Obtain data verification scope, data type, and data association conditions corresponding to the original data and the migrated data; Determining migration data verification rules between the original data and the migration data based on the data association condition; Based on the data type and the migration data verification rule, migration data verification is performed on the original data and the migration data within the data verification range.

2. The migration data verification method according to claim 1, characterized in that: The determining, based on the data association condition, a migration data verification rule between the original data and the migration data includes: Acquire data migration logic between the original data and the migrated data; Based on the data migration logic and the data association condition, a migration data verification rule is obtained; and / or, The data migration logic and the data association conditions are input into a verification rule matching model to obtain a migration data verification rule output by the verification rule matching model, wherein the verification rule matching model is obtained by iteratively training the historical data migration logic and the historical data association conditions.

3. The migration data verification method according to claim 2, characterized in that: The training process of the verification rule matching model includes: Obtain historical data migration logic and historical data association conditions; Based on the preset verification rule database, knowledge is injected into the initial verification rule matching model; The historical data migration logic and the historical data association conditions are input into the initial verification rule matching model after knowledge injection for iterative training to obtain a verification rule matching model.

4. The migration data verification method according to claim 1, characterized in that: The performing migration data verification on the original data and the migration data within the data verification range based on the data type and the migration data verification rule includes: Based on the data type, data division is performed on the original data and the migration data within the data verification range to obtain several groups of migration verification data; Based on the migration data verification rules, performing data integrity verification and data consistency verification on each migration verification data; Based on the migration data verification rule, data relevance verification is performed on the original data and the migration data within the data verification range.

5. The migration data verification method according to claim 4, characterized in that: The performing data relevance verification on the original data and the migration data within the data verification scope based on the migration data verification rule includes: Determine a first data association condition of the original data within the data verification range and a second data association condition of the migration data; Based on the migration data verification rule, data relevance verification is performed on the first data association condition and the second data association condition.

6. The migration data verification method according to claim 1, characterized in that: After performing migration data verification on the original data and the migration data within the data verification range based on the data type and the migration data verification rule, the method further includes: Get the migration data verification results; Performing data analysis on the migration data verification result to generate a migration data verification report; The migration data verification report is pushed to the target user for the target user to view.

7. The migration data verification method according to claim 6, characterized in that: The migration data verification result includes data comparison results, data comparison rules and historical comparison records; The performing data analysis on the migration data verification result to generate a migration data verification report includes: Performing data integration analysis on the data comparison result, the data comparison rule and the historical comparison record to obtain a data analysis result; Performing data visualization on the data analysis results to obtain visualized chart information; The data analysis results and the visual chart information are associated and combined to generate a migration data verification report.

8. A migration data verification device, characterized in that: include: An acquisition module, used to acquire data verification ranges, data types, and data association conditions corresponding to original data and migrated data; a determination module, configured to determine a migration data verification rule between the original data and the migration data based on the data association condition; The verification module is used to perform migration data verification on the original data and the migration data within the data verification range based on the data type and the migration data verification rule.

9. A migration data verification device, characterized in that: The migration data verification device comprises: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the migration data verification method according to any one of claims 1 to 7.

10. A storage medium, characterized in that: The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, the steps of the migration data verification method according to any one of claims 1 to 7 are implemented.

Images