Artificial intelligence-based data migration static checking method and system, and storage medium

Abstract

The application provides an artificial intelligence-based data migration static checking method and system and a storage medium, relates to the technical field of data processing, and comprises the following steps: obtaining normal data samples from a source system, obtaining to-be-checked migration data from a target system, and respectively performing pretreatment; multi-dimensional features of data records are extracted and converted into feature vectors, and the multi-dimensional features at least include numerical features, category features, date features, text features and cross-table association features; an unsupervised learning algorithm is used to train an anomaly detection model with the normal data samples, so that a normal data mode model is obtained; the to-be-checked data is input into the model to calculate an anomaly score and determine abnormal data; the abnormal data is classified and root cause analysis is performed, and abnormal reasons and repair suggestions are output; a checking report is generated and re-verified after repair. The application realizes high coverage, high efficiency and self-adaptive capacity of data migration checking, and significantly improves the intelligent level and problem positioning accuracy of checking.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and in particular to a method, system, and storage medium for static verification of data migration based on artificial intelligence. Background Technology

[0002] In the financial industry, upgrading core business systems has become commonplace, and data migration is an indispensable and crucial step in the system upgrade process. Taking a bank's credit system as an example, a mature system typically accumulates millions or even tens of millions of customer, account, and transaction records—a vital component of the bank's core assets. The essence of data migration is to accurately, completely, and consistently transfer data from the source system to the target system. Its quality directly determines whether the new system can be launched stably and whether business operations can continue uninterrupted. Quality issues with the migrated data can lead to system errors, business process interruptions, customer complaints, and even regulatory compliance risks. Therefore, after data migration is completed and before the new system goes live, a comprehensive and rigorous static check (i.e., quality inspection of the data itself without running business logic) must be conducted on the migrated data to ensure the accuracy and reliability of the migration.

[0003] Currently, the following two types of technical solutions are mainly used in the static verification stage of data migration: (1) Manual sampling inspection: In this method, inspectors extract a certain percentage (usually 5%~10%) of data samples from the source system and the target system. They then manually compare the data, use Excel to assist in calculations, and check each data item to ensure that the values ​​of key fields are consistent, that required fields are not empty, and that the amount is accurate. Its advantage is that it is highly flexible, and inspectors can dynamically adjust the inspection focus based on experience. However, it is limited by manpower, and the sampling coverage is extremely low. A large amount of unchecked data may contain potential quality problems.

[0004] (2) Automated verification method based on SQL scripts: This method is currently the mainstream practice in the industry. Verifiers write SQL scripts to perform batch verification of the migrated data, which mainly includes: field-level verification (null value, range, and format checks), table-level verification (comparison of total number of records), relationship verification (completeness of master and child table references and consistency of total amount), and business rule verification (such as loan balance not exceeding the credit limit). Compared with manual sampling, the SQL method has significantly improved efficiency and can achieve a certain degree of automated batch verification.

[0005] Although SQL script-based verification methods are widely used in the industry, the inventors discovered the following technical shortcomings during long-term data migration projects: (1) Low audit coverage and high risk of missed checks: SQL auditing is a rule-driven approach, which requires pre-defining rules and writing scripts for each field and relationship. However, medium-sized projects involve dozens of tables and hundreds of fields. A comprehensive audit requires writing hundreds or even thousands of scripts. Due to time constraints and manpower limitations, in actual projects, only core tables and key fields can be covered, with a coverage rate of only 30% to 40%. More importantly, this method can only detect foreseeable quality problems and cannot identify unforeseen or hidden data anomaly patterns, resulting in a serious risk of missed checks.

[0006] (2) Long verification cycle and performance bottlenecks in large-scale scenarios: Writing, debugging and executing a large number of SQL scripts consumes a lot of time. A complete verification usually requires a team of 5 to 8 people to spend 5 to 10 working days. Re-verification after repair will further extend the cycle. Data migration often becomes a bottleneck in project schedule. When the data volume reaches tens of millions or even hundreds of millions, complex multi-table join queries and full table scans will consume a lot of database resources, and the execution time will increase exponentially, making it difficult to complete the full verification within a limited time window.

[0007] (3) Maintenance costs rise sharply when business rules change: Financial business rules change frequently with regulatory policy adjustments and product innovations. Each rule change means that the corresponding verification scripts need to be manually searched and modified one by one. This process is not only time-consuming, but also very easy to introduce new logical errors. The long-term high maintenance costs seriously restrict the sustainability of the verification work.

[0008] (4) Lack of root cause analysis and repair guidance: The existing solution can only output the raw information of "which record, which field does not conform to which rule", and cannot automatically analyze the root cause of the anomaly (such as incorrect mapping rule configuration, ETL logic defects or source system data quality problems), nor can it provide targeted repair suggestions, and deep-seated problems are easily overlooked.

[0009] In addition, existing research has attempted to introduce machine learning into data quality assessment, but classification algorithms require a large number of labeled anomalous samples, clustering algorithms are not effective in high-dimensional mixed-type data, and statistical thresholding methods are only applicable to single-field numerical anomaly detection. All of these are difficult to meet the complex verification needs of data migration, such as multi-field combination, cross-table association, and unknown pattern recognition.

[0010] In summary, existing technical solutions have not yet been able to effectively address the comprehensive needs of static data migration checks, such as high coverage, efficient identification of unknown anomalies, large-scale data processing, adaptive maintenance, and root cause analysis. Summary of the Invention

[0011] To address the problems in the background technology, this invention provides a data migration static inspection method, system, and storage medium based on artificial intelligence. It establishes normal data patterns by training an unsupervised learning model from normal data samples of the source system, then uses this model to calculate and determine anomaly scores for the migrated data of the target system, classifies the anomaly data, performs root cause analysis, and outputs repair suggestions. Finally, it generates an inspection report, which is then re-verified after repair. This provides a high-coverage, high-efficiency, strong-identification, scalable, and adaptive intelligent static inspection solution for large-scale data migration scenarios in the financial industry.

[0012] To achieve the above objectives, this invention provides a static data migration verification method based on artificial intelligence, comprising: Normal data samples are obtained from the source system, and migration data to be checked is obtained from the target system. The normal data samples and the migration data to be checked are preprocessed respectively. The preprocessing includes data cleaning, missing value imputation and data standardization to convert the data into a unified format. For each preprocessed data record, multidimensional features are extracted. These multidimensional features include at least numerical features, categorical features, date features, text features, and cross-table association features. Each feature is then converted into a feature vector. An unsupervised learning algorithm is used to train an anomaly detection model using the feature vectors of the normal data samples as the training set. The model learns the statistical distribution patterns, inter-field relationships, and change patterns of the normal data to obtain a normal data pattern model. The feature vector of the migration data to be checked is input into the normal data pattern model. The degree of deviation of each data record from the normal data pattern is calculated as the anomaly score. Data with an anomaly score exceeding a preset threshold is judged as abnormal data. The abnormal data is classified into at least four categories: missing data, incorrect data, inconsistent data, and violation of business rules. Root cause analysis is performed on each category of abnormality, and the cause of the abnormality and corresponding repair suggestions are output. Generate a check report containing a list of abnormal data, anomaly statistics, root cause analysis results, and remediation suggestions. After repairing the abnormal data according to the remediation suggestions, re-execute the anomaly detection to verify the remediation effect.

[0013] As a further improvement of the present invention, the unsupervised learning algorithm is an autoencoder or an isolated forest.

[0014] As a further improvement of the present invention, the autoencoder includes an encoder and a decoder. The encoder compresses the input high-dimensional feature vector into a low-dimensional hidden layer representation, and the decoder restores the low-dimensional hidden layer representation into a reconstructed feature vector. The model training aims to minimize the reconstruction error of normal data.

[0015] As a further improvement of the present invention, the anomaly score is the reconstruction error between the feature vector of the data record and the reconstructed feature vector output by the normal data pattern model. The larger the reconstruction error, the higher the degree to which the data record deviates from the normal pattern.

[0016] As a further improvement of the present invention, the root cause analysis includes: Data missing root cause analysis is used to identify incomplete mapping rules or defects in ETL programs; Data error root cause analysis is used to identify field type mismatches or inconsistent encoding formats; Data inconsistency root cause analysis is used to identify unmet referential integrity constraints or data discrepancies between the master and child tables; Root cause analysis of business rule violations is used to identify missing or misconfigured business rule validation logic.

[0017] As a further improvement of the present invention, an incremental learning step is also included: when the business rules change, the normal data pattern model is incrementally trained using new labeled normal sample data, so that the model can be updated adaptively.

[0018] This invention also provides an artificial intelligence-based static data migration verification system, which implements the artificial intelligence-based static data migration verification method described above, including: The data acquisition module is used to acquire data samples from the source system and the target system and perform preprocessing. The preprocessing includes data cleaning, missing value imputation and data standardization. The preprocessed data is then output to the feature extraction module. The feature extraction module is used to receive the data output by the data acquisition module, extract multi-dimensional features and convert them into feature vectors, and output them to the model training module and the anomaly detection module respectively. The model training module is used to receive the feature vector of the normal data sample output by the feature extraction module, train the anomaly detection model using an unsupervised learning algorithm to obtain the normal data pattern model, and deploy the model to the anomaly detection module. An anomaly detection module is used to receive the feature vector of the kernel data to be inspected output by the feature extraction module and the normal data pattern model deployed by the model training module, calculate the anomaly score and determine the abnormal data, and output the abnormal data to the root cause analysis module. The root cause analysis module is used to receive the abnormal data output by the anomaly detection module, perform classification and root cause analysis, and output the cause of the anomaly and repair suggestions to the report generation module. The report generation module is used to receive the abnormal causes and repair suggestions output by the root cause analysis module, generate a check report and output it to the repair verification module; The repair verification module is used to receive the inspection report output by the report generation module, repair abnormal data according to the repair suggestions, and re-trigger the anomaly detection module after the repair is completed to verify the repair effect.

[0019] As a further improvement of the present invention, the feature extraction module includes: Numerical feature extraction unit, used to extract statistical features of numerical fields; Categorical feature extraction unit, used for one-hot encoding of categorical fields; The date feature extraction unit is used to extract the time component features of the date field; Text-based feature extraction unit, used to extract NLP semantic features from text fields; The correlation-type feature extraction unit is used to extract correlation features between tables or between master and child tables.

[0020] As a further improvement of the present invention, the anomaly detection module includes: The batch detection unit is used to perform anomaly detection on batch data that has been migrated. A real-time detection unit is used to perform streaming anomaly detection on data during the migration process; Anomaly scoring unit is used to calculate anomaly score for each data item; The anomaly classification unit is used to categorize detection results according to anomaly type.

[0021] The present invention also provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the data migration static verification method based on artificial intelligence as described above.

[0022] Compared with the prior art, the beneficial effects of the present invention are as follows: This invention achieves a paradigm shift in static data migration verification from rule-driven to AI model-driven by constructing a complete technical chain from data preprocessing, multi-dimensional feature extraction, unsupervised learning modeling, anomaly score detection, root cause analysis, to remediation verification. This invention eliminates the need for predefined verification rules; it automatically learns the inherent patterns and correlations of data using only normal data samples. It comprehensively covers various quality problem types, including single-field anomalies, cross-field joint anomalies, and cross-table correlation anomalies, significantly improving verification coverage. Simultaneously, it drastically shortens the verification cycle and significantly improves efficiency, effectively solving the technical challenges of low coverage, low efficiency, and inability to identify unknown anomalies in existing technologies.

[0023] This invention employs unsupervised learning algorithms such as autoencoders or isolated forests to construct an anomaly detection model. This model automatically learns the statistical distribution patterns and inter-field relationships of normal data without relying on manually labeled anomaly samples. By calculating reconstruction error or deviation as an anomaly score, it achieves accurate anomaly identification in the data to be checked. This mechanism not only detects explicit errors such as field values ​​exceeding reasonable ranges, but also identifies hidden problems that existing rule-driven solutions cannot foresee, such as abnormal field combinations and abrupt changes in data distribution, significantly improving the comprehensiveness and accuracy of anomaly detection.

[0024] This invention categorizes abnormal data into multiple dimensions, including missing, erroneous, inconsistent, and business rule violations, and performs root cause analysis for each category. This allows for the automatic identification of the root cause of anomalies, such as incorrect mapping rule configuration, flawed ETL transformation logic, inconsistent master-slave table migration order, or missing business rule validation, and provides targeted remediation suggestions. This mechanism deepens anomaly localization from the superficial level to the root cause level, significantly reducing the technical threshold and time cost of manual investigation, and improving the efficiency and accuracy of anomaly remediation.

[0025] This invention employs an incremental learning mechanism. When business rules change, the existing model can be incrementally trained using newly added normal sample data to achieve adaptive model updates, eliminating the need to search for and modify hundreds or thousands of SQL check scripts as required by existing technologies. This mechanism enables the model to continuously optimize with business development, significantly reducing maintenance costs and substantially improving the reusability and long-term sustainability of the data migration check solution across different business scenarios. Attached Figure Description

[0026] Figure 1 This is a flowchart of a data migration static verification method based on artificial intelligence, disclosed in one embodiment of the present invention. Figure 2 This is a diagram of an AI-based static data migration verification system architecture disclosed in one embodiment of the present invention. Figure 3 This is a schematic diagram illustrating the anomaly detection principle based on artificial intelligence, as disclosed in one embodiment of the present invention. Figure 4 This is a schematic diagram of a data feature extraction process disclosed in an embodiment of the present invention; Figure 5 This is a schematic diagram of the classification criteria and root cause analysis process for anomalies disclosed in one embodiment of the present invention; Figure 6 This is a deployment architecture diagram of an AI-based static data migration verification system disclosed in one embodiment of the present invention; Figure 7This is a comparison diagram showing the effect of data verification between the method of the present invention and a conventional method, as disclosed in one embodiment of the present invention. Detailed Implementation

[0027] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0028] The present invention will now be described in further detail with reference to the accompanying drawings: like Figure 1 As shown, this invention provides a static data migration verification method based on artificial intelligence. By introducing unsupervised learning technology, it can establish a normal data pattern model using only normal data samples, without the need for pre-labeling of abnormal samples. Through a multi-dimensional feature extraction mechanism, it not only captures single-field statistical features but also fully explores cross-field and cross-table correlation features. By combining anomaly detection with root cause analysis, it efficiently discovers various anomalies while automatically locating the causes and providing remediation suggestions. Incremental learning enables adaptive model updates, significantly reducing maintenance costs. Specifically, it includes: S1. Obtain normal data samples from the source system and migration data to be checked from the target system. Perform preprocessing on the normal data samples and migration data to be checked respectively. Preprocessing includes data cleaning, missing value imputation and data standardization to convert the data into a unified format. In one implementation, the normal data sample refers to existing data in the source system that has been business-verified and is running stably, including but not limited to all data in each business table of the source system or representative data selected through sampling. During sampling, it should be ensured that the sample covers various business scenarios and data distribution characteristics. For example, for the customer table, sufficient samples should be available for different customer types, different risk levels, and different age groups. The migration data to be verified is data in the target system that has already undergone migration operations, including all migrated data in each business table of the target system.

[0029] Preprocessing includes data cleaning, missing value imputation, and data standardization, specifically: (1) Data cleaning: This is used to remove duplicate records, obviously erroneous records, and records with abnormal formats from the source and target systems. In one implementation, for data tables with primary key or unique identifier fields, deduplication is performed based on the primary key or unique identifier field, retaining the record with the latest timestamp, and marking the rest as duplicate data and removing them from the check dataset; for data tables without primary keys, deduplication is performed using a combination of all fields as the unique identifier. The identification of obviously erroneous records includes, but is not limited to: records with empty primary keys or unique identifiers, records with key business field values ​​exceeding physical limits (such as negative age or over 150 years old), and records whose formats obviously do not conform to the field definition specifications, etc.

[0030] (2) Missing value imputation: Missing values ​​are imputed only for training data (i.e., normal data samples). Missing values ​​in the migration data to be checked are left as null values ​​so that they can be identified as anomalies in subsequent anomaly detection. For missing value imputation in normal data samples, the specific methods are as follows: numeric fields are imputed with the mean or median of all non-null values ​​in the field; categorical fields are imputed with the "unknown" category or the mode of the field; date fields are imputed with a preset default date or an inferred value determined according to business rules.

[0031] (3) Data standardization: This is used to convert data from different data sources and in different formats into a unified format. In one implementation, data standardization includes, but is not limited to: unifying all date fields to the same format (e.g., "yyyy-MM-dd"); unifying all amount fields to two decimal places (e.g., keeping two decimal places); unifying all character fields to a unified encoding format (e.g., UTF-8); and performing Z-score standardization or Min-Max normalization on all numeric fields to eliminate differences in the units of measurement between different fields.

[0032] In one alternative implementation, for cases where character encoding is inconsistent (e.g., some data is GBK encoded and some is UTF-8 encoded), when standardizing the data, first use a character encoding detection tool to identify the original encoding type of each field, then convert all character fields to UTF-8 encoding, and then perform subsequent feature extraction.

[0033] S2, such as Figure 4 As shown, multidimensional features are extracted from each preprocessed data record. The multidimensional features include at least numerical features, categorical features, date features, text features, and cross-table correlation features, and each feature is converted into a feature vector. Among them, such as Figure 4 As shown, extracting multidimensional features specifically includes: (1) Numerical feature extraction: This is used to extract the original values ​​and statistical characteristics of numerical fields in data records. In a specific implementation, for numerical fields such as age, annual income, loan amount, and loan balance in the customer table, not only are the original values ​​of the fields themselves extracted as features, but also the mean, variance, maximum value, minimum value of each field in the overall dataset, as well as the degree to which the field value of each record deviates from the mean, are calculated. These statistics and deviations are used as features to enhance the model's sensitivity to numerical anomalies.

[0034] (2) Categorical Feature Extraction: This is used to encode and transform categorical fields in data records. In one specific implementation, one-hot encoding is used to convert categorical fields into numerical features. For example, for the "Customer Type" field with values ​​of "Individual," "Enterprise," and "Institution," it is converted into three binary features ("Is it an individual?", "Is it an enterprise?", and "Is it an institution?"), with each record corresponding to three feature values. In another optional implementation, for categorical fields with too many value categories (e.g., more than 50), low-frequency categories are first merged before performing one-hot encoding to avoid excessively high feature dimensions that could reduce model training efficiency.

[0035] (3) Date-type feature extraction: This is used to extract time component features and derived features from date-type fields. In one specific implementation, for date fields such as account opening date, loan disbursement date, and maturity date, time components such as year, month, day of the month, and day of the week are extracted as numerical features; for date fields that appear in pairs (such as loan disbursement date and maturity date), the difference in the number of days between the two dates is calculated as a derived feature (such as loan term). In an optional implementation, business time features such as whether it is the beginning / end of the month, whether it is the beginning / end of the month, and whether it is a holiday can be further extracted.

[0036] (4) Text-based feature extraction: This is used to extract semantic features from text fields. In one specific implementation, for text fields such as customer names and addresses, the TF-IDF method or a pre-trained word vector model (such as Word2Vec, BERT, etc.) is used to convert the text content into fixed-length numerical vectors. For Chinese fields, word segmentation is required before vectorization. In an optional implementation, structured information such as province, city, and district can be further extracted from the address field as supplementary features to improve the model's ability to identify regionally related anomalies.

[0037] (5) Cross-table relationship feature extraction: This is used to extract relationship features between data tables. In a specific implementation, for the relationship between the customer master table and the account details table, the customer ID is used as the relationship key. The account details table is grouped and aggregated by customer ID, and the number of accounts, total account balance, mean account balance, and standard deviation of balance corresponding to each customer ID are counted. These statistics are used as the cross-table relationship features of the customer record. Similarly, for the relationship between the account table and the transaction record table, the account ID is used as the relationship key. The transaction record table is grouped and aggregated by account ID, and the number of transactions, total transaction amount, maximum transaction amount, minimum transaction amount, and standard deviation of transaction amount of each account are counted. These statistics are used as the cross-table relationship features of the account record.

[0038] After extracting the above types of features, all features of the same data record are concatenated into a high-dimensional feature vector and normalized to make the feature values ​​fall within a similar numerical range.

[0039] S3, such as Figure 3 As shown, an unsupervised learning algorithm is used to train an anomaly detection model using the feature vectors of normal data samples as the training set. This allows the model to learn the statistical distribution patterns, inter-field relationships, and change patterns of normal data, thus obtaining a normal data pattern model. In one specific implementation, the unsupervised learning algorithm is an autoencoder. An autoencoder consists of an encoder and a decoder. The encoder progressively compresses the input high-dimensional feature vector into a low-dimensional hidden layer representation; the decoder progressively restores the low-dimensional hidden layer representation to a reconstructed feature vector with the same dimension as the input. The goal of model training is to minimize the reconstruction error, that is, to minimize the difference between the reconstructed feature vector output by the decoder and the original input feature vector. After training, the autoencoder can accurately reconstruct normal data (with a small reconstruction error), but cannot accurately reconstruct abnormal data (with a large reconstruction error).

[0040] The network structure of an autoencoder can be adjusted according to the feature dimension and complexity of the input data. In one specific implementation, the autoencoder uses the following network structure: Input layer (dimension equal to the feature vector dimension) → First fully connected layer (128 neurons, ReLU activation function) → Second fully connected layer (64 neurons, ReLU activation function) → Third fully connected layer (32 neurons, ReLU activation function, as a compressed low-dimensional representation) → Fourth fully connected layer (64 neurons, ReLU activation function) → Fifth fully connected layer (128 neurons, ReLU activation function) → Output layer (dimension equal to the feature vector dimension, Linear activation function). In an alternative implementation, a deeper network structure (e.g., increasing the number of fully connected layers) or a wider hidden layer dimension (e.g., setting the compressed dimension to 64 or 128) can be used to accommodate more complex data distributions.

[0041] In another specific implementation, the unsupervised learning algorithm is Isolation Forest. Isolation Forest constructs multiple isolation trees by randomly selecting features and split values. Normal data points require multiple splits to be isolated (large path depth), while outlier data points are easily and quickly isolated (small path depth). The average path depth of a data point across all isolation trees is calculated as an anomaly score; a smaller path depth indicates a higher probability of an anomaly. Isolation Forest does not require building a neural network model, has low computational complexity, and is suitable for handling ultra-high-dimensional data or scenarios where training time is limited.

[0042] In both of the above implementation methods, model training does not require pre-labeling of abnormal samples; training can be completed using only normal data samples, thus reducing the cost of manual labeling.

[0043] Furthermore, after model training is complete, model performance needs to be evaluated. Specifically, a certain percentage of samples (e.g., 5%, no less than 1000) are extracted from the validation set and manually labeled by data quality experts to determine whether the model's detection results are genuine anomalies and whether any abnormal data has been missed. Based on the manual labeling results, the false positive rate, false negative rate, precision, recall rate, and F1 score are calculated. If all indicators meet the preset acceptance criteria, the model passes the evaluation and can be deployed to the anomaly detection module; if not, the model structure or parameters need to be adjusted and retrained. Simultaneously, multiple candidate thresholds are selected near the anomaly threshold for sensitivity analysis, and a false positive rate-false negative rate curve is plotted. The optimal threshold is selected based on the tolerance for false positives and false negatives in the business scenario. The evaluation process and results are summarized into a model evaluation report, which is saved along with the model for version tracking and quality control.

[0044] S4, such as Figure 3As shown, the feature vector of the migration data to be checked is input into the normal data pattern model, the deviation of each data record from the normal data pattern is calculated as the anomaly score, and data with anomaly scores exceeding a preset threshold are judged as abnormal data. In the implementation using an autoencoder, the anomaly score is calculated as follows: For each data record to be checked, its feature vector is input into the trained autoencoder, the reconstructed feature vector output by the decoder is obtained, and the reconstruction error (such as mean square error) between the original feature vector and the reconstructed feature vector is calculated. This reconstruction error is used as the anomaly score for that data record. The larger the reconstruction error, the greater the degree to which the data deviates from the normal pattern.

[0045] In the implementation using an isolation forest, the outlier score is calculated as follows: for each data record to be checked, its average path depth across all isolation trees is calculated, and this average path depth is converted into an outlier score. The smaller the path depth, the higher the outlier score, indicating that the data is more likely to be isolated, i.e., more likely to be anomalous data.

[0046] Regarding the setting of the preset threshold, in one specific implementation, the quantiles (such as the 95th or 99th quantiles) of the abnormal scores in the validation set data are used as the abnormal threshold. Specifically, the normal data samples in the training set are divided into a training subset and a validation subset in an 8:2 ratio. The model is trained using the training subset, and the model is input using the validation subset to obtain a set of abnormal scores. The specified quantile of this set of abnormal scores is taken as the abnormal threshold. In another optional implementation, the upper whisker (Q3 + 1.5 × IQR, where Q3 is the upper quartile and IQR is the interquartile range) of the box plot method can also be used as the abnormal threshold. The choice of threshold can be adjusted according to the actual business requirements for false positive and false negative rates: if the business has a low tolerance for false negatives, the threshold can be appropriately lowered to detect more potential anomalies; if the business has a low tolerance for false positives, the threshold can be appropriately increased to reduce the workload of manual review.

[0047] S5, such as Figure 5 As shown, abnormal data is classified into at least four categories: missing data, incorrect data, inconsistent data, and violation of business rules. Root cause analysis is performed for each category, and the cause of the abnormality and corresponding remediation suggestions are output. In one specific implementation, anomaly classification and root cause analysis are achieved in the following way: (1) Data Missing Anomalies: Identify abnormal data where key fields are null or the entire record is missing. Root cause analysis includes: First, check if the corresponding field has a value in the source system. If the source system has a value but the target system is empty, the root cause is determined to be incomplete mapping rules or ETL program defects; if the source system is also empty, the root cause is determined to be missing data in the source system itself. The corresponding remediation suggestions are: For mapping rule issues, supplement or correct the mapping rules and then re-migrate; for source system data missing issues, obtain the missing data from the business department and then supplement and migrate.

[0048] (2) Data Error Anomalies: Identify abnormal data where field values ​​are outside the reasonable range, data types are mismatched, or formats are incorrect. Root cause analysis includes: comparing the value formats and ranges of corresponding fields in the source and target systems to determine if the issue is due to field type mismatch, inconsistent encoding formats, or incorrect unit conversion. The corresponding remediation recommendations are: adjust ETL conversion rules, correct erroneous fields in the migrated data, and apply the correct conversion logic in subsequent migrations.

[0049] (3) Data Inconsistency Anomalies: Identify abnormal data where the master and child tables are inconsistent or cross-table referential integrity is violated. Root cause analysis includes: checking the migration order and timestamps of the master and detail tables to determine whether the data inconsistency between the master and child tables is caused by an improper migration order or a failure to meet referential integrity constraints. The corresponding remediation suggestions are: adjust the migration order (migrate the detail table first and then the master table, or use a transactional snapshot migration method) to ensure the consistency of the master and child table data.

[0050] (4) Business rule violation anomalies: Identify data that violates established business rules, such as loan balance exceeding credit limit. Root cause analysis includes: checking whether the corresponding data in the source system also violates business rules. If the source system itself has this problem, it is determined to be a data quality issue in the source system; if the source system complies with the rules but the target system violates the rules, it is determined to be insufficient business rule verification during the migration process. The corresponding remediation suggestions are: for source system data quality issues, submit a processing plan to the business department for confirmation; for insufficient migration verification issues, add a pre-processing business rule verification node in the ETL process.

[0051] The above classification and root cause analysis can be performed automatically by a preset rule engine, or semi-automatically by manual investigation and labeling combined with the rule engine.

[0052] S6. Generate a check report containing a list of abnormal data, anomaly statistics, root cause analysis results, and repair suggestions. After repairing the abnormal data according to the repair suggestions, re-execute the anomaly detection to verify the repair effect.

[0053] In one specific implementation, the inspection report includes, but is not limited to, the following: (1) Abnormal Data List: Displays the identification information (such as record ID or primary key value), the data table to which all abnormal records belong, the abnormal type, the abnormal description, and the abnormal score in the form of a list or table. The abnormal description specifically explains the problem, such as "the customer type field is null", "the interest rate field value is 500%, which is outside the reasonable range", "the balance in the main table is inconsistent with the summary balance in the detailed table (the difference is X yuan)", "the loan balance of Y yuan is greater than the credit limit of Z yuan", etc.

[0054] (2) Anomaly Statistics: Display the number, percentage, and distribution of various anomalies in different data tables in the form of charts. This includes, but is not limited to, bar charts or pie charts of anomaly type distribution, scatter plots or box plots of anomaly score distribution, and distribution charts of anomaly records in various data tables.

[0055] (3) Root cause analysis results: Summarize the root cause analysis conclusions of various anomalies in text form, including the root cause of the anomaly (such as incorrect mapping rule configuration, ETL transformation logic defects, improper migration order, source system data quality problems, etc.) and its scope of impact (the tables, fields and business impact involved).

[0056] (4) Repair suggestions: Provide specific and actionable repair steps for each type of anomaly or each anomaly record.

[0057] In one specific implementation, the remediation verification process is as follows: The data migration team remediates the abnormal data according to the remediation suggestions in the verification report. After the remediation is completed, the anomaly detection in step S4 is re-executed, and the number and type of abnormal records before and after the remediation are compared to calculate the anomaly remediation rate (remediation rate = (number of anomalies before remediation - number of anomalies after remediation) / number of anomalies before remediation). If the number of abnormal records after remediation is reduced to an acceptable level (e.g., the remediation rate exceeds 95%), and the remaining anomalies are confirmed by the business department to not affect system operation, then the verification is passed; otherwise, the above remediation and verification process is repeated until the number of anomalies reaches an acceptable level.

[0058] In one alternative implementation, data records that have been repaired and verified are marked as "verified normal" and can be used as new normal data samples for incremental training of the model to continuously improve the model's recognition accuracy.

[0059] S7. When business rules change, use new labeled normal sample data to incrementally train the normal data pattern model, so that the model can be updated adaptively.

[0060] In one specific implementation, incremental learning is carried out as follows: When business rules change (such as regulatory policy adjustments leading to changes in credit limit calculation rules, or the launch of new products resulting in new data field values), a batch of normal data samples that have been confirmed and verified by the business department and conform to the new business rules are collected. This batch of new sample data is preprocessed and feature extracted according to steps S1 and S2 to obtain new feature vectors. Then, based on the original model parameters, training continues using the new feature vectors as the training set (i.e., further iterative optimization using new data based on the already trained model weights), rather than retraining from scratch. The learning rate for incremental training is typically set to 1 / 10 to 1 / 5 of the original training learning rate to avoid drastic perturbations of the learned normal patterns by the new data.

[0061] In another specific implementation, if the business rules change significantly (such as adding or deleting data fields, or greatly adjusting the range of field values), causing the original model structure to no longer be applicable, a full retraining approach can be adopted, that is, merging the original normal data and the newly added normal data, and re-executing the model training process in step S3.

[0062] After incremental learning is complete, the system automatically evaluates the performance of the updated model, comparing the new model's anomaly detection accuracy, recall, and other metrics on the validation set with the old model. If the new model outperforms the old model, it is automatically deployed to the anomaly detection module; if the new model's performance deteriorates instead of improving, the old model is retained and an alert is issued to notify technical personnel to investigate.

[0063] In one specific implementation, the incremental learning step also includes model version management: after each incremental learning iteration, a new version of the model is generated, recording information such as the model version number, training time, range of training data used, and changes to business rules. When a new model experiences performance issues, it supports one-click rollback to any historical version.

[0064] like Figure 2 As shown, an artificial intelligence-based data migration static verification system includes: The data acquisition module is used to acquire data samples from the source system and the target system and perform preprocessing. Preprocessing includes data cleaning, missing value imputation and data standardization. The preprocessed data is then output to the feature extraction module. In one embodiment, the data acquisition module connects to the source and target databases via database connection methods such as JDBC (Java Database Connectivity) or ODBC (Open Database Connectivity), and executes SQL queries to retrieve data from specified tables. For tables with large amounts of data (e.g., more than 10 million records), paginated queries or batch extraction can be used to avoid memory overflow caused by loading all data at once. The specific implementation of preprocessing is consistent with the description in step S1 of the above method implementation, and will not be repeated here.

[0065] The feature extraction module is used to receive data output from the data acquisition module, extract multi-dimensional features and convert them into feature vectors, which are then output to the model training module and the anomaly detection module, respectively. In one specific implementation, the feature extraction module includes multiple feature extraction sub-units, corresponding to the extraction of numerical, categorical, date, textual, and relational features, respectively. Each sub-unit processes data records in parallel, and after extraction, the various features are concatenated into a unified feature vector. The feature extraction module also includes a standardization unit, used to perform Z-score standardization or Min-Max normalization on the concatenated feature vector. The specific implementation of feature extraction is consistent with the description in step S2 of the above method implementation, and will not be repeated here.

[0066] The model training module receives the feature vectors of normal data samples output by the feature extraction module, trains the anomaly detection model using an unsupervised learning algorithm, obtains the normal data pattern model, and deploys the model to the anomaly detection module. In one specific implementation, the model training module supports both autoencoder and isolated forest algorithms, allowing users to choose one based on data scale, feature dimensions, and business needs. After model training is complete, the module saves the trained model parameters as files (such as HDF5 or Pickle format) and loads them into the memory of the anomaly detection module for subsequent anomaly detection. The model training module also supports model version management; each time a new model version is generated, historical versions are retained for rollback in case of performance degradation.

[0067] The anomaly detection module receives the feature vector of the data to be inspected from the feature extraction module and the normal data pattern model deployed by the model training module, calculates the anomaly score and identifies the abnormal data, and outputs the abnormal data to the root cause analysis module. In one specific implementation, the anomaly detection module includes a batch detection unit and a real-time detection unit. The batch detection unit performs a one-time full anomaly detection on the migrated batch data, suitable for static verification scenarios after data migration. The real-time detection unit performs streaming anomaly detection on the data during the migration process, suitable for scenarios where online monitoring of migrated data quality is required. The anomaly detection module also includes an anomaly scoring unit and an anomaly classification unit. The anomaly scoring unit calculates an anomaly score for each data point based on the model type, and the anomaly classification unit marks records with anomaly scores exceeding a threshold as anomaly data and performs preliminary classification according to the anomaly type. The specific implementation of anomaly detection is consistent with the description in step S4 of the above method embodiment, and will not be repeated here.

[0068] The root cause analysis module receives abnormal data output by the anomaly detection module, performs classification and root cause analysis, and outputs the cause of the anomaly and repair suggestions to the report generation module. In one specific implementation, the root cause analysis module includes a rule engine and a knowledge base. The rule engine has pre-set judgment logic for various anomalies, and the knowledge base stores root cause templates and remediation suggestion templates for common anomaly types. After receiving anomaly data, the root cause analysis module first calls the rule engine to determine the anomaly type, and then matches the corresponding root cause template and remediation suggestion template in the knowledge base based on the anomaly type and specific field information, generating personalized root cause analysis and remediation suggestions for the anomaly record. The specific implementation of root cause analysis is consistent with the description in step S5 of the above method embodiment, and will not be repeated here.

[0069] The report generation module receives the abnormal causes and remediation suggestions output by the root cause analysis module, generates a check report, and outputs it to the remediation verification module. In one specific implementation, the report generation module supports output in multiple report formats, including but not limited to: PDF format (suitable for archiving and printing), HTML format (suitable for viewing and interacting in a browser), Excel format (suitable for data export and secondary analysis), and JSON format (suitable for data exchange between systems). The report generation module also includes a visualization unit for automatically generating visual charts such as anomaly type distribution maps, anomaly score scatter plots, and anomaly trend charts based on the abnormal data, and embedding them in the report. The specific implementation of report generation is consistent with the description in step S6 of the above method embodiment, and will not be repeated here.

[0070] The repair verification module receives the inspection report output by the report generation module, repairs abnormal data according to the repair suggestions, and re-triggers the anomaly detection module after the repair is completed to verify the repair effect.

[0071] In one specific implementation, the repair verification module provides repair task management functions, including the creation, allocation, tracking, and closure of repair tasks. The repair verification module automatically generates a repair task sheet based on the verification report and assigns it to the corresponding data migration engineer or developer. After repair is completed, the repair verification module automatically re-triggers the anomaly detection module to re-verify the repaired data and generates a repair verification report, comparing the changes in abnormal data before and after repair. If the repair verification passes, the repair verification module marks the repair task as "completed"; if the repair verification fails, a new repair task is generated for a second repair. The specific implementation of repair verification is consistent with the description in step S6 of the above method embodiment and will not be repeated here.

[0072] The present invention also provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the above-described artificial intelligence-based static data migration verification method.

[0073] In one specific implementation, the computer-readable storage medium can be any tangible medium that contains or stores a program, including but not limited to: USB flash drive, portable hard drive, read-only memory (ROM), random access memory (RAM), disk, optical disk, server storage space, etc. When the computer program is executed by the processor, it realizes a complete process from data acquisition and preprocessing, feature extraction, AI pattern learning, anomaly detection, anomaly classification and root cause analysis to report generation and repair verification.

[0074] In one specific implementation, the computer program can be deployed in a client-server architecture, where the data acquisition module, feature extraction module, and anomaly detection module are deployed on the server side close to the database to reduce data transmission overhead; the report generation module and repair verification module are deployed on the web server side, and users access the inspection report and perform repair tasks through a browser.

[0075] This invention is applicable to, but not limited to, scenarios such as upgrading core banking systems, integrating insurance business systems, collecting government data, replacing enterprise ERP systems, and replacing databases with domestic alternatives. In actual deployment, it can be implemented according to… Figure 6 With the architecture shown, in all the above application scenarios, the technical solution of this invention can achieve high coverage (over 85%) and high efficiency (20 times higher efficiency) static data migration checks, and has the ability to identify unknown anomalies, adapt to changes in business rules, and provide root cause analysis and remediation suggestions. Example 1:

[0076] This embodiment uses the scenario of a bank migrating from an old credit system to a new credit system to provide a detailed description of the method of the present invention. The data scale involved in this embodiment is: 1 million customer records, 5 million account records, and 20 million transaction records.

[0077] Step 1: Data Acquisition and Preprocessing Obtain normal data samples from the old credit system (source system), including existing data from the customer table, account table, and transaction table that has been verified by business operations. Obtain the corresponding data that has been migrated from the new credit system (target system).

[0078] The following preprocessing steps were performed: Duplicate records were removed using primary key deduplication; missing values ​​in normal data samples were imputed (numerical fields were filled with the mean, and categorical fields were filled with "unknown"); all dates were standardized to "yyyy-MM-dd" format, all monetary fields were rounded to two decimal places, all character fields were standardized to UTF-8 encoding, and all numeric fields were Z-score standardized to eliminate dimensional differences. Missing values ​​in the migration data to be checked were left as null values ​​and were not imputed.

[0079] Step 2, Feature Extraction Multidimensional features are extracted from each preprocessed data record and converted into a feature vector.

[0080] For customer records: extract numerical features such as age, annual income, loan amount, and loan balance, as well as their respective statistical features; perform one-hot encoding on categorical fields such as customer type (individual / enterprise / institution), occupation, industry, and risk level; extract time components such as year, month, and weekday, and derived features such as loan term (difference between the maturity date and the disbursement date) from account opening date, loan disbursement date, and maturity date; extract 50-dimensional text features from text fields such as customer name and address using TF-IDF; when extracting cross-table relational features, group and aggregate in the account details table with customer ID as the relational key, and count the number of accounts, total balance, mean balance, and standard deviation of balance for each customer, and use these statistics as the relational features of that customer record.

[0081] Similarly, for account records and transaction records, extract the corresponding multidimensional features.

[0082] After all features are concatenated into a high-dimensional feature vector, normalization is performed.

[0083] Step 3, AI mode learning An autoencoder was employed as the unsupervised learning algorithm. The autoencoder network structure was as follows: input layer (dimension equal to feature vector dimension) → fully connected layer (128 neurons, ReLU activation) → fully connected layer (64 neurons, ReLU activation) → fully connected layer (32 neurons, ReLU activation, compression layer) → fully connected layer (64 neurons, ReLU activation) → fully connected layer (128 neurons, ReLU activation) → output layer (dimension equal to feature vector dimension, linear activation). The Adam optimizer was used with a learning rate of 0.001, a batch size of 32, and 100 training epochs. The training and validation sets were partitioned in an 8:2 ratio. The training objective was to minimize the reconstruction error, and an early stopping strategy was employed to prevent overfitting.

[0084] The model is trained using the feature vectors of normal data samples (1 million customer records, 5 million account records, and 20 million transaction records verified in the source system) as the training set. This allows the model to learn the statistical distribution patterns, inter-field relationships, and change patterns of normal data, thus obtaining a normal data pattern model.

[0085] After model training, performance evaluation is performed: 5% of the samples (no less than 1000) are extracted from the validation set, manually labeled by data quality experts, and the false positive rate, false negative rate, precision, recall, and F1 score are calculated. In this embodiment, the F1 score is 0.87, meeting the preset acceptance standard (≥0.85). Sensitivity analysis is also performed within the 90th to 99th percentile range of the anomaly score threshold, and the false positive rate-false negative rate curve is plotted. Considering the low tolerance for false negatives in bank lending scenarios, the 95th percentile with the lower false negative rate is selected as the anomaly threshold.

[0086] Step 4, Anomaly Detection The data to be migrated from the new system (1 million customer records, 5 million account records, and 20 million transaction records) is input into the trained autoencoder model. The reconstruction error of each record is calculated as an anomaly score. Data with an anomaly score exceeding the 95th percentile threshold is judged as anomalous data.

[0087] In this embodiment, approximately 3800 abnormal records were detected, with an anomaly detection rate of 3.8%. Some examples of anomalies are as follows:

[0088] Step 5: Anomaly Classification and Root Cause Analysis Classify and perform root cause analysis on the detected abnormal data: (1) Data missing anomalies (approximately 1000 records): The "Customer Type" field in the customer records is empty. After comparison with the corresponding records in the source system, it was determined that this was caused by the mapping rules not being configured. Repair suggestion: Add the mapping rules and then re-migrate.

[0089] (2) Data error exceptions (approximately 500): Interest rate field values ​​are outside the reasonable range (>100%). Comparison with the source system shows the value is in percentage format, indicating that unit conversion was not performed during ETL conversion. Recommendation: Adjust the ETL conversion rules to divide the percentage value by 100 before writing.

[0090] (3) Data inconsistency anomalies (approximately 2000): The balance in the customer's master table is inconsistent with the summary balance in the account details table. Investigation revealed that the migration order of the master table and details table was incorrect (the master table was migrated first, followed by the details table), resulting in incomplete details data during the summary process. Recommendation: Adjust the migration order to ensure the details table is migrated first.

[0091] (4) Business rule violation anomalies (approximately 300): Loan balance exceeds credit limit. Comparison with the source system confirmed that the source system itself had this data quality issue, and the migration process lacked business rule verification. Recommendation: Submit a solution to the business department for confirmation, and add pre-processing verification of business rules to the ETL process.

[0092] Step 6: Report Generation and Repair Verification Generate an inspection report, including a list of abnormal data (record ID, table name, abnormality type, abnormality description, and abnormality score), abnormality statistics (data missing 26.3%, data error 13.2%, data inconsistency 52.6%, and business rule violation 7.9%, presented as a bar chart), root cause analysis results, and remediation suggestions. Push the report to the data migration team and the technical team.

[0093] After the data migration team completed the repairs based on the recommendations, they re-executed the anomaly detection, reducing the number of anomaly records from 3,800 to 50, achieving an anomaly repair rate of 98.7%. The remaining 50 records were confirmed to be historical data legacy issues from the source system itself. After confirmation by the business department, they were considered acceptable anomalies, passed the inspection, and the new system went live on schedule.

[0094] like Figure 7 As shown, compared with the traditional method, this embodiment has a check coverage rate of over 85%, which is much higher than the 30%-40% of the traditional method; the check time is only 0.5 people × 1 day, while the traditional method requires 2 people × 5 days, which is 20 times more efficient; 4 types of 3800 abnormal issues were found, including hidden cross-table association issues such as inconsistent data between master and child tables, while the traditional method can usually only find 1-2 obvious issues.

[0095] Example 2: This embodiment details the software implementation of autoencoder pattern learning. It is implemented using the Python language and the TensorFlow framework.

[0096] Step 1: Feature Extraction and Standardization Python import pandas as pd import numpy as np from sklearn.preprocessing import StandardScaler #Read data data = pd.read_csv('customer_data.csv') #Numerical Feature Extraction numeric_features = ['age', 'income', 'loan_amount', 'loan_balance'] for col in numeric_features: data[f'{col}_zscore'] = (data[col]- data[col].mean()) / data[col].std() #Categorical Feature Extraction (One-Hot Encoding) categorical_features = ['customer_type', 'occupation', 'industry', 'risk_level'] data = pd.get_dummies(data, columns=categorical_features) #Date-type feature extraction date_features = ['open_date', 'loan_date', 'due_date'] for col in date_features: data[col + '_year'] = pd.to_datetime(data[col]).dt.year data[col + '_month'] = pd.to_datetime(data[col]).dt.month data[col + '_day'] = pd.to_datetime(data[col]).dt.day #Cross-table association feature extraction account_summary = account_data.groupby('customer_id').agg({ 'account_balance': ['count', 'sum', 'mean', 'std'] }).reset_index() account_summary.columns = ['customer_id', 'account_count', 'balance_sum', 'balance_mean', 'balance_std'] data = data.merge(account_summary, on='customer_id', how='left').fillna(0) # Feature concatenation and standardization feature_cols = [col for col in data.columns if col not in ['customer_id']] features = data[feature_cols].values scaler = StandardScaler() features_scaled = scaler.fit_transform(features) Step 2, Construction and training of autoencoder model python import tensorflow as tf from tensorflow.keras import layers, Model, callbacks input_dim = features_scaled.shape[1] # Encoder input_layer = layers.Input(shape=(input_dim,)) encoded = layers.Dense(128, activation='relu')(input_layer) encoded = layers.Dense(64, activation='relu')(encoded) encoded = layers.Dense(32, activation='relu')(encoded) # Compression layer # Decoder decoded = layers.Dense(64, activation='relu')(encoded) decoded = layers.Dense(128, activation='relu')(decoded) decoded = layers.Dense(input_dim, activation='linear')(decoded) autoencoder = Model(input_layer, decoded) autoencoder.compile(optimizer=tf.keras.optimizers.Adam(learning_rate=0.001), loss='mse') # Training split_idx = int(0.8 * len(features_scaled)) train_data, val_data = features_scaled[:split_idx], features_scaled[split_idx:] autoencoder.fit(train_data, train_data, epochs=100, batch_size=32, validation_data=(val_data, val_data), callbacks=[callbacks.EarlyStopping(patience=10,restore_best_weights=True)]) Step 3, Anomaly detection and classification python # Calculate anomaly score reconstructed = autoencoder.predict(test_features_scaled) reconstruction_errors = np.mean(np.power(test_features_scaled -reconstructed, 2), axis=1) #Threshold Setting and Anomaly Detection val_reconstructed = autoencoder.predict(val_data) val_errors = np.mean(np.power(val_data - val_reconstructed, 2), axis=1) threshold = np.percentile(val_errors, 95) anomaly_indices = np.where(reconstruction_errors>threshold)[0] #Anomaly Classification and Root Cause Analysis (Taking Missing Data as an Example) for idx in anomaly_indices: record = test_data.iloc[idx] # Check if the key field is empty if pd.isna(record.get('customer_type')): # Query the corresponding record in the source system src_record = source_data[source_data['customer_id'] == record['customer_id']] if not src_record.empty and pd.notna(src_record.iloc[0].get('customer_type')): print(f"Record {record['customer_id']}: Data missing class, root cause is incomplete mapping rule") Example 3: Six months after a bank launched its new lending system, regulatory authorities issued new credit management rules requiring stricter credit limit controls on loans to specific industries, and changing the compliance criteria for loan balances and credit limits. Simultaneously, the new system added a "Loan Purpose" field, increasing the dimensions of data features. Based on these changes, the existing anomaly detection model needed to be updated due to the changes in business rules. The specific implementation steps are as follows: Step 1: New Sample Collection and Preprocessing: Collect 50,000 normal data samples that have been confirmed as compliant by the business department within one month after the new rules take effect. Then, fill in missing values ​​and standardize the newly added "Loan Purpose" field in accordance with the method of Step (I) of Example 1.

[0097] Step 2, Feature Dimension Alignment: Due to the addition of the "Loan Purpose" field, the feature dimension is expanded from the original dimension to the original dimension + 1. For the weights of the corresponding dimension of this field that do not exist in the original model parameters, random initialization is used.

[0098] Step 3, Incremental Training: Continue training on the existing model with a learning rate of 0.0001 (1 / 10 of the original training learning rate), iterating for 20 rounds. The learning rate is set to 1 / 10 of the original training learning rate to avoid drastic perturbations of the learned normal patterns by new data. After training, compare the F1 score of the new model (0.89) on the current validation set with that of the old model (0.87). The new model outperforms the old model, and the new model is automatically deployed to the anomaly detection module.

[0099] In another alternative implementation, if the changes in business rules are so significant that the original model structure is no longer applicable (e.g., the number of new fields exceeds 30% of the original feature dimensions), then the original normal data and the newly added normal data are merged and the entire model is retrained, rather than performing incremental training.

[0100] Step 4, Version Management: This incremental learning generates a v2.0 version model, recording the version number, training time, range of newly added data, and details of business rule changes. The original v1.0 version is retained; if performance issues arise with the new model, a one-click rollback to v1.0 is supported. At least the most recent three versions should be retained; earlier versions can be archived or deleted according to storage policies.

[0101] Example 4: In step 6 of Example 1, the report generation module generates an inspection report containing visual charts, specifically including: Step 6.1: Generation of the anomaly type distribution map Based on the anomaly classification results, generate a bar chart showing the distribution of anomaly types. The horizontal axis represents the anomaly type (missing data, incorrect data, inconsistent data, and business rule violation), and the vertical axis represents the number of anomalies. The data for each type is 1000, 500, 2000, and 300 records respectively. Specific values ​​are labeled on the bars, and the legend displays the percentage of each type (26.3%, 13.2%, 52.6%, 7.9%).

[0102] Step 6.2: Generation of the scatter plot of outlier score distribution A scatter plot of anomaly scores is generated with the anomaly record number on the horizontal axis and the anomaly score on the vertical axis. Different colors are used to identify different anomaly types (red for missing data, orange for data errors, blue for data inconsistencies, and green for violations of business rules), and an anomaly threshold line (red dashed line, corresponding to the 95th percentile) is marked to visually show the distribution of each anomaly record relative to the threshold.

[0103] Step 6.3: Generation of the Anomaly Impact Analysis Diagram An anomaly impact distribution chart is generated with the affected table type (customer table, account table, transaction table) as the horizontal axis and the number of affected records as the vertical axis. In this example, approximately 1800 records from the customer table, approximately 1200 records from the account table, and approximately 800 records from the transaction table are affected.

[0104] Step 6.4: Integration and Output of Inspection Reports The above charts and the list of abnormal data (including record ID, abnormality type, abnormality description, abnormality score, root cause analysis results, and remediation recommendations) are integrated into a complete HTML format audit report. The report includes the following sections: Audit Summary (total number of records, number of abnormal records, abnormality detection rate, audit time), abnormality statistical charts, abnormal data list, root cause analysis summary, and remediation recommendation summary. Users can view the report online through a browser or export it as a PDF file with one click.

[0105] In this embodiment, the inspection report takes about 5 seconds to generate and can be pushed to the data migration team (including the complete list of anomalies and remediation suggestions), the technical team (including root cause analysis and ETL remediation solutions), and the business team (only including business rule violation anomalies and their handling suggestions) at the same time, so as to realize differentiated push according to role.

[0106] Example 5: In Example 1, an autoencoder was used as an unsupervised learning algorithm. This example provides another optional implementation scheme, which uses an isolation forest instead of an autoencoder for anomaly detection.

[0107] In heterogeneous database data migration and verification scenarios, when the number of data tables is large (e.g., more than 100 tables) and the data volume of a single table is large (e.g., more than 50 million records), using an autoencoder for record-by-record feature extraction and reconstruction may face high computational resource consumption and time costs. Isolation Forest, as an unsupervised learning algorithm based on decision tree ensembles, does not require building a neural network model, and its computational complexity is O(n×t×ψ), where n is the number of samples, t is the number of trees, and ψ is the subsampling size. This is typically much lower than the training and inference costs of an autoencoder. This embodiment provides a lightweight alternative implementation scheme for the above scenario, including: Step 1, Feature Extraction: Similar to Step 2 in Example 1, extract multi-dimensional feature vectors for each data record and normalize them. In this example, to improve the detection efficiency of isolated forests in high-dimensional feature spaces, a feature filtering step is added after feature extraction: calculate the correlation between each feature and the target business field (such as loan balance, credit limit, etc.), retain features with an absolute correlation coefficient greater than 0.1, remove irrelevant features, and reduce the feature dimension from 300 dimensions to 150 dimensions.

[0108] Step 2, Model Training: The Isolation Forest algorithm is used, with the feature vectors of normal data samples as the training set. The number of isolation trees is set to 200, and the subsampling size is 256 records. For each isolation tree, the data is recursively partitioned by randomly selecting a feature and a split value between the minimum and maximum values ​​of that feature, until each leaf node contains only one record or the preset tree height limit is reached. After training, an Isolation Forest model containing 200 isolation trees is obtained.

[0109] Step 3, Anomaly Detection: Input the kernel migration data to be checked into the trained Isolation Forest model. For each data record, calculate its average path depth (i.e., the number of edges traversed from the root node to the leaf node) in the 200 isolated trees, and convert it into an anomaly score. The anomaly score ranges from 0 to 1; a higher score indicates that the data is more likely to be isolated (shorter path depth), i.e., a higher probability of an anomaly. Similar to Example 1, the 95th percentile of the validation set anomaly scores is used as the anomaly threshold to determine abnormal data.

[0110] According to the test, under the same data scale, the model training time in this embodiment is about 5 minutes (about 50 minutes in the autoencoder in embodiment 1), the anomaly detection time is about 8 minutes (about 30 minutes in the autoencoder in embodiment 1), the anomaly detection rate is about 3.5% (about 3.8% in the autoencoder), and the F1 value is 0.84, which is slightly lower than the autoencoder's 0.87 but meets the acceptance criteria for most actual business scenarios (usually requiring ≥0.80).

[0111] This embodiment is applicable to data migration verification scenarios that require moderate detection accuracy but high training and detection efficiency, and can be used as a substitute or supplement to the technical solution of Embodiment 1.

[0112] Advantages of this invention: This invention employs an unsupervised learning algorithm to train an anomaly detection model using only normal data samples. Without pre-defining check rules or labeling abnormal samples, it can automatically learn the statistical distribution patterns, inter-field relationships, and change patterns of normal data. It can identify hidden problems that traditional rule-driven solutions cannot foresee, such as abnormal field combinations and abrupt changes in data distribution. The check coverage rate is over 85%, and the number of problems discovered is 2 to 5 times higher than that of traditional methods.

[0113] This invention constructs an end-to-end automated inspection process from data preprocessing, multi-dimensional feature extraction, AI pattern learning, anomaly detection, root cause analysis to repair verification. It eliminates the need for manual writing of SQL inspection scripts line by line, and a complete inspection can be completed in just 0.5 people × 1 day. Compared with the traditional method of 2 people × 5 days, the inspection efficiency is improved by 20 times. At the same time, it supports efficient inspection of tens of millions or even hundreds of millions of data through vectorized calculation and batch prediction mechanisms.

[0114] This invention categorizes abnormal data in multiple dimensions (data missing, data error, data inconsistency, and business rule violation) and automatically performs root cause analysis for each category. This allows for precise identification of the root cause of the anomaly—such as incorrect mapping rule configuration, ETL transformation logic defects, improper master-slave table migration order, or missing business rule validation. It also outputs targeted remediation suggestions, deepening anomaly identification from the "surface level" to the "root cause level," significantly reducing the technical threshold and time cost of manual investigation.

[0115] This invention establishes an incremental learning mechanism, which allows for adaptive updates when business rules change. It only requires incremental training of the existing model using newly added normal sample data, eliminating the need to search for and modify hundreds or thousands of SQL check scripts as in existing technologies. This reduces maintenance costs by more than 50%, enabling the data migration check solution to be continuously optimized with business development, and providing excellent reusability and long-term sustainability.

[0116] This invention generates a visual inspection report that includes a list of abnormal data, statistical charts of abnormalities, root cause analysis results, and remediation suggestions. It intuitively displays the distribution of abnormality types, abnormality scores, and the scope of abnormality impact in a combination of charts and text. It can deliver differentiated pushes according to the needs of different roles (data migration team, technical team, business team), helping various users to quickly understand the inspection results and accurately locate problems.

[0117] The above are merely preferred embodiments of the present invention and are not intended to limit the present invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A static data migration verification method based on artificial intelligence, characterized in that, include: Normal data samples are obtained from the source system, and migration data to be checked is obtained from the target system. The normal data samples and the migration data to be checked are preprocessed respectively. The preprocessing includes data cleaning, missing value imputation and data standardization to convert the data into a unified format. For each preprocessed data record, multidimensional features are extracted. These multidimensional features include at least numerical features, categorical features, date features, text features, and cross-table association features. Each feature is then converted into a feature vector. An unsupervised learning algorithm is used to train an anomaly detection model using the feature vectors of the normal data samples as the training set. The model learns the statistical distribution patterns, inter-field relationships, and change patterns of the normal data to obtain a normal data pattern model. The feature vector of the migration data to be checked is input into the normal data pattern model. The degree of deviation of each data record from the normal data pattern is calculated as the anomaly score. Data with an anomaly score exceeding a preset threshold is judged as abnormal data. The abnormal data is classified into at least four categories: missing data, incorrect data, inconsistent data, and violation of business rules. Root cause analysis is performed on each category of abnormality, and the cause of the abnormality and corresponding repair suggestions are output. Generate a check report containing a list of abnormal data, anomaly statistics, root cause analysis results, and remediation suggestions. After repairing the abnormal data according to the remediation suggestions, re-execute the anomaly detection to verify the remediation effect.

2. The data migration static verification method based on artificial intelligence according to claim 1, characterized in that, The unsupervised learning algorithm is an autoencoder or an isolated forest.

3. The data migration static verification method based on artificial intelligence according to claim 2, characterized in that, The autoencoder includes an encoder and a decoder. The encoder compresses the input high-dimensional feature vector into a low-dimensional hidden layer representation, and the decoder restores the low-dimensional hidden layer representation to the reconstructed feature vector. The model training aims to minimize the reconstruction error of normal data.

4. The data migration static verification method based on artificial intelligence according to claim 1, characterized in that, The anomaly score is the reconstruction error between the feature vector of the data record and the reconstructed feature vector output by the normal data pattern model. The larger the reconstruction error, the higher the degree to which the data record deviates from the normal pattern.

5. The data migration static verification method based on artificial intelligence according to claim 1, characterized in that, The root cause analysis includes: Data missing root cause analysis is used to identify incomplete mapping rules or defects in ETL programs; Data error root cause analysis is used to identify field type mismatches or inconsistent encoding formats; Data inconsistency root cause analysis is used to identify unmet referential integrity constraints or data discrepancies between the master and child tables; Root cause analysis of business rule violations is used to identify missing or misconfigured business rule validation logic.

6. The data migration static verification method based on artificial intelligence according to claim 1, characterized in that, It also includes an incremental learning step: when business rules change, the normal data pattern model is incrementally trained using new labeled normal sample data, so that the model can be updated adaptively.

7. A data migration static verification system based on artificial intelligence, implementing the data migration static verification method based on artificial intelligence as described in any one of claims 1 to 6, characterized in that, include: The data acquisition module is used to acquire data samples from the source system and the target system and perform preprocessing. The preprocessing includes data cleaning, missing value imputation and data standardization. The preprocessed data is then output to the feature extraction module. The feature extraction module is used to receive the data output by the data acquisition module, extract multi-dimensional features and convert them into feature vectors, and output them to the model training module and the anomaly detection module respectively. The model training module is used to receive the feature vector of the normal data sample output by the feature extraction module, train the anomaly detection model using an unsupervised learning algorithm to obtain the normal data pattern model, and deploy the model to the anomaly detection module. An anomaly detection module is used to receive the feature vector of the kernel data to be inspected output by the feature extraction module and the normal data pattern model deployed by the model training module, calculate the anomaly score and determine the abnormal data, and output the abnormal data to the root cause analysis module. The root cause analysis module is used to receive the abnormal data output by the anomaly detection module, perform classification and root cause analysis, and output the cause of the anomaly and repair suggestions to the report generation module. The report generation module is used to receive the abnormal causes and repair suggestions output by the root cause analysis module, generate a check report and output it to the repair verification module; The repair verification module is used to receive the inspection report output by the report generation module, repair abnormal data according to the repair suggestions, and re-trigger the anomaly detection module after the repair is completed to verify the repair effect.

8. The data migration static verification system based on artificial intelligence according to claim 7, characterized in that, The feature extraction module includes: Numerical feature extraction unit, used to extract statistical features of numerical fields; Categorical feature extraction unit, used for one-hot encoding of categorical fields; The date feature extraction unit is used to extract the time component features of the date field; Text-based feature extraction unit, used to extract NLP semantic features from text fields; The correlation-type feature extraction unit is used to extract correlation features between tables or between master and child tables.

9. The data migration static verification system based on artificial intelligence according to claim 7, characterized in that, The anomaly detection module includes: The batch detection unit is used to perform anomaly detection on batch data that has been migrated. A real-time detection unit is used to perform streaming anomaly detection on data during the migration process; Anomaly scoring unit is used to calculate anomaly score for each data item; The anomaly classification unit is used to categorize detection results according to anomaly type.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the data migration static verification method based on artificial intelligence as described in any one of claims 1 to 6.

Images