Database migration anomaly detection method, system, equipment, medium and product

Abstract

The invention discloses a database migration anomaly detection method, system and device, a medium and a product, and relates to the technical field of anomaly detection.The method comprises the steps that field information which comprises anomaly annotations and has a mapping relation in a source database and a target database is obtained to serve as data samples; the abnormal annotation comprises precision loss and association failure; performing abnormal data enhancement on the data sample by increasing disturbance and an SMOTE algorithm to generate an enhanced sample data set; according to the enhanced sample data set, establishing and training a random forest model used for distinguishing normal data and abnormal data; and capturing data change events in a database migration process, and classifying the data change events by adopting the random forest model to obtain an anomaly detection result. By adopting the embodiment of the invention, data representation and processing differences among different databases can be adapted, and implicit anomalies such as precision loss and association failure in heterogeneous database migration can be sensitively detected.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of anomaly detection, and in particular to a database migration anomaly detection method, system, device, medium and product. BACKGROUND

[0002] In the scenarios of enterprise scale expansion, traditional architecture transformation or cost optimization, the demand for database migration is promoted. The traditional anomaly identification method often relies on preset rules and parameters to identify data problems.

[0003] However, in the actual heterogeneous database migration process, some anomalies caused by special data structures or business logic often have strong concealment. This is because these anomalies do not break the field type constraints of the target database at the technical level, so they will not trigger an error during migration, and it is also difficult to be captured by the conventional data integrity tool that relies on field rule verification. SUMMARY

[0004] The technical problem to be solved by the present application is how to improve the integrity of heterogeneous database anomaly detection. By providing a database migration anomaly detection method, system, device, medium and product, the implicit anomalies such as precision loss and correlation failure in heterogeneous database migration can be detected sensitively.

[0005] To solve the above technical problems, the present application provides a database migration anomaly detection method, comprising:

[0006] Obtain field information including anomaly annotations and having a mapping relationship in the source database and the target database as data samples; the anomaly annotations include precision loss and correlation failure;

[0007] Perform anomaly data enhancement on the data samples by increasing perturbation and SMOTE algorithm to generate an enhanced sample data set;

[0008] According to the enhanced sample data set, a random forest model for distinguishing normal data and abnormal data is established and trained;

[0009] Capture data change events in the database migration process, classify the data change events using the random forest model, and obtain an anomaly detection result.

[0010] As an improvement of the above-mentioned scheme, the obtaining of the field information including anomaly annotations and having a mapping relationship in the source database and the target database as data samples comprises:

[0011] According to the field type mapping relationship and the constraint rule mapping relationship of the source database and the target database, the target field and the database table where the target field is located are determined;

[0012] According to the actual migration data of the source database and the target database, a data set including information of the target field and information of the database table is created;

[0013] According to the precision loss and association failure of each item in the data set, the data set is marked as an exception to form a data sample.

[0014] As an improvement of the above scheme, the attributes of the data set include a mapping relationship identifier, a table name of the source database, a table name of the target database, a column name of the source database, a column name of the target database, a field data type of the source database, a field data type of the target database, a field value of the source database, a field value of the target database, an exception flag bit, and an exception type.

[0015] As an improvement of the above scheme, the data sample is enhanced by adding disturbance and SMOTE algorithm to generate an enhanced sample data set, including:

[0016] The abnormal data samples are filtered from the data sample, and the abnormal data samples are divided into numerical data and classification data according to the field data type;

[0017] The numerical data is disturbed by adding Gaussian noise to generate first enhanced abnormal data;

[0018] The classification data is enhanced by SMOTE algorithm to generate second enhanced abnormal data;

[0019] The data sample, the first enhanced abnormal data, and the second enhanced abnormal data are merged to generate an enhanced sample data set.

[0020] As an improvement of the above scheme, the numerical data is disturbed by adding Gaussian noise to generate first enhanced abnormal data, including:

[0021] According to the original value of the source database before migration and the target value of the target database, a noise factor is generated;

[0022] The standard deviation of the numerical data is obtained, and the standard deviation and the noise factor are used to obtain a volatility benchmark;

[0023] Random Gaussian noise with a mean of 0 and a standard deviation of the volatility benchmark is added to the numerical data to generate first enhanced abnormal data.

[0024] As an improvement of the above scheme, the noise factor is generated according to the original value of the source database before migration and the target value of the target database, including:

[0025] According to the original value of the source database before and after migration and the target value of the target database, the precision loss ratio and the coefficient of variation are calculated;

[0026] Obtain a plurality of preset reference values, a preset adjustment range and a preset amplification factor parameter combination;

[0027] According to the precision loss ratio and the coefficient of variation, the initial noise factor corresponding to each parameter combination is calculated;

[0028] Create a linear sample set, and add disturbance to the linear sample set using the initial noise factor to obtain an initial disturbance result corresponding to each parameter combination;

[0029] According to the initial disturbance result, a linear regression model is trained to obtain the mean square error of each parameter combination;

[0030] According to the mean square error, the optimal parameter combination is selected from the parameter combination;

[0031] According to the initial noise factor corresponding to the optimal parameter combination, a noise factor is obtained.

[0032] As an improvement of the above scheme, the data enhancement of the classification data through the SMOTE algorithm to generate second enhanced abnormal data, comprising:

[0033] Through a preset label encoder, the field value attribute of the target database of the classification data is converted into a numerical value;

[0034] The converted field value attribute of the target database in the classification data is taken as a label, and other attributes are taken as features, and a SMOTE oversampler is used to oversample the label and the feature to generate initial enhanced data;

[0035] Through a preset label encoder, the field value attribute of the target database in the initial enhanced data is converted into text to obtain second enhanced abnormal data.

[0036] As an improvement of the above scheme, the random forest model for distinguishing normal data and abnormal data is established and trained according to the enhanced sample data set, comprising:

[0037] An initial random forest model for distinguishing normal data and abnormal data is established;

[0038] According to the number of attributes of the enhanced sample data set, the maximum tree depth of the initial random forest model is obtained through cross-validation;

[0039] The enhanced sample data set is divided into a training set and a test set;

[0040] According to the training set, the initial random forest model is trained by a random forest algorithm;

[0041] The trained initial random forest model is fitted and evaluated by using the test set, and the maximum tree depth or the number of trees of the initial random forest model is adjusted according to the fitting evaluation result to obtain a random forest model.

[0042] As an improvement of the above scheme, the maximum tree depth of the initial random forest model is obtained by cross-validation according to the number of attributes of the enhanced sample data set, comprising:

[0043] According to the number of attributes of the enhanced sample data set, an initial depth range is set;

[0044] According to a predetermined random state, each tree depth in the initial depth range is iterated, and the average accuracy of each tree depth is calculated by 5-fold cross-validation;

[0045] According to the stability of the average accuracy, the maximum tree depth of the initial random forest model is selected from the initial depth range.

[0046] As an improvement of the above scheme, after the random forest model for distinguishing normal data and abnormal data is established and trained according to the enhanced sample data set, the database migration anomaly detection method further comprises:

[0047] According to the enhanced sample data set, the precision and recall rate of the random forest model are calculated;

[0048] If the precision is greater than a preset first threshold and the recall rate is less than a preset second threshold, the abnormal data enhancement is performed again, and the random forest model is established and trained by using the enhanced sample data set after increment;

[0049] If the recall rate is greater than a preset third threshold and the precision is less than a preset fourth threshold, the data sample is cleaned, the abnormal data is enhanced by using the cleaned data sample, and the random forest model is regenerated.

[0050] The embodiment of the application also provides a database migration anomaly detection system, comprising:

[0051] A data sample acquisition module is configured to acquire field information including abnormal annotations and having a mapping relationship in a source database and a target database as a data sample; the abnormal annotations include precision loss and association failure;

[0052] A data enhancement module is configured to perform abnormal data enhancement on the data sample by increasing disturbance and SMOTE algorithm to generate an enhanced sample data set;

[0053] A model construction module is configured to establish and train a random forest model for distinguishing normal data and abnormal data according to the enhanced sample dataset;

[0054] An anomaly detection module is configured to capture a data change event in a database migration process, and classify the data change event by using the random forest model to obtain an anomaly detection result.

[0055] Embodiments of the present application also provide a database migration anomaly detection device, which comprises a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, and the processor implements the database migration anomaly detection method as described above when executing the computer program.

[0056] Embodiments of the present application also provide a computer readable storage medium, which comprises a stored computer program, wherein the computer readable storage medium controls a device where the computer readable storage medium is located to execute the database migration anomaly detection method as described above when the computer program runs.

[0057] Embodiments of the present application also provide a computer program product, which comprises a computer program or computer instructions, and the computer program or the computer instructions execute the database migration anomaly detection method as described above when executed by a processor.

[0058] Compared with the prior art, the database migration anomaly detection method, system, device, medium and product provided by the embodiments of the present application can obtain field information including abnormal annotations and having a mapping relationship in a source database and a target database as data samples, wherein the abnormal annotations include precision loss and correlation failure; the data samples are subjected to abnormal data enhancement by increasing disturbance and SMOTE algorithm to generate an enhanced sample dataset; a random forest model for distinguishing normal data and abnormal data is established and trained according to the enhanced sample dataset; a data change event in a database migration process is captured, and the data change event is classified by using the random forest model to obtain an anomaly detection result. By using the embodiments of the present application, the data representation and processing differences between different databases can be adapted, and the implicit anomalies such as precision loss and correlation failure in heterogeneous database migration can be sensitively detected. BRIEF DESCRIPTION OF DRAWINGS

[0059] Figure 1 is a step flowchart of a database migration anomaly detection method provided by the embodiments of the present application;

[0060] Figure 2 is a visualization heat map of mean square errors of different parameter combinations provided by the embodiments of the present application;

[0061] Figure 3is a structural schematic diagram of an abnormality detection system for database migration provided by an embodiment of the present application.

[0062] Figure 4 is a structural schematic diagram of an abnormality detection device for database migration provided by an embodiment of the present application. DETAILED DESCRIPTION

[0063] The technical solutions in the embodiments of the present application will be clearly and completely described in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by a person of ordinary skill in the art without creative labor fall within the scope of protection of the present application.

[0064] In the description and claims of the specification, it is to be understood that the terms first, second, etc. are used only for the purpose of description and are not to be construed as indicating or implying relative importance or an indicated number of features. They are also not necessarily used to describe a sequence or an order, unless explicitly stated otherwise. The terms are interchangeable under appropriate circumstances. Thus, a feature described as“first” can be explicitly or implicitly included in at least one of the features.

[0065] The embodiments of the present application provide an abnormality detection method for database migration. Please refer to Figure 1 In the present embodiment, the abnormality detection method for database migration is specifically executed through steps S1 to S4.

[0066] S1, obtain field information including abnormal annotations and having a mapping relationship in a source database and a target database as data samples; the abnormal annotations include precision loss and correlation failure.

[0067] Traditional abnormality detection methods often rely on pre-set rules and parameters to identify data problems. When the abnormal data does not technically violate any explicit data integrity rules, the traditional method cannot identify it as abnormal. In the embodiments of the present application, the detection model is trained by pre-annotating the data samples of precision loss and correlation failure, breaking the limitation of static detection rules, and being able to sensitively detect implicit abnormalities such as precision loss and correlation failure in heterogeneous database migration.

[0068] S2, perform abnormal data enhancement on the data samples by increasing perturbation and SMOTE algorithm to generate an enhanced sample data set.

[0069] Since the magnitude of implicit abnormal data is small and not common, the traditional rule checking method cannot cover all edge cases based on the currently known abnormal data. In the above scheme, by increasing the disturbance and SMOTE (Synthetic Minority Over-sampling Technique) algorithm to synthesize minority class samples, not only the data imbalance problem is solved, but also the implicit abnormal mode of the data is covered, so that the model can learn these complex edge patterns.

[0070] The embodiment of the application effectively improves the number of abnormal samples in the enhanced sample data set by performing abnormal data enhancement on the data samples, and further improves the accuracy of abnormal detection.

[0071] S3, according to the enhanced sample data set, a random forest model for distinguishing normal data and abnormal data is established and trained.

[0072] In the embodiment of the application, the data enhancement technology widens the response range of the model, improves the efficiency and accuracy of the model in identifying and processing rare abnormal events. The enhanced sample data set not only helps the model to better generalize, but also ensures that the model can still effectively learn and identify the abnormalities in the case of rare abnormal data.

[0073] S4, capturing data change events in the database migration process, using the random forest model to classify the data change events to obtain an abnormal detection result.

[0074] By synchronously detecting during the database migration process, compared with full checking after migration, the abnormal situation can be found earlier, the repair cycle is shortened, and the impact on the business is reduced.

[0075] Considering that the source database and the target database use different data models and storage formats, data precision loss and relationship integrity problems may be caused, which seriously affect the data quality and the success rate of migration, so the embodiment of the application proposes the above technical scheme.

[0076] Precision loss usually occurs during data conversion or calculation, and the processing method of the heterogeneous database system for numerical precision may be different, which leads to rounding error data in the migration process. Especially when floating point and fixed point conversion is involved.

[0077] Association failure is generally caused by the association of two tables in the source database through foreign key constraints, and in the data migration, due to migration sequence problems or data filtering problems, the foreign key in the target database points to a non-existent record, which further leads to inconsistent data.

[0078] Anomalies involving precision loss and association failure may appear to conform to field rules in the target database, but due to mishandling during migration, they do not accurately reflect the true intent or accuracy of the original data. Existing anomaly detection methods cannot directly identify these anomalies based on pre-defined rules and strategies when observing data solely in the target database.

[0079] The aforementioned anomaly detection method for database migration overcomes the limitations of traditional rule-based validation in its insufficient coverage of latent anomalies and edge cases. It lays the foundation for supervised learning by acquiring anomaly-labeled samples of the mapping fields of the source and target databases. By combining perturbation and the SMOTE algorithm, it solves the problems of small anomalous data volume and imbalanced distribution. The generated augmented samples enable the random forest model to fully learn diverse anomaly patterns. The optimized and trained model can accurately identify and locate anomalies by capturing data change events in real time during the migration process. It reduces the dependence on preset rules and static mapping, can adapt to differences in data representation and processing between different databases, and sensitively detects latent anomalies such as precision loss and association failure in heterogeneous database migration.

[0080] As a preferred implementation, step S1, obtaining field information from the source database and target database that includes anomaly annotations and has a mapping relationship, as a data sample, includes:

[0081] Based on the field type mapping relationship and constraint rule mapping relationship between the source database and the target database, determine the target field and the database table where the target field is located;

[0082] Based on the actual migration data from the source and target databases, create a dataset that includes information about the target fields and information about the database tables;

[0083] Based on the accuracy loss and association failure of each item in the dataset, the dataset is annotated to form a data sample.

[0084] It should be noted that the target field is the field that needs to be focused on during the migration process, which may cause anomalies and has a clear mapping relationship between the source database and the target database.

[0085] For example, if there is a mapping relationship between the first field in the source database and the second field in the target database, and the source and target databases have different requirements for the numerical precision of this field, then the target field can be filtered out based on the field type mapping relationship and the constraint rule mapping relationship. By locking the target field, the waste of resources caused by indiscriminate detection can be avoided, and high-risk fields can be focused on.

[0086] It should also be noted that, in this embodiment of the invention, all data of a set of target fields are considered as an item in the dataset. Each item has the same attribute type, and the items are distinguished by specific attribute values. For example, one row in the dataset is a set of attributes for one item, and one column represents the same attribute for several items.

[0087] In the above scheme, from mapping relationship to data creation and then anomaly labeling, the high consistency between data samples and actual migration scenarios is ensured. At the same time, noise introduced by irrelevant fields or false mappings is avoided, so that subsequent enhancement techniques and model training have clear learning objectives.

[0088] Further, preferably, the attributes of the dataset include a mapping identifier, table names of the source database, table names of the target database, column names of the source database, column names of the target database, field data types of the source database, field data types of the target database, field values ​​of the source database, field values ​​of the target database, anomaly flags, and anomaly types.

[0089] In some preferred embodiments, the information of the target field in the actual migration data and the information of the database table are integrated into the same data storage system and stored according to preset attributes to obtain an initial dataset. The attributes of each item include a mapping identifier (MappingID), the source database table name (SourceTableName), the target database table name (TargetTableName), the source database column name (SourceColumnName), the target database column name (TargetColumnName), the source database field data type (SourceDataType), the target database field data type (TargetDataType), the source database field value (SourceData), the target database field value (TargetData), an exception flag (ErrFlag), and an exception type (ErrType). Specifically, an exception flag of 0 indicates normal data, and an exception flag of 1 indicates abnormal data; an exception type of 1 (ErrType) indicates loss of precision, and an exception type of 2 (ErrType) indicates association failure.

[0090] Next, the initial dataset is cleaned to ensure the uniqueness and integrity of each data entry. For example, during data cleaning, for each row in the initial dataset, if any data is missing in either the source or target data, that row is deleted, and duplicate records are further checked and removed.

[0091] In one specific embodiment, a portion of the dataset is shown in Table 1.

[0092] Table 1

[0093]

[0094]

[0095] As a preferred implementation, step S2, performing anomalous data augmentation on the data samples by adding perturbation and the SMOTE algorithm to generate an augmented sample dataset, is executed through steps S21-S24:

[0096] S21. Filter out abnormal data samples from the data samples, and divide the abnormal data samples into numerical data and categorical data according to the field data type.

[0097] Preferably, abnormal data samples are filtered from the data samples by using the ErrFlag attribute, and then the data rows containing numerical data and categorical data are filtered according to the data types SourceDataType of the source database and TargetDataType of the target database to classify the abnormal data samples.

[0098] In this embodiment of the invention, the abnormal data samples are divided in order to better adapt to the enhancement methods of different data types. The difference in characteristics between numerical and categorical data determines the different abnormal patterns and enhancement requirements. The type division ensures the targeting of the enhancement strategy.

[0099] S22. Add Gaussian noise to the numerical data to increase perturbation and generate the first enhanced anomaly data.

[0100] Adding Gaussian noise to numerical anomaly data can generate variant anomaly samples with subtle differences while preserving the original anomaly characteristics, effectively simulating continuous numerical anomalies that may occur in actual migration.

[0101] S23. The classified data is augmented using the SMOTE algorithm to generate second augmented anomaly data.

[0102] The spatial representation of categorical data does not have the continuous numerical spatial representation of numerical data. This invention enhances categorical data through the SMOTE algorithm, which avoids the risk of overfitting caused by simple replication, while retaining the core features of categorical anomalies. It solves the problem of scarce and sparsely distributed categorical data and effectively expands the sample space of categorical anomalies.

[0103] S24. Merge the data samples, the first enhanced anomaly data, and the second enhanced anomaly data to generate an enhanced sample dataset.

[0104] By merging the original data samples with two types of enhanced anomaly data, the resulting enhanced sample dataset retains the true distribution characteristics of the original data and compensates for the deficiencies of insufficient number and single type of anomaly samples through targeted enhancement, thereby reducing the risk of missed detection due to insufficient samples.

[0105] Further, preferably, step S22, adding Gaussian noise to the numerical data to generate first enhanced anomalous data, includes:

[0106] Noise factors are generated based on the original values ​​of the source database before and after migration and the target values ​​of the target database.

[0107] Obtain the standard deviation of the numerical data, and obtain the volatility benchmark based on the standard deviation and the noise factor;

[0108] Random Gaussian noise with a mean of 0 and a standard deviation equal to the volatility benchmark is added to the numerical data to generate the first enhanced anomaly data.

[0109] By introducing a noise factor, the intensity of the disturbance is correlated with the magnitude of the deviation that may occur in the actual migration, thus avoiding the distortion of anomalous features caused by unfounded random disturbances. Furthermore, the volatility benchmark based on the standard deviation of the data itself ensures that the range of disturbance is compatible with the dispersion of the data itself. The generated first enhanced anomalous data can not only retain the core features of the original numerical anomaly, but also simulate more potential marginal anomalies through reasonable noise expansion.

[0110] In some preferred embodiments, a Python implementation is used as an example. First, the NumPy and Pandas libraries are imported. The standard deviation of the TargetData column of the target database field values ​​in the numerical data is calculated. Then, the product of the standard deviation and the noise factor is calculated to obtain the volatility benchmark. Next, a TargetData_enhanced column is created in the dataset formed by the numerical data. The value of the TargetData_enhanced column is obtained by adding random Gaussian noise with a mean of 0 and a standard deviation equal to the volatility benchmark to the field values ​​of the target database.

[0111] Furthermore, preferably, the step of generating a noise factor based on the original values ​​of the source database before and after migration and the target values ​​of the target database includes:

[0112] Calculate the precision loss ratio and coefficient of variation based on the original values ​​of the source database and the target values ​​of the target database before and after migration;

[0113] Obtain a combination of parameters including several preset reference values, preset adjustment ranges, and preset amplification factors;

[0114] Calculate the initial noise factor corresponding to each of the parameter combinations based on the accuracy loss ratio and the coefficient of variation;

[0115] Create a linear sample set, and add perturbation to the linear sample set using the initial noise factor to obtain the initial perturbation result corresponding to each parameter combination;

[0116] A linear regression model is trained based on the initial perturbation results to obtain the mean square error of each parameter combination;

[0117] Based on the mean square error, select the optimal parameter combination from the parameter combinations;

[0118] The noise factor is obtained based on the initial noise factor corresponding to the optimal parameter combination.

[0119] It should be noted that in the above scheme, the proportion of precision loss is measured by the average ratio of the absolute value of the difference between the original value and the target value to the original value; the relative dispersion is measured by the coefficient of variation; the benchmark value is generally set at a moderate level of noise to enhance the robustness of the data without excessively affecting the authenticity of the data; the adjustment range is the range in which the adjustment factor can be changed to ensure that the noise adjustment is not too large and to avoid data distortion in extreme cases; the amplification factor can make the adjustment factor more sensitive to data variability and precision loss.

[0120] It should also be noted that, in the embodiments of the present invention, multiple reference values, adjustment ranges and amplification factors may be preset respectively, and these preset reference values, adjustment ranges and amplification factors may be combined to form several parameter combinations.

[0121] In some preferred embodiments, the noise factor NF is represented as:

[0122]

[0123] in, For the percentage of accuracy loss, σ is the coefficient of variation, b is the baseline value, r is the adjustment range, α is the amplification factor, SourceDatai is the i-th original data in the source database, TargetDatai is the i-th target data in the target database, σ is the standard deviation of the original data, and μ is the mean of the original data.

[0124] For example, preset baseline values ​​of 0.05, 0.075, and 0.1 represent different levels of base noise, which will be used to observe how noise levels affect the accuracy of the model. Preset adjustment ranges of 0.01, 0.025, and 0.05 are used to test how different levels of noise intensity adjustment affect the data. Preset amplification factors of 5, 10, and 15 are used to understand the model's sensitivity to data variations.

[0125] A set of data points that follow a defined linear relationship is randomly generated to form a linear sample set. For example, the linear relationship is represented as twice the eigenvalue of each data point plus random noise.

[0126] Next, nested loops are used to iterate through preset baseline values, adjustment ranges, and amplification factors to obtain each possible parameter combination. For each parameter combination, the current baseline value, adjustment range, and amplification factor are applied to the linear sample set, and noise is added using the add_noise() function to obtain the initial perturbation result, which is used as training data.

[0127] The initial perturbation results are divided into training and test sets according to a preset ratio. A linear regression model is trained on the training set and used to make predictions on the test set. Please refer to [link to relevant documentation]. Figure 2 , Figure 2 The mean squared error of each parameter combination was visualized. If a parameter combination has the lowest mean squared error, it indicates that the model has the highest prediction accuracy under that parameter combination, and this parameter combination is considered the optimal parameter combination.

[0128] Preferably, step S23, performing data augmentation on the classified data using the SMOTE algorithm to generate second augmented anomaly data, includes:

[0129] The field value attributes of the target database of the categorized data are converted into numerical values ​​using a preset label encoder.

[0130] The field value attributes of the target database after transformation in the classified data are used as labels, and other attributes are used as features. The SMOTE oversampler is used to oversample the labels and features to generate initial augmented data.

[0131] The field value attributes of the target database in the initial augmented data are converted into text using a preset tag encoder to obtain the second augmented anomaly data.

[0132] It should be noted that the preset label encoder is used for label conversion. Specifically, in this embodiment of the invention, the field values ​​of the categorized data are converted between text and numeric types. When converting the categorized data to numeric values, a unique numeric number is assigned to each text value to establish a mapping relationship. This mapping relationship is then used to convert text to numeric values, satisfying the SMOTE algorithm's requirements for the input data format. After generating the initial augmented data, the reverse mapping is also performed based on this mapping relationship to obtain the second augmented anomaly data.

[0133] The above scheme addresses the problem of scarce abnormal samples in classification data and difficulty in directly enhancing discrete features by combining the label encoder with the SMOTE algorithm. This improves the practicality of the enhanced samples, enables the model to learn abnormal patterns in classification data more comprehensively, and enhances the ability to detect discrete anomalies.

[0134] In some preferred embodiments, when performing step S24, merging the data samples, the first enhanced anomaly data, and the second enhanced anomaly data to generate an enhanced sample dataset, the data sources in the enhanced sample dataset are labeled.

[0135] For example, it is labeled as the original data sample by Original, as the first augmented anomalous data by NumericalEnhanced, and as the second augmented anomalous data by SMOTEEnhanced.

[0136] As a preferred implementation, step S3, based on the augmented sample dataset, establishes and trains a random forest model to distinguish between normal and abnormal data, and is executed through steps S31-S35:

[0137] S31. Establish an initial random forest model to distinguish between normal and abnormal data;

[0138] S32. Based on the number of attributes in the augmented sample dataset, the maximum tree depth of the initial random forest model is obtained through cross-validation;

[0139] S33. Divide the augmented sample dataset into a training set and a test set;

[0140] S34. Based on the training set, train the initial random forest model using the random forest algorithm;

[0141] S35. The initial random forest model after training is fitted and evaluated using the test set. The maximum tree depth or number of trees of the initial random forest model is adjusted according to the fitting and evaluation results to obtain the random forest model.

[0142] For example, step S31 imports a random forest classifier using the sklearn library in Python to establish the initial random forest model.

[0143] It should be noted that by repeatedly training and evaluating the model performance at different maximum tree depths through cross-validation, parameter bias caused by the randomness of a single data partition can be avoided, making the determined maximum tree depth more in line with the actual complexity of the data. Furthermore, cross-validation will select a tree depth that can capture complex multi-dimensional anomalies without overlearning noise, thus balancing the model's fitting ability and complexity.

[0144] In the above scheme, the maximum tree depth of the random forest model is dynamically adjusted according to the number of attributes of the augmented samples, so that the model complexity matches the complexity of the abnormal features in the database migration, which can improve the computational efficiency while ensuring detection accuracy. Furthermore, through cross-validation and test set evaluation, the overfitting of the random forest model is effectively avoided, ensuring that the model can still stably identify newly emerging anomalies in real database migration scenarios. The resulting random forest model can fully learn various anomaly patterns in the augmented sample dataset and sensitively capture subtle differences between normal and abnormal data.

[0145] Further, preferably, step S32, obtaining the maximum tree depth of the initial random forest model through cross-validation based on the number of attributes in the augmented sample dataset, includes:

[0146] The initial depth range is set based on the number of attributes in the augmented sample dataset;

[0147] Based on a preset random state, the depth of each tree in the initial depth range is traversed cyclically, and the average accuracy of each tree depth is calculated by 5-fold cross-validation.

[0148] Based on the stability of the average accuracy, the maximum tree depth of the initial random forest model is selected from the initial depth range.

[0149] The number of attributes in the augmented sample dataset reflects the data complexity, which serves as the basis for setting the initial depth range, avoiding blindly testing depths that are too large or too small. Furthermore, the augmented sample dataset is randomly divided into 5 subsets using 5-fold cross-validation. Each time, 4 subsets are used for training and 1 subset for validation, and this process is repeated 5 times, with the average accuracy calculated. This approach ensures that every sample in the dataset participates in both training and validation, avoiding bias caused by the randomness of a single data partition. Moreover, calculating the average accuracy through multiple rounds of validation ensures that the final selected depth is the optimal solution that performs stably across diverse data distributions.

[0150] In some preferred embodiments, the step of selecting the maximum tree depth of the initial random forest model from the initial depth range based on the stability of the average accuracy specifically involves finding the depth value before the cross-validation accuracy begins to stabilize or begins to decline for the first time, and using this depth value as the maximum tree depth. The maximum tree depth can achieve the best balance between bias and variance.

[0151] For example, the initial depth range is [3, 20], and the preset random state is 34. The fixed random state is used to ensure the reproducibility of the results. The average accuracy obtained by 5-fold cross-validation is shown in Table 2.

[0152] Table 2

[0153]

[0154]

[0155] Based on the stability of the average accuracy, the optimal maximum tree depth is 13.

[0156] In some preferred embodiments, step S34 involves training the initial random forest model using the random forest algorithm based on the training set. This is achieved by calling the fit method of the random forest classifier instance, passing the feature dataset X_train and the corresponding target variable set y_train from the training set to this method. The classifier will then iterate multiple times on the training set (X_train, y_train) to try to find the optimal decision tree structure so as to make accurate predictions on unseen data.

[0157] In some preferred embodiments, step S35 involves using the test set to evaluate the fit of the trained initial random forest model, and adjusting the maximum tree depth or number of trees in the initial random forest model based on the evaluation results to obtain the random forest model. Alternatively, the training and test sets can be used together for fitting evaluation. Specifically, the `score` method of the random forest classifier instance is called, and the feature dataset X_train and the corresponding target variable set y_train are passed to this method to calculate the training accuracy. The same method is also used to calculate the test accuracy, which is achieved by passing the feature dataset X_test and the corresponding target variable set y_test. Finally, the values ​​of these two accuracies are output to evaluate the model's performance.

[0158] If the training accuracy and test accuracy are close and remain at a high level, it indicates that the model is neither overfitting nor underfitting, but has achieved good generalization ability.

[0159] If the model exhibits overfitting, for example, outperforming the test set by more than 5% on the training set, reduce the maximum depth or the number of trees to simplify the model. If the model is underfitting, for example, performing poorly on both the training and test sets, increase the maximum depth or use more trees to increase the model's complexity.

[0160] As a preferred embodiment, after performing step S3, the database migration anomaly detection method further includes:

[0161] Based on the augmented sample dataset, calculate the precision and recall of the random forest model;

[0162] If the precision is greater than a preset first threshold and the recall is less than a preset second threshold, then abnormal data augmentation is performed again, and a random forest model is built and trained using the augmented sample dataset after the increment.

[0163] If the recall rate is greater than a preset third threshold and the precision is less than a preset fourth threshold, then the data samples are cleaned, and the cleaned data samples are used for abnormal data augmentation and a random forest model is regenerated.

[0164] In some preferred embodiments, the present invention is implemented using Python. When executing the above scheme, the `predict` method of the random forest model is called to predict the test set `X_test`, and the prediction result is assigned to `y_pred`. Next, the `precision_score` function in the `sklearn.metrics` module is called to calculate the model's prediction precision, passing the true label `y_test` and the predicted label `y_pred` as parameters, and assigning the calculated precision to the variable `precision`. Subsequently, the `recall_score` function in the `sklearn.metrics` module is called to calculate the model's prediction recall, passing the true label `y_test` and the predicted label `y_pred` as parameters, and assigning the calculated recall to the variable `recall`. Analysis of the variables `precision` and `recall` enables automatic iterative optimization of the model, providing a more reliable and practical anomaly detection capability for database migration.

[0165] In the above scheme, precision reflects the degree of misjudgment in anomaly identification by the model, while recall reflects the degree of missed anomaly identification by the model. By dynamically adjusting the data processing and model training strategies based on these two model performance indicators, precision and recall, the problems of missed or false detections that may exist in single training are solved, and the automatic iterative optimization of the model is realized.

[0166] As a preferred implementation, step S4 involves capturing data change events during the database migration process, classifying the data change events using the random forest model to obtain anomaly detection results, and then deploying a data monitoring system to receive database migration data in real time and perform anomaly detection on it.

[0167] In some preferred embodiments, data changes during database migration are captured in real time using database triggers, log listeners, or change data capture mechanisms. For example, middleware or stream processing systems that support real-time data capture, such as Apache Kafka, Amazon Kinesis, or a custom stream processor, are selected. A service or daemon is developed that can subscribe to data change events and send them to a random forest model to obtain anomaly detection results.

[0168] The anomaly detection method for database migration provided by this invention can achieve comprehensive, accurate and real-time detection of hidden anomalies during database migration, significantly reducing data quality problems and rework costs after migration, while improving adaptability to new anomalies and complex scenarios. It provides efficient and reliable intelligent detection support for different types of database migration, ensuring the smoothness of the migration process and the consistency of data.

[0169] This invention provides an anomaly detection system for database migration. Please refer to [link to relevant documentation]. Figure 3 The database migration anomaly detection system includes a data sample acquisition module 11, a data augmentation module 12, a model building module 13, and an anomaly detection module 14, wherein:

[0170] The data sample acquisition module 11 is used to acquire field information from the source database and the target database that includes anomaly annotations and has a mapping relationship, as data samples; the anomaly annotations include precision loss and association failure;

[0171] Data augmentation module 12 is used to perform anomalous data augmentation on the data samples by adding perturbations and using the SMOTE algorithm, thereby generating an augmented sample dataset;

[0172] Model building module 13 is used to build and train a random forest model to distinguish between normal and abnormal data based on the augmented sample dataset.

[0173] The anomaly detection module 14 is used to capture data change events during the database migration process, and uses the random forest model to classify the data change events to obtain anomaly detection results.

[0174] In a preferred embodiment, the data sample acquisition module 11 is specifically used for:

[0175] Based on the field type mapping relationship and constraint rule mapping relationship between the source database and the target database, determine the target field and the database table where the target field is located;

[0176] Based on the actual migration data from the source and target databases, create a dataset that includes information about the target fields and information about the database tables;

[0177] Based on the accuracy loss and association failure of each item in the dataset, the dataset is annotated to form a data sample.

[0178] Further, preferably, the attributes of the dataset include a mapping identifier, table names of the source database, table names of the target database, column names of the source database, column names of the target database, field data types of the source database, field data types of the target database, field values ​​of the source database, field values ​​of the target database, anomaly flags, and anomaly types.

[0179] In a preferred embodiment, the data enhancement module 12 includes:

[0180] An abnormal data segmentation unit is used to filter abnormal data samples from the data samples and divide the abnormal data samples into numerical data and categorical data according to the field data type.

[0181] The first enhanced anomaly data generation unit is used to add perturbation to the numerical data by adding Gaussian noise to generate the first enhanced anomaly data.

[0182] The second augmented anomaly data generation unit is used to perform data augmentation on the classified data using the SMOTE algorithm to generate second augmented anomaly data.

[0183] An enhanced sample dataset generation unit is used to merge the data sample, the first enhanced abnormal data, and the second enhanced abnormal data to generate an enhanced sample dataset.

[0184] Further, preferably, the first enhanced anomaly data generation unit is specifically used for:

[0185] Noise factors are generated based on the original values ​​of the source database before and after migration and the target values ​​of the target database.

[0186] Obtain the standard deviation of the numerical data, and obtain the volatility benchmark based on the standard deviation and the noise factor;

[0187] Random Gaussian noise with a mean of 0 and a standard deviation equal to the volatility benchmark is added to the numerical data to generate the first enhanced anomaly data.

[0188] Furthermore, preferably, the step of generating a noise factor based on the original values ​​of the source database before and after migration and the target values ​​of the target database includes:

[0189] Calculate the precision loss ratio and coefficient of variation based on the original values ​​of the source database and the target values ​​of the target database before and after migration;

[0190] Obtain a combination of parameters including several preset reference values, preset adjustment ranges, and preset amplification factors;

[0191] Calculate the initial noise factor corresponding to each of the parameter combinations based on the accuracy loss ratio and the coefficient of variation;

[0192] Create a linear sample set, and add perturbation to the linear sample set using the initial noise factor to obtain the initial perturbation result corresponding to each parameter combination;

[0193] A linear regression model is trained based on the initial perturbation results to obtain the mean square error of each parameter combination;

[0194] Based on the mean square error, select the optimal parameter combination from the parameter combinations;

[0195] The noise factor is obtained based on the initial noise factor corresponding to the optimal parameter combination.

[0196] Preferably, the second enhanced anomaly data generation unit is specifically used for:

[0197] The field value attributes of the target database of the categorized data are converted into numerical values ​​using a preset label encoder.

[0198] The field value attributes of the target database after transformation in the classified data are used as labels, and other attributes are used as features. The SMOTE oversampler is used to oversample the labels and features to generate initial augmented data.

[0199] The field value attributes of the target database in the initial augmented data are converted into text using a preset tag encoder to obtain the second augmented anomaly data.

[0200] In a preferred embodiment, the model building module 13 includes:

[0201] The initial model building unit is used to build an initial random forest model to distinguish between normal and abnormal data.

[0202] The model parameter selection unit is used to obtain the maximum tree depth of the initial random forest model through cross-validation based on the number of attributes in the augmented sample dataset.

[0203] A dataset partitioning unit is used to divide the augmented sample dataset into a training set and a test set;

[0204] The model training unit is used to train the initial random forest model using the random forest algorithm based on the training set.

[0205] The model determination unit is used to evaluate the fit of the trained initial random forest model using the test set, and adjust the maximum tree depth or number of trees of the initial random forest model according to the fitting evaluation results to obtain the random forest model.

[0206] Further, preferably, the model parameter selection unit is specifically used for:

[0207] The initial depth range is set based on the number of attributes in the augmented sample dataset;

[0208] Based on a preset random state, the depth of each tree in the initial depth range is traversed cyclically, and the average accuracy of each tree depth is calculated by 5-fold cross-validation.

[0209] Based on the stability of the average accuracy, the maximum tree depth of the initial random forest model is selected from the initial depth range.

[0210] In a preferred embodiment, the database migration anomaly detection system further includes a model iteration module, which, in response to the completion signal of the model building module 13, is used to:

[0211] Based on the augmented sample dataset, calculate the precision and recall of the random forest model;

[0212] If the precision is greater than a preset first threshold and the recall is less than a preset second threshold, then abnormal data augmentation is performed again, and a random forest model is built and trained using the augmented sample dataset after the increment.

[0213] If the recall rate is greater than a preset third threshold and the precision is less than a preset fourth threshold, then the data samples are cleaned, and the cleaned data samples are used for abnormal data augmentation and a random forest model is regenerated.

[0214] The database migration anomaly detection system provided by this invention can achieve comprehensive, accurate and real-time detection of hidden anomalies during database migration, significantly reducing data quality problems and rework costs after migration, while improving adaptability to new anomalies and complex scenarios. It provides efficient and reliable intelligent detection support for different types of database migration, ensuring the smoothness of the migration process and the consistency of data.

[0215] Please see Figure 4 ,Figure 4 This is a structural block diagram of a database migration anomaly detection device provided in an embodiment of the present invention. The database migration anomaly detection device includes a processor 31, a memory 32, and a computer program stored in the memory 32 and executable on the processor 31. When the processor 31 executes the computer program, it implements the steps in the various database migration anomaly detection method embodiments described above, such as steps S1 to S4.

[0216] For example, the computer program may be divided into one or more modules / units, which are stored in the memory 32 and executed by the processor 31 to complete the present invention. The one or more modules / units may be a series of computer program instruction segments capable of performing specific functions, which describe the execution process of the computer program in the database migration anomaly detection device.

[0217] The database migration anomaly detection device may include, but is not limited to, processor 31 and memory 32. Those skilled in the art will understand that the schematic diagram is merely an example of a database migration anomaly detection device and does not constitute a limitation on the device. It may include more or fewer components than illustrated, or combine certain components, or use different components. For example, the database migration anomaly detection device may also include input / output devices, network access devices, buses, etc.

[0218] The processor 31 can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor. The processor 31 is the control center of the database migration anomaly detection device, connecting various parts of the entire database migration anomaly detection device via various interfaces and lines.

[0219] The memory 32 can be used to store the computer programs and / or modules. The processor 31 implements various functions of the database migration anomaly detection device by running or executing the computer programs and / or modules stored in the memory 32 and calling the data stored in the memory 32. The memory 32 may mainly include a program storage area and a data storage area. The program storage area may store the operating system, at least one application program required for a function (such as sound playback function, image playback function, etc.), etc.; the data storage area may store data created according to the use of the mobile phone (such as audio data, phonebook, etc.). In addition, the memory 32 may include high-speed random access memory, and may also include non-volatile memory, such as hard disk, memory, plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, at least one disk storage device, flash memory device, or other volatile solid-state storage device.

[0220] If the modules / units integrated into the database migration anomaly detection device are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the above embodiments of the present invention can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by the processor 31, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include: any entity or device capable of carrying the computer program code, recording media, USB flash drives, portable hard drives, magnetic disks, optical disks, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc.

[0221] The above description represents the preferred embodiments of the present invention. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principles of the present invention, and these improvements and modifications are also considered to be within the scope of protection of the present invention.

Claims

1. A method for detecting anomalies in database migration, characterized in that, include: Obtain field information from the source and target databases that includes anomaly annotations and has a mapping relationship, and use it as a data sample; The anomaly annotations include precision loss and association failure; The data samples are augmented with perturbations and the SMOTE algorithm to generate an augmented sample dataset. Based on the augmented sample dataset, a random forest model for distinguishing between normal and abnormal data is established and trained. Data change events during database migration are captured, and the random forest model is used to classify these events to obtain anomaly detection results.

2. The anomaly detection method for database migration as described in claim 1, characterized in that, The acquisition of field information from the source and target databases, including anomaly annotations and mapping relationships, as data samples includes: Based on the field type mapping relationship and constraint rule mapping relationship between the source database and the target database, determine the target field and the database table where the target field is located; Based on the actual migration data from the source and target databases, create a dataset that includes information about the target fields and information about the database tables; Based on the accuracy loss and association failure of each item in the dataset, the dataset is annotated to form a data sample.

3. The anomaly detection method for database migration as described in claim 2, characterized in that, The attributes of the dataset include mapping identifier, table name of source database, table name of target database, column name of source database, column name of target database, field data type of source database, field data type of target database, field value of source database, field value of target database, exception flag, and exception type.

4. The anomaly detection method for database migration as described in claim 1, characterized in that, The process of augmenting the data samples by adding perturbations and using the SMOTE algorithm to generate an augmented sample dataset includes: From the data sample, abnormal data samples are filtered out, and the abnormal data samples are divided into numerical data and categorical data according to the field data type; The numerical data is perturbed by adding Gaussian noise to generate the first enhanced anomalous data. The classified data is augmented using the SMOTE algorithm to generate second augmented outlier data. The data samples, the first enhanced anomaly data, and the second enhanced anomaly data are merged to generate an enhanced sample dataset.

5. The anomaly detection method for database migration as described in claim 4, characterized in that, The step of adding Gaussian noise to the numerical data to generate first enhanced anomalous data includes: Noise factors are generated based on the original values ​​of the source database before and after migration and the target values ​​of the target database. Obtain the standard deviation of the numerical data, and obtain the volatility benchmark based on the standard deviation and the noise factor; Random Gaussian noise with a mean of 0 and a standard deviation equal to the volatility benchmark is added to the numerical data to generate the first enhanced anomaly data.

6. The anomaly detection method for database migration as described in claim 5, characterized in that, The step of generating noise factors based on the original values ​​of the source database before and after migration and the target values ​​of the target database includes: Calculate the precision loss ratio and coefficient of variation based on the original values ​​of the source database and the target values ​​of the target database before and after migration; Obtain a combination of parameters including several preset reference values, preset adjustment ranges, and preset amplification factors; Calculate the initial noise factor corresponding to each of the parameter combinations based on the accuracy loss ratio and the coefficient of variation; Create a linear sample set, and add perturbation to the linear sample set using the initial noise factor to obtain the initial perturbation result corresponding to each parameter combination; A linear regression model is trained based on the initial perturbation results to obtain the mean square error of each parameter combination; Based on the mean square error, select the optimal parameter combination from the parameter combinations; The noise factor is obtained based on the initial noise factor corresponding to the optimal parameter combination.

7. The anomaly detection method for database migration as described in claim 4, characterized in that, The step of augmenting the classified data using the SMOTE algorithm to generate second augmented outlier data includes: The field value attributes of the target database of the categorized data are converted into numerical values ​​using a preset label encoder. The field value attributes of the target database after transformation in the classified data are used as labels, and other attributes are used as features. The SMOTE oversampler is used to oversample the labels and features to generate initial augmented data. The field value attributes of the target database in the initial augmented data are converted into text using a preset tag encoder to obtain the second augmented anomaly data.

8. The anomaly detection method for database migration as described in claim 1, characterized in that, The step of establishing and training a random forest model to distinguish between normal and abnormal data based on the augmented sample dataset includes: Establish an initial random forest model to distinguish between normal and abnormal data; Based on the number of attributes in the augmented sample dataset, the maximum tree depth of the initial random forest model is obtained through cross-validation; The augmented sample dataset is divided into a training set and a test set; Based on the training set, the initial random forest model is trained using the random forest algorithm; The initial random forest model after training is fitted and evaluated using the test set. Based on the fitting and evaluation results, the maximum tree depth or the number of trees in the initial random forest model is adjusted to obtain the random forest model.

9. The anomaly detection method for database migration as described in claim 8, characterized in that, The step of obtaining the maximum tree depth of the initial random forest model through cross-validation based on the number of attributes in the augmented sample dataset includes: The initial depth range is set based on the number of attributes in the augmented sample dataset; Based on a preset random state, the depth of each tree in the initial depth range is traversed cyclically, and the average accuracy of each tree depth is calculated by 5-fold cross-validation. Based on the stability of the average accuracy, the maximum tree depth of the initial random forest model is selected from the initial depth range.

10. The anomaly detection method for database migration as described in claim 1, characterized in that, After establishing and training a random forest model to distinguish between normal and abnormal data based on the augmented sample dataset, the anomaly detection method for database migration further includes: Based on the augmented sample dataset, calculate the precision and recall of the random forest model; If the precision is greater than a preset first threshold and the recall is less than a preset second threshold, then abnormal data augmentation is performed again, and a random forest model is built and trained using the augmented sample dataset after the increment. If the recall rate is greater than a preset third threshold and the precision is less than a preset fourth threshold, then the data samples are cleaned, and the cleaned data samples are used for abnormal data augmentation and a random forest model is regenerated.

11. An anomaly detection system for database migration, characterized in that, include: The data sample acquisition module is used to acquire field information from the source database and the target database that includes anomaly annotations and has a mapping relationship, as data samples; The anomaly annotations include precision loss and association failure; The data augmentation module is used to perform anomalous data augmentation on the data samples by adding perturbations and using the SMOTE algorithm, thereby generating an augmented sample dataset; The model building module is used to build and train a random forest model to distinguish between normal and abnormal data based on the augmented sample dataset. The anomaly detection module is used to capture data change events during the database migration process. The random forest model is used to classify the data change events to obtain anomaly detection results.

12. An anomaly detection device for database migration, characterized in that, The system includes a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, wherein the processor, when executing the computer program, implements the database migration anomaly detection method as described in any one of claims 1 to 10.

13. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a stored computer program, wherein, when the computer program is executed, it controls the device where the computer-readable storage medium is located to perform the database migration anomaly detection method as described in any one of claims 1 to 10.

14. A computer program product, characterized in that, The computer program product includes a computer program or computer instructions, which, when executed by a processor, perform the database migration anomaly detection method as described in any one of claims 1 to 10.

Images