Regression defect relieving method for deep learning table recognition model evolution

Abstract

The invention discloses a deep learning table recognition model evolution-oriented regression defect relieving method, which comprises the following steps of: firstly, screening new features by applying a CONMI evaluation function in combination with specific hyper-parameters in a feature selection stage, and taking a new feature set obtained by combining the new features with an old feature set as model training data to realize contribution degree control-based retraining; on one hand, the feature contribution degree is calculated through the expected gradient, on the other hand, an overall loss function containing cosine similarity and a cross entropy loss function is designed to stabilize the old feature contribution degree and learn new features, and a model is trained through forward propagation and back propagation processes based on a new feature set; and under the condition that the accuracy rate of the table recognition model of the new version meets the condition, constructing a feature evolution scene, realizing a model evolution process according to a regression mitigation strategy, and taking the accuracy rate of the table recognition model of the new version as an evaluation index to obtain an evolution process regression defect mitigation strategy. The method is beneficial to improving the quality and reliability of a deep learning table recognition model.

Description

Technical Field

[0001] The present invention relates to the fields of software engineering and software testing, and in particular to designing a regression defect mitigation method for deep learning table recognition model evolution. Background Art

[0002] The related technologies of the present invention are introduced as follows:

[0003] a) Table recognition model

[0004] Given the great success of deep learning in the fields of images, audio, and text, researchers have shown great interest in extending deep learning technology to data stored in tabular format. With the advent of the big data era, the increase in data volume and the continuous iteration of user characteristics have gradually shown the limitations of traditional machine learning algorithms in processing these data. For example, when faced with a large amount of e-commerce user shopping record tabular data (such as tables containing many columns of information such as user ID, purchased product name, purchase time, purchase amount, etc.), traditional machine learning algorithms may find it difficult to accurately mine the purchase preferences of users at different consumption levels and the association rules between products. The application of deep learning technology has gradually received widespread attention, and it can better deal with such complex situations.

[0005] The deep learning table recognition model is a model that uses deep learning technology to understand the information in the table. The model can recognize the structure, content and relationship of the table, and process and analyze the table data more effectively. Tabular data presents typical structured characteristics, consisting of rows and columns arranged in order, where each row can be regarded as a sample, covering multi-dimensional information; each column represents a feature, which is used to describe the attributes of a certain dimension of the sample. Taking the e-commerce user purchase behavior analysis table as an example, each row represents a user's purchase record, which can be used as an independent sample, covering information such as the name of the purchased product, the purchase quantity, the purchase time, the payment amount, etc., and comprehensively reflects the user's consumption behavior; and each column corresponds to a feature, such as the "purchased product name" column, which is specifically used to describe the specific category of the product purchased by the sample (that is, each user). Each sample is also accompanied by a category variable, such as whether the user is a high-value customer. This type of model is mainly divided into two modules: information extraction module and classification module. The information extraction module is mainly responsible for extracting meaningful key information from the original table data, which usually involves the use of deep neural network structures, such as feedforward neural network (FNN) and convolutional neural network (CNN), to effectively capture the structure and semantic information in the table. The classification module mainly maps the extracted information to predefined categories to achieve data classification.

[0006] In actual applications, when developers train deep learning table recognition models, they first perform feature selection. Given that there may be a large number of redundant or irrelevant features in the original data, such as temporary promotional activity tags that appear by chance and have no substantial impact on user classification in the analysis of e-commerce user purchase behavior, it is necessary to use professional feature selection methods to delete them to reduce data dimensions and complexity. The model is then trained using the selected effective features and the corresponding category variables, and the trained model can finally be used for prediction.

[0007] b) Deep learning model feature evolution

[0008] In the development process of deep learning systems, in order to meet the ever-changing product requirements or fix defects reported by users, the system needs to undergo continuous evolution and update. In this process, feature-driven evolution is becoming more and more common. Given a deep learning model And the feature set used Adjust by adding, removing, or replacing existing features To obtain and by using To retrain or fine-tune the model to improve To get a new model This patent will is called the original model. called In this field, adding new features is a common practice. The introduction of new features to regress deep learning systems is called feature-driven model evolution. The definition is as follows: Given the old feature set A deep learning model trained on Then collect a set of new features The regression process is defined as Retrain and fine-tune the model to improve it Developers need to improve model performance while mitigating regression flaws introduced by evolution.

[0009] c) Regression error of deep learning table recognition model

[0010] For the classification task, given a k-class classification model and an input set Model Enter Mapped to a k-dimensional vector It contains the confidence of each class. The class with the highest confidence is represented as And regarded as the prediction result. Given a deep learning model The original version of its regression model A test input x, and its corresponding true label y. If The label of x is predicted correctly, but Prediction error, i.e. Then x triggers the regression defect. In addition, this patent defines x as the regression fault trigger input and the triggered error behavior is positioned as Given the continuous iteration and optimization of deep learning models, deep learning model regression defects are extremely hidden, so detecting and repairing deep learning regression defects is crucial to the ecological security of software systems.

[0011] d) Regression mitigation for deep learning table recognition models

[0012] In deep learning systems, regression defects have not received much attention. However, regression defect mitigation is crucial to ensure the safety and reliability of continuous iterations of deep learning systems. Regression defect mitigation can be viewed as reducing regression defects introduced by model iterations. Specifically, given an original model Its regression model and a test set Regression defect mitigation is defined in Train a new model based on Make the regression defect of the model smaller than few.

[0013] Currently, there are few studies on regression defect mitigation in deep learning systems, and only a few works focus on the field of regression defect repair in deep learning systems. However, in practical applications, these methods face several challenges: for example, model performance may degrade during retraining, post-processing techniques are not effective for out-of-distribution test inputs, and model training itself consumes a lot of time and computing resources. Therefore, how to effectively reduce newly introduced regression defects during the iteration process of the model has become a more meaningful and challenging issue.

[0014] In the feature evolution scenario, when the tabular model learns new features, it may ignore the knowledge obtained from the old features and their important contributions, thus causing regression defects (in this article, "knowledge" specifically refers to the cognitive system built by the model for the feature, that is, the contribution of the feature to the model prediction. For example, in a deep learning tabular model for predicting house prices, there are two features: house area and house construction year. If the model finds based on past training and feedback that the house area feature has a greater impact on the final prediction result than the house construction year feature in each prediction of house prices, it means that the house area feature contributes more than the house construction year, which means that the model is more inclined to use the house area feature for prediction). Secondly, if the model fails to capture effective information in the new features and over-relies on the old features, this will have an adverse effect on the generalization ability of the model and may cause regression defects. In order to alleviate these problems, ideally, the model should strike a balance between learning the knowledge of new and old features. This patent designs a table model regression defect mitigation method TRFMitigator (hereinafter "this patent" and "TRFMitigator" have the same meaning and refer to the mitigation method proposed in this patent). TRFMitigator designs a feature selection method that effectively selects new feature sets to ensure that no parts that may cause regression defects are introduced. At the same time, a reasonable memory mechanism is designed for the knowledge of old features, and an interpretability method is introduced to accurately quantify the contribution of old features to model decisions. Contribution constraints are imposed during the training process to ensure that the tabular model is not affected by the knowledge of old features, achieving robust and continuous evolution, thereby reducing the number of regression defects introduced during the evolution of deep learning tabular models. Experiments have shown that this method can effectively alleviate regression defects introduced in feature-driven model evolution scenarios.

[0015] At present, there are few studies on regression defect mitigation of deep learning tabular models, and only a few works focus on the regression defect repair of tabular models. Most of the existing methods have the following shortcomings:

[0016] 1) Significant drop in accuracy: Some methods retrain the model by exploiting the generated regression defects. Although this method can reduce the regression defects, it may also cause a significant drop in the overall accuracy of the model. This is because the retraining process may change the effective feature representation originally learned by the model, thereby affecting its generalization ability.

[0017] 2) Insufficient consideration of feature-driven evolution scenarios: Existing methods generally do not fully consider the specific scenario of feature-driven evolution. With the introduction of new features and the deepening of model dependence, the model may inadvertently reduce the focus on old features and change the weight of these features in decision making. When learning new features, the model partially or completely forgets the previously learned knowledge, which may cause serious regression defects and threaten the stability of the model.

[0018] 3) Not conducive to continuous iteration and updating of tabular models: Some methods need to integrate the prediction results of multiple versions of models (here, multiple versions of models refer to models before and after evolution) to make decisions, which not only increases the computational complexity, but also is not conducive to continuous iteration and updating of models. In the rapidly developing field of deep learning, it is crucial to flexibly adapt to new data and feature changes, and this type of method cannot meet this demand.

[0019] In practical applications, regression defects of deep learning tabular models are often discovered only after deployment. At this time, repairing them is not only very difficult, but also wastes a lot of resources. Unlike repair tasks, mitigation tasks can be performed during the model iteration process, that is, when the new model has not yet been trained, regression defects can be effectively mitigated by pre-processing data and optimizing the model training process. Therefore, this patent proposes a new method for mitigating regression defects in the evolution of deep learning tabular models to improve the stability and generalization ability of the model. Specifically, this patent proposes a regression defect mitigation method based on feature selection and contribution control. This method will introduce additional processing strategies during the training process to strictly control the changes in feature contributions to ensure that the knowledge learned by the deep learning model from the old features will not be excessively destroyed during the evolution process, thereby effectively mitigating regression defects in feature-driven model evolution scenarios. Summary of the invention

[0020] In view of the defects of the above-mentioned prior art, the present invention aims to propose a regression defect mitigation method for the evolution of deep learning table recognition models, perform feature selection on the table recognition model based on the designed deep learning framework, and realize model retraining based on contribution control by selecting a new feature set, so as to realize regression defect mitigation of model evolution according to the regression mitigation strategy in the constructed feature evolution scenario.

[0021] A regression defect mitigation method for deep learning table recognition model evolution, comprising:

[0022] S1, design a feature selection algorithm for the evolution of the table recognition model, take the newly added feature set of the user as the input of the algorithm, select features suitable for subsequent model training based on the feature forward search algorithm, remove redundant features, and obtain a new feature set;

[0023] S2, merging the obtained new feature set with the old feature set, using the merged new feature set as model training data, calculating the feature contribution of the new feature set based on the expected gradient, realizing feature retraining with contribution control, calculating the cosine similarity of the feature contribution on the new / old model, introducing a new loss function in the table model training process to control the change of feature contribution, combining the cosine similarity and the loss function to obtain the overall loss function, using the overall loss function to maintain the contribution of the old features, while learning new features, and training the model based on the new feature set through the process of forward propagation and back propagation;

[0024] S3, randomly extract the total number of features as original features, train to obtain the initial table recognition model, and then combine the remaining features as newly added features with the original features to train to obtain a new version of the table recognition model, until the accuracy of the new version of the table recognition model is higher than that of the initial table recognition model, so as to construct a feature evolution scenario;

[0025] S4, in the feature evolution scenario, the model evolution process is implemented according to the regression mitigation strategy, the accuracy of the new version of the table recognition model is used as the evaluation constraint, and the regression defect mitigation strategy of the evolution process is obtained: the lower the negative flip rate and the relative negative flip rate, the smaller the regression defect of the evolution process, and the regression defect repair is implemented according to this strategy.

[0026] In some embodiments, the feature selection suitable for subsequent model training based on the feature forward search algorithm further includes: combining the feature forward search algorithm with the Pearson correlation coefficient and the normalized mutual information to fully describe the dependency relationship between the feature and the categorical variable, wherein the categorical variable is the category to which each sample belongs; calculating the feature selection evaluation function CONMI value between the new feature set to be processed and the features of the selected feature set, performing feature dependency evaluation, selecting the new feature with the largest CONMI value to add to the selected feature set, until the CONMI values ​​of all features are less than or equal to 0 or all new features are selected; the expression of the feature selection evaluation function CONMI is as follows:

[0027]

[0028] Among them, λ represents a hyperparameter, NMI is the normalized mutual information used to measure the nonlinear correlation between two variables, and CO is the absolute value of the Pearson correlation coefficient.

[0029] In some embodiments, the introduction of a new loss function during the table model training process to control the change of feature contribution further includes: introducing a monitoring mechanism during the table model training process, that is, in each round of iteration, calculating the difference between the contribution of each feature in the new version of the model and the contribution in the old version of the model through a similarity calculation method, and converting the difference into a loss function, and guiding the model through back propagation to reduce the difference.

[0030] In some embodiments, S2 further includes retraining the model based on feature contribution control of the new feature set, calculating the cosine similarity of the feature contribution on the new / old model, and obtaining the change in the contribution of the old feature during the evolution of the feature contribution vector between different model versions.

[0031] In some embodiments, S2 further includes combining the cosine similarity and the cross entropy loss function to obtain an overall loss function, and using the overall loss function to maintain the contribution of old features while learning new features.

[0032] In some embodiments, the overall loss function The expression is as follows:

[0033]

[0034] In the formula, x is a single sample, For the old version model, For the new version model, Features In the old version model The contribution on is the true label of sample x, For sample x in the model The predicted labels on It is a cross entropy loss function commonly used in model training.

[0035] Compared with the prior art, the beneficial technical effects and technical progress achieved by the present invention are as follows:

[0036] 1) The CONMI evaluation function is used, which combines the Pearson correlation coefficient (linear) and the normalized mutual information (non-linear) to fully describe the dependency between features and categorical variables, so as to better remove irrelevant or redundant features, thereby improving model performance and accuracy;

[0037] 2) After the new feature set is fused with the original old features, in order to ensure that the contribution of the old features in the model is not affected by the new features, a new loss function is introduced, and an overall loss function including cosine similarity and cross entropy loss functions is designed to stabilize the contribution of the old features and learn the new features;

[0038] 3) The regression defect mitigation strategy for the evolution process is obtained: the lower the negative flip rate and the relative negative flip rate, the smaller the regression defect of the evolution process. According to this strategy, regression defect repair is achieved, so it performs well in reducing and repairing regression defects, which helps to improve the quality and reliability of deep learning table recognition models and has important application value in the field of deep learning table recognition;

[0039] 4) During the training process, the model can learn and optimize based on these screened high-quality features, avoiding the interference and overfitting risks caused by irrelevant or redundant features. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 It is a schematic diagram of the overall process of a regression defect mitigation method for deep learning table recognition model evolution of the present invention;

[0041] Figure 2 Design a flowchart for the loss function;

[0042] Figure 3 Schematic diagram of the performance of repairing regression defects under different distributions;

[0043] Figure 4 A schematic diagram of the implementation process of a regression defect mitigation method for deep learning table recognition model evolution according to the present invention. DETAILED DESCRIPTION

[0044] The present invention is further described in detail below in conjunction with the accompanying drawings. It should be understood that the specific implementation methods described herein are only used to explain the present invention and are not used to limit the present invention.

[0045] like Figure 1 As shown, a regression defect mitigation method for deep learning table recognition model evolution of the present invention specifically includes the following steps:

[0046] Step 1: Design an algorithm for feature selection for the evolution of the table recognition model. The algorithm uses a feature forward search algorithm to select features before training the deep learning model, remove redundant features, and obtain a new feature set. The feature set screened by the algorithm can be directly input into the table recognition model (such as a feedforward neural network (FNN) and a convolutional neural network (CNN). The input of the algorithm is a new feature set added by the user. After being processed by the algorithm, the features that are harmful to the subsequent model training will be screened out, and the beneficial features will be retained to finally obtain a feature set. Among them, the table recognition model is used to process tabular data. The tabular data consists of rows and columns arranged in order, where each row can be regarded as a sample, covering multi-dimensional information; each column represents a feature, which is used to describe the attributes of a certain dimension of the sample; the table recognition model is often a feedforward neural network or a convolutional neural network. Specifically, the algorithm is based on the feature forward search algorithm, combined with the Pearson correlation coefficient (linear) and the normalized mutual information (non-linear) to fully describe the dependency between features and categorical variables, where the categorical variable is the category to which each sample belongs. The algorithm uses the feature selection evaluation function CONMI to evaluate the dependency of features, including calculating the CONMI value between the new feature set to be processed and the features (columns in the table) of the selected feature set, selecting the new feature with the largest CONMI value to add to the selected feature set, until the CONMI values ​​of all features are less than or equal to 0 or all new features are selected. This process removes irrelevant or redundant features and obtains a new feature set; specifically, when the maximum CONMI value between the new feature set and the features in the selected feature set is greater than 0, the algorithm performs feature selection processing, calculates the CONMI value between each feature in the new feature set and the features in the selected feature set, and then selects the new feature with the largest CONMI value to add to the feature set; the hyperparameter λ used to combine the two is set to 0.5 in this method. With the gradual introduction of new features, subsequent features will not have a positive impact on the model, so this patent uses a forward search algorithm to filter out unnecessary feature subsets, and when the CONMI value of the remaining feature subset is less than 0, the screening process ends; the expression of CONMI is as follows:

[0047]

[0048] Among them, λ represents a hyperparameter, NMI is the normalized mutual information used to measure the nonlinear correlation between two variables, and CO is the absolute value of the Pearson correlation coefficient;

[0049] The hyperparameter λ is used to balance the weights of the normalized mutual information (NMI) and the absolute value of the Pearson correlation coefficient (CO) in the evaluation function, which is used to measure the linear correlation between two variables.

[0050] The feature forward search algorithm in this step specifically includes: calculating the CONMI value between the new feature set and the features in the selected feature set, selecting the new feature with the largest CONMI value to add to the selected feature set, until the CONMI values ​​of all features are less than or equal to 0 or all new features have been selected. When the maximum CONMI value between the features in the new feature set and the features in the selected feature set is greater than 0, the algorithm performs a feature selection process. The specific process is: first calculate the CONMI value between each feature of the new feature set and the features in the selected feature set, then select the new feature with the largest CONMI value, and determine whether its CONMI value is greater than 0. If it is greater than 0, add the new feature to the selected feature set, update the selected feature set, and delete the feature in the new feature set. Finally, return the selected feature set as the feature selection result. The final feature set obtained through the above strict feature selection process is input into the deep learning table recognition model for training;

[0051] Step 2: Merge the new feature set obtained in step 1 with the old feature set, use the merged new feature set as model training data, retrain the model based on feature contribution control of the new feature set, calculate the feature contribution based on the expected gradient, and introduce a new loss function in the table model training process to control the change of feature contribution: in the initialization stage, use the weight of the original table recognition model to assign weights to the new table recognition model to be trained to ensure the effective transfer of old knowledge; then, in the model training process, introduce a monitoring mechanism, that is, in each round of iteration, calculate the difference between the contribution of each feature in the new version model and the contribution in the old version model by a similarity calculation method, and convert the difference into a loss function, guide the model to reduce this difference by back propagation, and ensure that the table recognition model will not weaken the understanding of the old features when learning new features. This balanced learning strategy aims to maintain the stability of the deep learning model during the evolution process, avoid performance fluctuations or regression defects caused by the introduction of new features, and keep the deep learning model stable during the evolution process;

[0052] Step 2.1: retrain the model based on feature contribution control of the new feature set. The specific process is as follows:

[0053] Accurately quantify the contribution of old features to model predictions and ensure that they remain stable during model updates. To this end, advanced interpretability methods are used to help users understand the internal working principles of the model. This method conducts in-depth analysis of the model's internal structure to explore the degree of dependence of the model's prediction results on each feature, and further converts the calculated degree of dependence of the model on each feature into a quantifiable contribution indicator;

[0054] Specifically, the feature contribution calculation based on the expected gradient is realized. The specific process is as follows:

[0055] A feature contribution calculation method based on expected gradient is adopted: during the model training process, the gradient of each feature relative to the model output is calculated one by one through the back-propagation algorithm and the chain rule. These gradient values ​​are processed by expectation, and finally the expected gradient of each feature contribution is obtained. The expected gradient can not only intuitively reflect the degree of dependence of the model on each feature, but also serve as an effective indicator for evaluating feature contribution. The larger the expected gradient value of the feature, the more significant its role in the decision-making of the deep learning table model. Therefore, the differentiability of the deep learning model is fully utilized and the back-propagation mechanism is effectively combined;

[0056] Step 2.2, calculate the cosine similarity of the contribution of the feature in the new and old models, and obtain the change of the contribution of the old feature during the evolution of the feature contribution vector between different model versions. Specifically, cosine similarity is often used to measure the similarity between two vectors. It has high computational efficiency, is not affected by vector length, and has strong versatility. The higher the cosine similarity, the closer the contribution of the feature between the new and old versions of the model, and the better its stability. By applying cosine similarity, the model can be effectively constrained to maintain the stability of the features during the evolution process;

[0057] Step 2.3: Introduce the original cross entropy loss function to control the change of feature contribution, ensure that the model can learn new features during the evolution process, and improve the overall performance. Using the obtained new feature set, the algorithm can train the model through the process of forward propagation and back propagation, and continuously iterate and update the weight parameters through back propagation on the training set until the gap between the model's predicted results and the actual results on the test set tends to stabilize.

[0058] The design of the cross entropy loss function is based on the constraint that the contribution of old features to model prediction remains stable during training. Two core requirements are considered: computational efficiency and function differentiability. Efficiency ensures that the loss value can be calculated quickly during model training to avoid unnecessary computational burden. Differentiability ensures that model parameters can be effectively adjusted and optimized through methods such as gradient descent.

[0059] like Figure 2 As shown in the figure, the design process of the loss function. In order to ensure that the contribution of the old features in the prediction process does not deviate significantly, TRFMitigator designed a new loss function The expression is as follows:

[0060]

[0061] In the formula, x is a single sample, For the old version model, For the new version model, Features In the old version model The contribution on is the true label of sample x, For sample x in the model The predicted labels on . It represents the cross entropy loss function commonly used in model training. This loss function can ensure that the model learns relevant knowledge of new features and improve the overall accuracy.

[0062] Cosine similarity is used as a measurement indicator to ensure that the contribution vector of the old features remains as unchanged as possible during the model evolution process. The higher the cosine similarity, the closer the contribution of the old features between different versions of the model. By applying cosine similarity, this loss function can effectively constrain the deep learning model to maintain the stability of the contribution of the old features during the evolution process and avoid excessive changes. In order to ensure that the deep learning model can learn new features while maintaining the contribution of the old features, the original cross entropy loss function is also introduced Combining the cosine similarity constraint and the cross entropy loss function, the overall loss function is obtained. This design aims to ensure that the model can maintain effective use of old features and efficiently learn new features during the evolution process, thereby improving the overall performance of the model.

[0063] Step 3: Construct a feature evolution scenario, specifically including: first, randomly extract 1-a% of the total number of features as original features, and train the initial table recognition model Then the remaining a% features are taken as newly added features and combined with the original features to train a new version of the table recognition model. Require The accuracy is higher than To ensure the validity of the feature evolution scenario; otherwise, the scenario will be discarded and the features will be reselected, and the above process will be repeated;

[0064] Step 4: Implement the model evolution process based on the regression mitigation strategy, including: The accuracy is used as the evaluation index, denoted as In order to measure the effectiveness of regression defect mitigation, two indicators are used, namely negative flip rate (NFR) and relative negative flip rate (NFR). rel The negative flip rate (NFR) is used to measure the number of regression failures caused during model evolution. NFR represents the proportion of test inputs that trigger regression failures in the test set. relThe lower the value of, the fewer regression failures are caused in the regression process, which means that the model update or improvement leads to fewer misclassifications of previously correctly classified samples, thus proving the effectiveness of the regression mitigation strategy.

[0065] The specific implementation details are as follows:

[0066] In step 1, high-dimensional data usually contains a lot of noise and irrelevant information, which may cause the trained model to overfit and perform poorly on new data. Feature selection is to select the most valuable feature subset for model construction and prediction from the many features of the original data. For example, in an e-commerce data set containing many user behavior features, if there are some features that are not substantially related to user purchase decisions, such as the time a user stays on an irrelevant page, retaining these features may interfere with the model's learning of truly important factors, causing the model to deviate when predicting user purchase behavior. After removing them through feature selection, the model can more accurately capture the relationship between key features such as user historical purchase records and browsed product categories and purchase behavior, thereby improving the accuracy of prediction. TRFMitigator uses feature selection technology to reduce the dimension of the feature space and remove irrelevant or redundant features, thereby improving model performance and accuracy. This technology uses the CONMI evaluation function as the evaluation criterion for feature selection. The CONMI function combines the Pearson correlation coefficient (linear) and the normalized mutual information (non-linear) to fully describe the dependency between features and categorical variables.

[0067] In step 2, the contribution calculation function is selected: TRFMitigator uses the expected gradient scheme to calculate the contribution of the feature. The calculated feature contribution can be used to measure the importance of the feature for model prediction. This method enhances the interpretability of deep learning models by integrating axiomatic attribution priors, extends the integrated gradients method, eliminates the need for predefined baseline inputs, and thus solves many key limitations of existing attribution techniques. By integrating over a dataset rather than relying on a single baseline, the expected gradient provides a more comprehensive and generalized method to understand the contribution of features under different inputs. At the same time, the method follows important explanatory axioms such as completeness and implementation invariance, ensuring that the attribution method is both comprehensive and applicable to different model implementations. This feature is crucial to achieving high-quality attribution, and it accurately reflects the importance of input features in model prediction. The value of the expected gradient can clearly rank the importance of different features. In a complex deep learning tabular model, there may be multiple input features. For example, in a model that analyzes the purchase behavior of e-commerce users, there are many features such as user age, purchase frequency, and the number of browsed product categories. The expected gradient can quantify the influence of each feature on the final purchase decision, thereby determining which features play a key role in the model decision-making process and which are relatively minor.

[0068] In order to verify the effectiveness of the present invention, the experimental results are scientifically analyzed. The present invention proposes a novel deep learning table recognition model regression defect mitigation method (TRFMitigator), all implementations are implemented in Python language, using TensorFlow 2.3.0 as the front-end high-level deep learning library. This patent is implemented on an Intel(R) Xeon(R) machine with 125Gb RAM, the CPU model is E5-2640 v4, with 40 cores and a frequency of 2.40GHz, and the operating system used by the machine is Ubuntu 18.04.

[0069] The present invention analyzes the data sets of 14 evolution scenarios and their corresponding deep learning models as experimental objects. These data sets cover a variety of application scenarios and task types to ensure the wide applicability of the experimental results. Since the data of feature evolution scenarios are relatively common but not open source in industrial applications, a series of open source data sets are selected to simulate the feature evolution process. Specifically, a total of 14 scenarios are constructed in this paper, of which 12 are from manually constructed regression scenarios (ID: 1-12) and 2 are from real scenarios (ID: 13-14). All models and data sets used are listed in Table 1. Including ID number, data set name and its abbreviation, model structure, training set, validation set and test set size, number of categories, accuracy of each model version and number of features used in each model version. These data sets have been widely used in previous deep learning table model tasks, covering multiple fields such as mobile phone price range prediction and disease diagnosis. In order to ensure the wide use of data sets, data sets with less than 50 likes on Kaggle or less than 10 stars on GitHub were filtered out. To prevent the model from overfitting, datasets with less than 500 samples and less than 8 features are excluded. In existing open source datasets, data publishers usually provide a complete set of features that are comprehensively and carefully collected, rather than features collected in previous stages. Therefore, in the experiment, in order to simulate the feature evolution process, a% of the total features are removed, and these removed features are regarded as newly added features. Then, the remaining 1-a% features are used as the original features of the previous version of the training model. Specifically, the feature evolution scenario is constructed by following the steps below: First, 1-a% of the total number of features are randomly extracted as the original features, and the initial model is trained Then, the remaining a% features are used as newly added features and combined with the original features to train the model. Require The accuracy is higher than To ensure the validity of the feature evolution scenario; otherwise, the scenario will be discarded and the features will be reselected, and the above process will be repeated.

[0070] In the process of model evolution, it is crucial to balance the improvement of model accuracy and the number of regression defects introduced. Therefore, evaluation indicators were selected from existing work, including one indicator for measuring accuracy improvement and three indicators for measuring regression defects. Accuracy is often used to quantify model performance. The accuracy is used as the evaluation index, denoted as In order to measure the effectiveness of regression defect mitigation, two indicators widely used in existing research are adopted: negative flip rate (NFR) and the relative negative flip rate (NFR). rel), these indicators are used to measure the number of regression faults caused during the model evolution process. NFR represents the proportion of test inputs that trigger regression faults in the test set. Since the error rate (1-ACC) is the upper limit of NFR, it becomes challenging to compare regressions between models with different error rates. To this end, this paper introduces NFR rel , whose denominator reflects the The expected error rate on the correctly predicted sample subset is used to measure the proportion of incorrect predictions caused by regression. NFR and NFR rel The lower the value of, the fewer regression faults are caused in the regression process, which means that the model update or improvement leads to fewer misclassifications of previously correctly classified samples, thus proving the effectiveness of the regression mitigation strategy. The present invention further analyzes the performance of the regression defect mitigation method in processing test inputs that trigger regression faults under different data distributions and regression defects in the test set. And its regression model Generate test inputs that trigger regression faults. Then, evaluate whether the models obtained by different repair methods can correctly predict these inputs. The accuracy of the generated test inputs is denoted as Fix. This indicator can be regarded as the accuracy of the regression model for the generated inputs. Table 1 shows the test data set and model information.

[0071] Table 1

[0072]

[0073] The dataset size is recorded in the form of (a / b / c), where a / b / c represent the size of the training set / validation set / test set, respectively.

[0074] By comparing the proposed method (TRFMitigator) with the most advanced regression defect repair methods, the effectiveness of TRFMitigator in 14 different scenarios was evaluated. Since there are significant differences between the ensemble method and other methods in essence: the ensemble method estimates uncertainty through the prediction results of multiple models, providing more reliable predictions, but with higher computational complexity and not conducive to continuous iteration of the model; while other methods directly repair the original regression model, with lower computational complexity, but may not be as accurate or reliable as the ensemble method in some cases. Therefore, in order to more comprehensively evaluate the performance of TRFMitigator, the present invention evaluates the ensemble method and other methods separately.

[0075] a) Comparison of the method of the present invention TRFMitigator with non-integrated methods

[0076] In 14 scenarios, TRFMitigator achieved the best accuracy improvement in 78.5% (11 / 14) of the scenarios, Finetuning performed best in only 7.14% (1 / 14) of the scenarios, and NeuRecover and FS did not achieve the best results in 14 scenarios. At the same time, in all scenarios, TRFMitigator significantly outperformed all the comparison methods in terms of NFR. Compared with DT, FS, Finetuning and NeuRecover, TRFMitigator reduced NFR by 89.64%, 83.89%, 95.71% and 125.23%, respectively. Experimental results show that TRFMitigator can maintain accuracy while mitigating regression failures. At the same time, the effectiveness of TRFTitigator in repairing regression faults under different distributions is evaluated. In all scenarios, TRFTitigator is significantly better than other comparison methods in terms of Fix, with an average repair rate of 84.60%. Compared with FS, Finetuning and NeuRecover, TRFTitigator improves by 19.43%, 68.95% and 200.98% respectively.

[0077] This paper also evaluates the effectiveness of TRFTitigator in repairing regression faults under different distributions. In all scenarios, TRFTitigator significantly outperforms other comparison methods in terms of Fix, with an average repair rate of 84.60%. Compared with FS, Finetuning, and NeuRecover, TRFTitigator improves by 19.43%, 68.95%, and 200.98%, respectively. Specifically, Figure 3 As shown in scenarios 2 and 11 in , the X-axis represents the confidence distribution of regression defects on the model obtained by direct training, and the Y-axis represents the repair rate of these inputs. It can be seen that the repair rate of TRFTitigator in different confidence intervals is higher than that of other methods, which shows that TRFTitigator can stably repair regression defects with various confidence ranges. This is mainly because TRFTitigator solves the root cause of regression defects in feature evolution scenarios and enhances the robustness of the model. In contrast, the NeuRecover method can only repair regression defects near the decision boundary, which shows that its performance is limited when encountering regression defects that deviate from the training data. The above experimental results show that \tech{} is significantly effective in repairing regression faults under different distributions.

[0078] b) Comparison between TRFMitigator and non-integrated methods

[0079] The average results of the proposed method TRFMitigator and three advanced ensemble methods on 14 scenarios are shown in Table 3. Among the 14 scenarios, TRFMitigator achieved the best accuracy improvement on 57.14% (8 / 14) of the scenarios, and Dropout and Scaling achieved the best results on 7.14% (1 / 14) and 21.42% (4 / 14) of the scenarios, respectively. At the same time, TRFMitigator achieved the lowest NFR on 78.57% (11 / 14) of the scenarios, and compared with Dropout, Perturbation and Scaling, TRFMitigator reduced the NFR by 53.59%, 49.80% and 21.77%, respectively. The repair rate of TRFMitigator in different confidence intervals is higher than that of other methods, which shows that TRFMitigator can stably repair regression defects with various confidence ranges. In contrast, the ensemble processing method can only repair regression defects near the decision boundary, which is mainly because the ensemble model is susceptible to the overconfident decision of a single model. The above experimental results demonstrate the effectiveness of TRFMitigator.

[0080] It should be noted that, although the present invention has been shown and described with reference to specific exemplary embodiments of the present invention, those skilled in the art should understand that the present invention is not limited to the above-mentioned embodiments and all kinds of changes to the present invention fall within the scope of protection of the present invention.

Claims

1. A regression defect mitigation method for deep learning table recognition model evolution, characterized in that: include: S1, design a feature selection algorithm for the evolution of the table recognition model, take the newly added feature set of the user as the input of the algorithm, select features suitable for subsequent model training based on the feature forward search algorithm, remove redundant features, and obtain a new feature set; S2, merging the obtained new feature set with the old feature set, using the merged new feature set as model training data, calculating the feature contribution of the new feature set based on the expected gradient, realizing feature retraining with contribution control, calculating the cosine similarity of the feature contribution on the new / old model, introducing a new loss function in the table model training process to control the change of feature contribution, combining the cosine similarity and the loss function to obtain the overall loss function, using the overall loss function to maintain the contribution of the old features, while learning new features, and training the model based on the new feature set through the process of forward propagation and back propagation; S3, randomly extract the total number of features as the original features, train to obtain the initial table recognition model, and then combine the remaining features as newly added features with the original features to train to obtain a new version of the table recognition model, until the accuracy of the new version of the table recognition model is higher than that of the initial table recognition model, and then construct a feature evolution scenario; S4, in the feature evolution scenario, the model evolution process is implemented according to the regression mitigation strategy, the accuracy of the new version of the table recognition model is used as the evaluation constraint, and the regression defect mitigation strategy of the evolution process is obtained: the lower the negative flip rate and the relative negative flip rate, the smaller the regression defect of the evolution process, and the regression defect repair is implemented according to this strategy.

2. The method for mitigating regression defects in deep learning table recognition model evolution according to claim 1, characterized in that: The feature selection suitable for subsequent model training based on the feature forward search algorithm further includes: combining the feature forward search algorithm with the Pearson correlation coefficient and the normalized mutual information to fully describe the dependency relationship between the feature and the category variable, wherein the category variable is the category to which each sample belongs; calculating the feature selection evaluation function CONMI value between the new feature set to be processed and the features of the selected feature set, performing feature dependency evaluation, selecting the new feature with the largest CONMI value to add to the selected feature set, until the CONMI values ​​of all features are less than or equal to 0 or all new features are selected; the expression of the feature selection evaluation function CONMI is as follows: Among them, λ represents a hyperparameter, NMI is the normalized mutual information used to measure the nonlinear correlation between two variables, and CO is the absolute value of the Pearson correlation coefficient.

3. The regression defect mitigation method for deep learning table recognition model evolution according to claim 1, characterized in that: The introduction of a new loss function during the table model training process to control the change of feature contribution further includes: introducing a monitoring mechanism during the table model training process, that is, in each round of iteration, calculating the difference between the contribution of each feature in the new version model and the contribution in the old version model through a similarity calculation method, and converting the difference into a loss function, and guiding the model through back propagation to reduce the difference.

4. The regression defect mitigation method for deep learning table recognition model evolution according to claim 1, characterized in that: The S2 further includes retraining the model based on feature contribution control of the new feature set, calculating the cosine similarity of the feature contribution on the new / old model, and obtaining the change of the old feature contribution during the evolution of the feature contribution vector between different model versions.

5. The regression defect mitigation method for deep learning table recognition model evolution according to claim 1, characterized in that: The overall loss function The expression is as follows: In the formula, x is a single sample, For the old version model, For the new version model, Features In the old version model The contribution on is the true label of sample x, For sample x in the model The predicted labels on It is a cross entropy loss function commonly used in model training.

Images