Deep learning test sample selection method and system based on feature distribution analysis

Abstract

The present invention discloses a method and system for selecting deep learning test samples based on feature distribution analysis. The method comprises: using a clustering method to divide the feature distribution of training samples to obtain feature distribution clusters of training samples; calculating the feature difference value of each training sample / test sample distributed on each feature distribution cluster based on the feature distribution clusters of training samples; calculating the feature dispersion of training samples based on the output vectors of training samples; constructing a sorting model based on a sorting learning algorithm, and realizing the prediction and sorting of test samples; generating a synthetic test set, sorting the synthetic test set using the sorting model, setting a sampling ratio, and selecting test samples with the highest sorting to form a test subset. The method of the present invention screens out test inputs that can quickly and fully detect DNN model faults under limited resources, thereby alleviating the impact of data distribution changes on DNN model accuracy while reducing test labeling costs and improving DNN model testing efficiency.

Description

Technical Field

[0001] The present invention belongs to the field of deep learning test technology, and specifically relates to a deep learning test sample selection method and system based on feature distribution analysis. Background Art

[0002] With the rapid development of deep learning (DL) technology, deep neural network (DNN) models trained on large amounts of data have been widely used in a variety of fields, including autonomous driving, facial recognition, speech recognition, medical diagnosis, aircraft collision avoidance systems, and software engineering. While DNNs have achieved significant success, their quality issues have also led to numerous safety incidents. In practice, DNNs are mostly tested on a test set drawn from the same dataset as the training set, with a consistent data distribution. However, when DNNs are deployed in real-world scenarios, new test data is continuously generated over time, and the data distribution becomes increasingly diverse. As the data distribution shifts, the model's effectiveness in the test environment decreases.

[0003] Therefore, in order to ensure that the model can adapt to data with different distributions in the new environment, in DNN testing, testers retrain the original model by collecting new unlabeled test data, thereby effectively detecting DNN model failures while updating the model's weight parameters, further improving the model quality to adapt to the new data distribution.

[0004] However, a large amount of test data needs to be correctly labeled before it can be put into testing, and manual labeling is currently the main method. In order to ensure the correctness of the labeling, multiple users are usually required to complete the labeling task collaboratively. Some test data involving professional fields (such as images, pictures and text from the medical field) requires domain experts to label, which results in a large amount of labeling costs and seriously affects the efficiency of DNN model testing. Therefore, maintaining a small-scale test set with good testing capabilities has become an important research issue in the current field of deep learning testing. The goal of deep learning test selection methods is to select representative test input data from the original unlabeled test set to form a test subset, and then label the test subset to reduce the overall labeling and test execution costs.

[0005] To improve the efficiency of DNN model testing, many test selection methods have been proposed. Shen et al. proposed a test selection method called MCP. This method clusters test samples into multiple boundary regions based on the maximum and second-highest confidence levels of the DNN model's predictions for the test input. Prioritizing the selection of test samples uniformly from all boundary regions to form a test subset, this method guides DNN model retraining and improves DNN model quality.

[0006] However, the problems with the existing technology are:

[0007] 1) Existing methods do not consider the problem of data distribution changes and cannot effectively select test sample data with different distributions from the training samples to retrain the model, thereby improving the model's ability to adapt to the new data distribution.

[0008] 2) The test subsets screened by existing methods fail to ensure the diversity of their test samples and lack test adequacy. Summary of the Invention

[0009] In view of the above-mentioned deficiencies in the prior art, the purpose of the present invention is to provide a deep learning test sample selection method and system based on feature distribution analysis to solve the problems of insufficient testing and unknown data distribution changes in the test subsets selected in the prior art; the method of the present invention screens out test inputs that can quickly and fully detect DNN model faults under limited resources, so as to alleviate the impact of data distribution changes on the accuracy of the DNN model, reduce test labeling costs, and improve DNN testing efficiency.

[0010] In order to achieve the above object, the technical solution adopted by the present invention is as follows:

[0011] The present invention provides a method for selecting deep learning test samples based on feature distribution analysis, comprising the following steps:

[0012] 1) Use clustering method to divide the feature distribution of training samples and obtain the feature distribution clusters of training samples;

[0013] 2) Based on the feature distribution clusters of the training samples obtained in step 1), calculate the feature difference value of each training sample / test sample distributed in each feature distribution cluster;

[0014] 3) Calculate the feature dispersion of the training sample based on the output vector of the training sample;

[0015] 4) Build a ranking model based on the ranking learning algorithm and realize the prediction and ranking of test samples;

[0016] 5) Use the adversarial attack method to generate a synthetic test set, use the sorting model to sort the synthetic test set, set the sampling ratio, and select the top-ranked test samples to form a test subset.

[0017] Furthermore, the step 1) specifically includes: inputting all training samples into a DNN model for prediction to obtain respective output vectors, i.e., feature vectors of all training samples; clustering the training samples based on the feature vectors using the K-means clustering algorithm to divide the feature distributions of different categories of the training samples on the corresponding DNN model to obtain multiple feature distribution clusters, each feature distribution cluster represents a class of feature distribution, and the training samples under the same feature distribution cluster have the same class of feature distribution.

[0018] Furthermore, the step 2) specifically includes:

[0019] Calculation of feature difference value (FDF): Given a sample set S of size m (the sample set includes a training subset R and a test set T), a model to be tested D, and w feature distribution clusters of the training samples obtained by the K-means clustering algorithm on the model to be tested D, the set of feature distribution clusters is C = {c1, c2, ..., c w}; Let s h is the hth sample in the sample set S, c j is the jth cluster, according to the MMD-critic algorithm from c j Select k prototypes to form the set X j ={x j，1 , x j，2 ,...,x j，k},x j，p For the set X j The p-th prototype in; according to sample s h The characteristic vector (ie output vector) of , calculate the sample s h With set X j The distance of all prototypes in , the average of all distances is taken as sample s h Distributed in cluster c j The characteristic difference value FDF h，j ; The calculation is as follows:

[0020]

[0021] Then sample s h Distributed in cluster set C = {c1, c2, ..., c w The set of characteristic difference values ​​on} is FDF h ={FDF h，1 , FDF h，2 ,...,FDF h，w}.

[0022] Furthermore, the step 3) specifically includes:

[0023] Calculation of feature dispersion (FDS): Given a training set R and a model to be tested D, the number of categories in the training set R is n. Assume that the predicted probability vector of a training sample r (r∈R) on the model to be tested D is P(r) = <p r，1 , p r，2 ,...,p r，n >, p r，n Indicates the probability value of the training sample r being predicted as the nth category; assuming that the true category of the training sample r is the tth category, according to the predicted probability p on the tth category r，t Construct a reference vector P′ of length n (r) = <p r，t , p r，t ,...,p r，t >,p r，t Indicates the probability value of training sample r being predicted as the t-th category; the feature dispersion (FDS) of training sample r is calculated as follows:

[0024]

[0025] The higher the feature dispersion FDS value of the training sample, the more dispersed its feature distribution.

[0026] Furthermore, the step 4) specifically includes:

[0027] 41) Construction of new training set: The new training set is denoted as FP = {fp 1，2 , fp 2，3 ,...,fp n-1，n}, fp n-1，n Represents the original training sample r n-1 and r n A feature pair consisting of the feature difference value of i and r j A feature pair fp consisting of the feature difference value p，i , denoted as fp p，i =<FDF i , FDF j >; This feature has an effect on fp p，i label i，j The value is -1, 0 or 1; label label i，j From the original training sample r i and r j The relative size of the feature dispersion is determined by the label value; the value of the label reflects the relative size of the training model ability between the original training samples in the feature pair; when FDF i >FDF j When, label i，j =1, indicating that the original training sample r iThe feature of the original training sample r j The feature distribution is dispersed and the ability to train the model is strong; when FDF i =FDF j When label i，j =0, indicating that the original training sample r i The feature distribution and original training sample r j The dispersion of the feature distribution is the same, and the ability to train the model is the same; when FDF i <FDF j When label i，i =-1, indicating that the original training sample r j The feature of the original training sample r i The feature distribution is dispersed, and the training model has strong capabilities. The obtained new training set FP is input into the xgboost sorting algorithm. The algorithm uses a tree ensemble model and sets the number of generated trees to m. New trees are added by continuously splitting the basic features of the training samples to learn more complex features from the training set. At the same time, the prediction residuals are continuously fitted to minimize the training error. After generating the set number of m trees, the sorting model is constructed.

[0028] 42) Construction of new test set: construct new test samples by extracting the feature difference value set of the test samples; assuming a test sample t i The characteristic difference value set is FDF i , then according to the test sample t i The new test sample constructed is a feature pair containing only one feature difference value set, that is, <FDF i >

[0029] 43) Prediction of test samples: Input a new test sample into the constructed ranking model, add up the scores of each feature of the new test sample at the corresponding leaf nodes on each tree in the ranking model, and obtain the predicted score of each test sample. The predicted score reflects the correlation between the test sample and revealing model errors; the larger the predicted score, the easier it is for the test sample to reveal model errors; based on the size of the predicted score, the test samples with large predicted scores are prioritized to achieve ranking of the test samples.

[0030] Furthermore, the step 5) specifically includes: using five adversarial attack methods, namely FGSM, BIM-A, BIM-B, CW and JSMA, to perturb each test sample in the test set to generate an adversarial sample set, randomly selecting 20% ​​of the adversarial samples from the adversarial sample set, and selecting 80% of the test samples from the test set to form a synthetic test set with different data distributions; using the sorting model to sort each synthetic test set, setting the sampling ratio, and selecting the test samples ranked first from the sorted synthetic test set according to the ratio to form a test subset to complete the selection of test samples.

[0031] The present invention also provides a deep learning test sample selection system based on feature distribution analysis, comprising:

[0032] The feature distribution partitioning module is used to partition the feature distribution of the training samples using a clustering method to obtain feature distribution clusters of the training samples;

[0033] A feature difference value calculation module is used to calculate the feature difference value of each training sample / test sample distributed on each feature distribution cluster based on the feature distribution cluster of the training samples;

[0034] A feature dispersion calculation module is used to calculate the feature dispersion of the training sample according to the output vector of the training sample;

[0035] The sorting model building module builds a sorting model based on the sorting learning algorithm and realizes the prediction and sorting of test samples;

[0036] A generation module for generating synthetic test sets using adversarial attack methods;

[0037] The test sample selection module is used to sort the synthetic test set using the sorting model, set the sampling ratio, and select the test samples with the highest sorting to form the test subset.

[0038] The present invention also provides a selection terminal, comprising:

[0039] one or more processors;

[0040] a memory for storing one or more programs;

[0041] When the one or more programs are executed by the one or more processors, the one or more processors implement the deep learning test sample selection method based on feature distribution analysis.

[0042] The present invention also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the deep learning test sample selection method based on feature distribution analysis.

[0043] Beneficial effects of the present invention:

[0044] The present invention has effective error detection capabilities: Based on the analysis of sample feature distribution, the present invention utilizes learning to rank to construct a ranking model by learning the relationship between the feature distribution information of training samples and the ability of the training model to predict the correlation between the feature distribution information of test samples and the ability to reveal model errors, thereby obtaining a set of correlation scores. The size of the score reflects the ability of the test sample to reveal errors. By sorting and selecting test samples according to the scores, effective test cases can be quickly found for labeling, faults in the DNN model can be detected, the labeling cost can be greatly reduced, and the efficiency of DNN testing can be improved.

[0045] This method mitigates data distribution shifts by measuring the differences in the test sample's feature distributions between those of the training samples in each category. Based on these differences, the model is retrained using test samples with significantly different distributions. This allows the retrained model to effectively adapt to the new data distribution, mitigating the impact of data distribution shifts on model accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 Schematic diagram of the method of the present invention.

[0047] Figure 2 This is a schematic diagram of clustering training samples in the present invention.

[0048] Figure 3 Schematic diagram of extracting feature difference values ​​in the present invention.

[0049] Figure 4 Schematic diagram of calculating feature dispersion in the present invention. DETAILED DESCRIPTION

[0050] In order to facilitate understanding by those skilled in the art, the present invention will be further described below with reference to embodiments and drawings. The contents mentioned in the embodiments are not intended to limit the present invention.

[0051] Reference Figure 1 As shown, the present invention provides a method for selecting deep learning test samples based on feature distribution analysis, and the steps are as follows:

[0052] 1) Use clustering method to divide the characteristic distribution of training samples and obtain characteristic distribution clusters of training samples; Figure 2As shown in FIG, specifically, the method includes: inputting all training samples (regardless of their labels) into a DNN model for prediction to obtain respective output vectors, i.e., feature vectors of all training samples; clustering the training samples according to the feature vectors using the K-means clustering algorithm to divide the feature distributions of different categories of the training samples on the corresponding DNN model to obtain multiple feature distribution clusters, each of which represents a type of feature distribution. The training samples under the same feature distribution cluster have the same type of feature distribution.

[0053] 2) According to the feature distribution clusters of the training samples obtained in step 1), calculate the feature difference value (FDF) of each training sample / test sample distributed on each feature distribution cluster; refer to Figure 3 As shown, specifically including:

[0054] Calculation of feature difference value (FDF): Given a sample set S of size m (the sample set includes a training subset R and a test set T), a model to be tested D, and w feature distribution clusters of the training samples obtained by the K-means clustering algorithm on the model to be tested D, the set of feature distribution clusters is C = {c1, c2, ..., c w}; Let s h is the hth sample in the sample set S, c j is the jth cluster, according to the MMD-critic algorithm from c j Select k prototypes to form the set X j ={x j，1 , x j，2 ,...,x j，k},x j，p For the set X j The p-th prototype in; according to sample s h The characteristic vector (ie output vector) of , calculate the sample s h With set X j The distance of all prototypes in , the average of all distances is taken as sample s h Distributed in cluster c j The characteristic difference value FDF h，j ; The calculation is as follows:

[0055]

[0056] Then sample s h Distributed in cluster set C = {c1, c2, ..., c w The set of characteristic difference values ​​on} is FDF h ={FDF h，1 , FDF h，2 ,...,FDF h，w}.

[0057] 3) Calculate the feature dispersion (FDS) of the training sample based on the output vector of the training sample; refer to Figure 4 As shown, specifically including:

[0058] Calculation of feature dispersion (FDS): Given a training set R and a model to be tested D, the number of categories in the training set R is n. Assume that the predicted probability vector of a training sample r (r∈R) on the model to be tested D is P(r) = <p r，1 , p r，2 ,...,p r，n >, p r，n Indicates the probability value of the training sample r being predicted as the nth category; assuming that the true category of the training sample r is the tth category, according to the predicted probability p on the tth category r，t Construct a reference vector P′ of length n (r) = <p r，t , p r，t ,...,p r，t >,p r，t Indicates the probability value of training sample r being predicted as the t-th category; the feature dispersion (FDS) of training sample r is calculated as follows:

[0059]

[0060] The higher the feature dispersion FDS value of the training sample, the more dispersed its feature distribution.

[0061] 4) Build a ranking model based on the learning to rank algorithm and implement prediction and ranking of test samples; specifically,

[0062] 41) Construction of new training set: The new training set is denoted as FP = {fp 1，2 , fp 2，3 ,...,fp n-1，n}, fp n-1，n Represents the original training sample r n-1 and r n A feature pair consisting of the feature difference value of i and r j A feature pair fp consisting of the feature difference value p，i , denoted as fp p，i =<FDF i , FDF j >; This feature has an effect on fp p，i label i，j The value is -1, 0 or 1; label label i，j From the original training sample r i and r jThe relative size of the feature dispersion is determined by the label value; the value of the label reflects the relative size of the training model ability between the original training samples in the feature pair; when FDF i >FDF j When, label i，j =1, indicating that the original training sample r i The feature of the original training sample r j The feature distribution is dispersed and the ability to train the model is strong; when FDF i =FDF j When label i，j =0, indicating that the original training sample r i The feature distribution and original training sample r j The dispersion of the feature distribution is the same, and the ability to train the model is the same; when FDF i <FDF j When label i，j =-1, indicating that the original training sample r j The feature of the original training sample r i The feature distribution is dispersed, and the training model has strong capabilities. The obtained new training set FP is input into the xgboost sorting algorithm. The algorithm uses a tree ensemble model and sets the number of generated trees to m. New trees are added by continuously splitting the basic features of the training samples to learn more complex features from the training set. At the same time, the prediction residuals are continuously fitted to minimize the training error. After generating the set number of m trees, the sorting model is constructed.

[0063] 42) Construction of new test set: construct new test samples by extracting the feature difference value set of the test samples; assuming a test sample t i The characteristic difference value set is FDF i , then according to the test sample t i The new test sample constructed is a feature pair containing only one feature difference value set, that is, <FDF i >

[0064] 43) Prediction of test samples: Input a new test sample into the constructed ranking model, add up the scores of each feature of the new test sample at the corresponding leaf nodes on each tree in the ranking model, and obtain the predicted score of each test sample. The predicted score reflects the correlation between the test sample and revealing model errors; the larger the predicted score, the easier it is for the test sample to reveal model errors; based on the size of the predicted score, the test samples with large predicted scores are prioritized to achieve ranking of the test samples.

[0065] 5) Use the adversarial attack method to generate a synthetic test set, use the sorting model to sort the synthetic test set, set the sampling ratio, and select the test samples with the highest ranking to form a test subset; specifically, use the five adversarial attack methods of FGSM, BIM-A, BIM-B, CW and JSMA to perturb each test sample in the test set to generate an adversarial sample set, randomly select 20% of the adversarial samples from the adversarial sample set, and select 80% of the test samples from the test set to form a synthetic test set with different data distributions; use the sorting model to sort each synthetic test set, set the sampling ratio, and select the test samples with the highest ranking from the sorted synthetic test set according to the ratio to form a test subset to complete the selection of test samples.

[0066] In the example:

[0067] Select the more mature image dataset CIFAR10 and DNN network model VGG-16. Use the training set of CIFAR10 to train ResNet-20 and obtain a trained model (abbreviated as L). Use five adversarial attack methods FGSM, BIM-A, BIM-B, CW, and JSMA to slightly perturb the test set of CIFAR10, and synthesize 20% of the perturbed images and 80% of the original images into a new test set. The method of the present invention is used to sample 1% of the synthetic test set (10,000 test images), that is, 1,000 valid test samples are selected from the test set to form a small-scale test subset, so as to fully and effectively detect the faults of model L, improve the model accuracy, and effectively alleviate the impact of data distribution transfer on model quality. The specific implementation process is as follows:

[0068] 1. Set the number of clusters k = 10, use the K-means clustering algorithm to cluster the CIFAR10 training samples, and obtain 10 different feature distribution clusters C = {c1, c2, ..., c 10}.

[0069] 2. 1% of the training samples from the CIFAR10 training set are selected to form a training subset R. These selected training samples are fed into the model under test to obtain the output vector, i.e., the feature vector of the training sample. A benchmark vector is constructed based on the predicted value of the training sample's true category. The feature dispersion (FDS) of the training sample is calculated using the feature vector and the benchmark vector.

[0070] 3. Using the MMD-critic algorithm, select 5 prototypes from each feature distribution cluster to approximately represent the entire cluster. Given the training subset R and the synthetic test set T, we get the sample set S (S = R∪T); given the hth sample s h (s h ∈S) and a cluster c j, according to s h The output vector on the model under test L, calculate s h with c j The Euclidean distance of the five prototypes. The average of all Euclidean distances is taken as the sample s h Distributed in cluster c j The characteristic difference value FDF on h，j . And so on, s h The set of feature difference values ​​distributed on 10 feature clusters is FDF h ={FDF h，1 , FDF h，2 ,...,FDF h，10}.

[0071] 4. Based on the feature difference value set of any two training samples in the training subset R, construct a feature pair, and use the set of feature pairs as a new training set for training the ranking model based on learning to rank. Then, use the feature difference value set of each test sample in the synthetic test set T as a new test sample for the ranking model. The new test sample is input into the ranking model, and the ranking model automatically scores the test sample based on the feature difference value of the sample. The size of the score value reflects the degree of correlation between the test sample and the error of the model L; the test samples are sorted according to the size of the correlation based on the score value; the higher the ranking of the test sample, the more likely it is to reveal the error of the model L.

[0072] 5. Select 1% of the top test samples from the sorted test set and mark them, input them into the original tested model L, perform retraining and then perform the test, report the improvement effect of the retrained model accuracy, and thus prove the effectiveness of the present invention.

[0073] The present invention also provides a deep learning test sample selection system based on feature distribution analysis, comprising:

[0074] The feature distribution partitioning module is used to partition the feature distribution of the training samples using a clustering method to obtain feature distribution clusters of the training samples;

[0075] A feature difference value calculation module is used to calculate the feature difference value of each training sample / test sample distributed on each feature distribution cluster based on the feature distribution cluster of the training samples;

[0076] A feature dispersion calculation module is used to calculate the feature dispersion of the training sample according to the output vector of the training sample;

[0077] The sorting model building module builds a sorting model based on the sorting learning algorithm and realizes the prediction and sorting of test samples;

[0078] A generation module for generating synthetic test sets using adversarial attack methods;

[0079] The test sample selection module is used to sort the synthetic test set using the sorting model, set the sampling ratio, and select the test samples with the highest sorting to form the test subset.

[0080] The present invention has many specific application paths. The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements can be made without departing from the principles of the present invention. These improvements should also be considered as the scope of protection of the present invention.

Claims

1. A deep learning test sample selection method based on feature distribution analysis, characterized in that: Here are the steps: 1) Use clustering method to divide the feature distribution of training samples and obtain the feature distribution clusters of training samples; 2) Based on the feature distribution clusters of the training samples obtained in step 1), calculate the feature difference value of each training sample / test sample distributed in each feature distribution cluster; 3) Calculate the feature dispersion of the training sample based on the output vector of the training sample; 4) Build a ranking model based on the ranking learning algorithm and realize the prediction and ranking of test samples; 5) Use the adversarial attack method to generate a synthetic test set, use the sorting model to sort the synthetic test set, set the sampling ratio, and select the top-ranked test samples to form a test subset.

2. The method for selecting deep learning test samples based on feature distribution analysis according to claim 1, characterized in that: The step 1) specifically includes: inputting all training samples into a DNN model for prediction to obtain respective output vectors, i.e., feature vectors of all training samples; clustering the training samples based on the feature vectors using a K-means clustering algorithm to divide the feature distributions of different categories of the training samples on the corresponding DNN model to obtain multiple feature distribution clusters, each feature distribution cluster representing a class of feature distribution, and the training samples under the same feature distribution cluster have the same class of feature distribution.

3. The method for selecting deep learning test samples based on feature distribution analysis according to claim 1, characterized in that: The step 2) specifically includes: Calculation of feature difference value: Given a sample set S of size m, a model to be tested D and the number of feature distribution clusters of the training samples obtained by the K-means clustering algorithm on the model to be tested D, the set of feature distribution clusters is C = {c1, c2, ..., c w }; Let s h is the hth sample in the sample set S, c j is the jth cluster, according to the MMD-critic algorithm from c j Select k prototypes to form the set X j ={X j，1 , x j，2 ,...,x j，k }, x j，p For the set X j The p-th prototype in; according to sample s h The characteristic vector of sample s is calculated h With set X j The distance of all prototypes in , the average of all distances is taken as sample s h Distributed in cluster c j The characteristic difference value FDF h，j ; The calculation is as follows: Then sample s h Distributed in cluster set C = {c1, c2, ..., c w The set of characteristic difference values ​​on} is FDF h ={FDF h，1 , FDF h，2 ,...,FDF h，w }.

4. The method for selecting deep learning test samples based on feature distribution analysis according to claim 1, characterized in that: The step 3) specifically includes: Calculation of feature dispersion: Given a training set R and a model to be tested D, the number of categories in the training set R is n. Assume that the predicted probability vector of a training sample r (r∈R) on the model to be tested D is P(r) = <p r，1 , p r，2 ,...,p r，n >, p r，n Indicates the probability value of the training sample r being predicted as the nth category; assuming that the true category of the training sample r is the tth category, according to the predicted probability p on the tth category r，t Construct a reference vector P′ of length n (r) = <P r，t , p r，t ,...,p r，t >, p r，t Indicates the probability value of training sample r being predicted as the t-th category; the feature dispersion (FDS) of training sample r is calculated as follows: The higher the feature dispersion FDS value of the training sample, the more dispersed its feature distribution.

5. The method for selecting deep learning test samples based on feature distribution analysis according to claim 1, characterized in that: The step 4) specifically includes: 41) Construction of new training set: The new training set is denoted as FP = {fp 1，2 , fp 2，3 ,…,fp n-1，n }, fp n-1，n Represents the original training sample r n-1 and r n A feature pair consisting of the feature difference value of i and r j A feature pair fp consisting of the feature difference value p，i , denoted as fp p，i = <FDF i , FDF j > This feature has an impact on fp p，i label i，j The value is -1, 0 or 1; label label i，j From the original training sample r i and r j The relative size of the feature dispersion is determined by the label value; the value of the label reflects the relative size of the training model ability between the original training samples in the feature pair; when FDF i >FDF j When label i，j =1, indicating that the original training sample r i The feature of the original training sample r j The feature distribution is dispersed and the ability to train the model is strong; when FDF i =FDF j When label i，j =0, indicating that the feature distribution of the original training sample ri is the same as that of the original training sample r j The dispersion of the feature distribution is the same, and the ability to train the model is the same; when FDF i <FDF j When label i，j =-1, indicating that the original training sample r j The feature of the original training sample r i The feature distribution is dispersed, and the training model has strong capabilities. The obtained new training set FP is input into the xgboost sorting algorithm to complete the construction of the sorting model. 42) Construction of new test set: construct new test samples by extracting the feature difference value set of the test samples; assuming a test sample t i The characteristic difference value set is FDF i , then according to the test sample t i The new test sample constructed is a feature pair containing only one feature difference value set, that is, <FDF i > 43) Test sample prediction: Input a new test sample into the constructed ranking model, add up the scores of each feature of the new test sample at the corresponding leaf nodes on each tree in the ranking model, and obtain the predicted score of each test sample; based on the size of the predicted score, prioritize the test samples with larger predicted scores to achieve ranking of the test samples.

6. The method for selecting deep learning test samples based on feature distribution analysis according to claim 1, characterized in that: The step 5) specifically includes: using five adversarial attack methods, namely FGSM, BIM-A, BIM-B, CW and JSMA, to perturb each test sample in the test set to generate an adversarial sample set, randomly selecting 20% ​​of the adversarial samples from the adversarial sample set and 80% of the test samples from the test set to form a synthetic test set with different data distributions; using a sorting model to sort each synthetic test set, setting a sampling ratio, and selecting the test samples with the highest sorting from the sorted synthetic test set according to the ratio to form a test subset to complete the selection of test samples.

7. A deep learning test sample selection system based on feature distribution analysis, characterized in that: include: The feature distribution partitioning module is used to partition the feature distribution of the training samples using a clustering method to obtain feature distribution clusters of the training samples; A feature difference value calculation module is used to calculate the feature difference value of each training sample / test sample distributed on each feature distribution cluster based on the feature distribution cluster of the training samples; A feature dispersion calculation module is used to calculate the feature dispersion of the training sample according to the output vector of the training sample; The sorting model building module builds a sorting model based on the sorting learning algorithm and realizes the prediction and sorting of test samples; A generation module for generating synthetic test sets using adversarial attack methods; The test sample selection module is used to sort the synthetic test set using the sorting model, set the sampling ratio, and select the test samples with the highest sorting to form the test subset.

8. A selection terminal, characterized in that: include: one or more processors; a memory for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors are caused to implement the method according to any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the selection method according to any one of claims 1 to 6 is implemented.

Images