Deep neural network test image selection method and system based on multi-objective optimization

Abstract

The application discloses a kind of deep neural network test picture selection method and system based on multi-objective optimization, to improve the accuracy of image recognition system.The application first collects the picture test data in different application scenarios as the original picture test set of pre-training image recognition model.Then according to the size of picture test set, design picture test subset selection individual, and design the uncertainty and diversity of two indexes of picture test subset, use multi-objective optimization algorithm NSGA-II to solve the problem, finally obtain the Pareto optimal solution set.Finally, according to the Pareto optimal solution set, the optimal picture test subset is obtained, and the pre-trained image recognition model is retrained to improve the accuracy of the model.The application introduces the uncertainty and diversity of two indexes of picture test subset, so that the model can learn more diversified features during retraining, thereby achieving better retraining effect and improving the quality of image recognition system to the greatest extent.

Description

Technical Field

[0001] This invention relates to the fields of image recognition and computer software, and mainly to a method and system for selecting test images based on multi-objective optimization deep neural networks. Background Technology

[0002] With the advancement of science and technology and the prosperity of the social economy, coupled with my country's large population, the collection, identification, authentication, and protection of citizens' identity information have become of paramount importance. Among the numerous authentication technologies available, facial recognition, a type of biometric identification technology, stands out due to its speed, convenience, and contactless nature, becoming a hot research topic in the industry and maintaining rapid and continuous development.

[0003] Due to the rapid development of deep learning, image recognition methods based on deep learning have been proposed one after another, such as face recognition. A complete face recognition system includes two stages: training and recognition. In the training stage, a face dataset containing at least one million data points is collected and cleaned before training the deep learning model. In the recognition stage, the face image to be recognized is input into the model and preprocessed and its features are extracted to obtain the recognition result.

[0004] However, the different distribution of image data used to build the model and image data collected from real-world application environments can lead to lower-than-expected model performance in practical applications. For example, variations in lighting, facial expressions, age, and target image resolution can cause unexpected errors in the model. Therefore, to make deep learning models more adaptable to real-world application environments, testers collect useful image test inputs from a large amount of image test data, label them, and retrain the current model to improve its quality and the accuracy of the face recognition system. However, labeling these image samples requires significant manpower, which is very time-consuming and expensive. Therefore, researchers sample the image test data to obtain a high-quality subset of image test data, maximizing the effect of model retraining.

[0005] Most existing techniques focus on selecting test images that are easily misclassified by the model for retraining. Current research indicates that test images closer to the model's decision boundary are more beneficial for improving retraining performance. Therefore, researchers are working to develop an effective metric to measure the probability of a test image being misclassified by the model, ranking them from highest to lowest, and sampling a subset of test images according to a certain sampling rate. However, existing metrics often calculate the output probability of the test image in each class, such as using the quotient of the second-highest predicted probability and the highest predicted probability to represent the probability of the test image being misclassified. However, these metrics only consider the influence of the classes with the highest and second-highest predicted probabilities on the probability of the test image being misclassified, without considering the influence of other classes. Therefore, this metric is not a comprehensive evaluation of the test image. Other researchers have used the difference between the sum of the squares of the predicted probabilities of the test image belonging to each class and 1 as an evaluation metric for the probability of the test image being misclassified. However, this evaluation metric simply uses the probability of the image test input belonging to each class for evaluation, ignoring the internal feature information of the image test input, making the evaluation of the image test input incomplete and inaccurate.

[0006] Furthermore, if the image test subset is selected solely based on the probability that the image test input will be misclassified by the model, images with similar probabilities will be chosen. Similar probabilities imply that the image test inputs have similar properties, leading to redundancy in the selected subset and wasting labeling costs. Moreover, these techniques cannot guarantee the diversity of the selected image test subset, which may prevent the model from learning more diverse features during retraining, thus hindering model quality improvement. Summary of the Invention

[0007] Purpose of the invention: To address the problems existing in the above-mentioned background technology, this invention provides a method and system for selecting test images based on multi-objective optimization of deep neural networks. It introduces two indicators: uncertainty and diversity of the image test subset, and uses the multi-objective optimization algorithm NSGA-II to solve the problem, so that the model can learn more diverse features during retraining, thereby achieving better retraining results and maximizing the quality of the image recognition system.

[0008] Technical solution: To achieve the above objectives, the technical solution adopted by this invention is as follows:

[0009] A deep neural network-based test image selection method based on multi-objective optimization includes the following steps:

[0010] Collect image test data from different application scenarios as the original image test set for pre-trained image recognition models;

[0011] Based on the size n of the image test set, an initial population for the multi-objective optimization algorithm NSGA-II is generated. Each individual in the population is an n-dimensional vector, and each dimension of the individual vector uses a real number to represent the weight of the test image selected at the corresponding position. Each individual corresponds to an image test subset. After sorting the elements in the individual from largest to smallest, a preset number of test images at the corresponding positions are selected to form the image test subset.

[0012] With the goal of maximizing the uncertainty and diversity of the image test subset, the NSGA-II algorithm is used for population iterative solution to obtain the Pareto optimal solution set. Each Pareto optimal solution is an n-dimensional vector, and the real number of each dimension represents the weight of the test image selected at the corresponding position.

[0013] The pre-trained image recognition model is retrained based on the optimal image test subset corresponding to the Pareto optimal solution set.

[0014] Furthermore, adversarial sample images are generated for the original image test set, and the adversarial sample images are mixed with the original image test set to simulate different application scenarios.

[0015] Furthermore, the formal representation of the uncertainty of the image test subset is as follows:

[0016]

[0017] Among them, uncertainty(X) s ) represents the image test subset X s Uncertainty; x i It is a subset X of the image test s For any test image, C is the number of categories classified by the image recognition model; Indicates test image x i The probability of belonging to category c.

[0018] Furthermore, the formal representation of the diversity of the image test subset is as follows:

[0019]

[0020] Among them, diversity(X) s ) represents the image test subset X s diversity; x i and x j It is a subset X of the image test s Any two different test images; ||x i -x j ||2 represents the test image x i and xj The Euclidean distance between them.

[0021] Furthermore, the specific steps for using the multi-objective optimization algorithm NSGA-II to solve the multi-objective problem and obtain the Pareto optimal solution set include:

[0022] An initial population is randomly generated, with each individual in the population corresponding to a subset of test images; the first generation of offspring population is obtained through three basic operations: selection, crossover, and mutation.

[0023] Starting from the second generation, the parent population and the offspring population are merged. In this newly generated population, non-dominated sorting is performed based on the two objectives of uncertainty and diversity of the image test subset. At the same time, the crowding degree of individuals in each non-dominated layer is calculated. Individuals are selected to form a new parent population based on the non-dominated relationship and the crowding degree of individuals.

[0024] New offspring are generated through basic operations; when the iteration reaches the specified termination condition, the iteration process ends and returns the Pareto optimal solution set.

[0025] Furthermore, for each solution in the Pareto optimal solution set, the elements are sorted from largest to smallest, and then a preset number of test images at corresponding positions are selected to form an optimal image test subset.

[0026] Furthermore, the method for retraining an image recognition model involves using a selected subset of test images to retrain the pre-trained model for several epochs to obtain the retrained model.

[0027] A deep neural network-based test image selection system based on multi-objective optimization includes:

[0028] The preprocessing module is used to collect image test data from different application scenarios as the original image test set for pre-trained image recognition models;

[0029] The image test subset selection module generates an initial population for the multi-objective optimization algorithm NSGA-II based on the size n of the image test set. Each individual in the population is an n-dimensional vector, and each dimension of the individual vector represents the weight of the test image selected at the corresponding position using real numbers. Each individual corresponds to an image test subset. After sorting the elements in the individual from largest to smallest, a preset number of test images at the corresponding positions are selected to form the image test subset. With the goal of maximizing the uncertainty and diversity of the image test subset, the NSGA-II algorithm is used to iteratively solve the population problem to obtain the Pareto optimal solution set. Each Pareto optimal solution is an n-dimensional vector, and each dimension represents the weight of the test image selected at the corresponding position using real numbers.

[0030] In addition, there is a retraining module, which is used to retrain the pre-trained image recognition model based on the optimal image test subset corresponding to the Pareto optimal solution set.

[0031] A computer system includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program, when loaded onto the processor, implements the steps of the deep neural network test image selection method based on multi-objective optimization.

[0032] Beneficial Effects: Compared with existing technologies, the deep neural network test image selection method based on multi-objective optimization provided by this invention comprehensively considers the uncertainty and diversity of the image test subset, enabling the selected image test subset to allow the image recognition model to learn more diverse features during retraining, thereby maximizing the quality of the image recognition system. Furthermore, this invention introduces the classic multi-objective optimization algorithm NSGA-II to solve the problem, effectively ensuring the quality of the solution and thus guaranteeing the effectiveness of model retraining. Attached Figure Description

[0033] Figure 1 This is a flowchart of the test image selection method provided in the embodiments of the present invention. Detailed Implementation

[0034] The present invention will be further described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without inventive effort are within the scope of protection of the present invention.

[0035] like Figure 1 As shown, this embodiment of the invention provides a method for selecting test images using a deep neural network based on multi-objective optimization, mainly including a preprocessing part, an image test subset selection part, and a retraining part. Specifically,

[0036] Step S1, Preprocessing stage:

[0037] For convolutional neural network models, input images are often represented as vectors. After standardization or normalization, an appropriate optimizer and training iterations are selected to train the constructed image recognition model (such as a face recognition model, scene recognition module, etc.) to obtain a pre-trained model.

[0038] Then, image test data from different application scenarios are collected as the original image test set. Furthermore, this embodiment uses classic image adversarial attack methods, namely FGSM, JSMA, BIM-A, and BIM-B, to generate adversarial example images against the collected original image test data. The adversarial example images and the original image test data are then mixed in a 2:8 ratio to simulate different application scenarios. Next, this embodiment evaluates the accuracy of the pre-trained model in each application scenario. This accuracy is the accuracy before model retraining and serves as a benchmark for comparing the accuracy improvement after model retraining.

[0039] Step S2, Image Test Subset Selection Stage:

[0040] This embodiment designs two indicators: uncertainty and diversity of the image test subset, and uses the multi-objective optimization algorithm NSGA-II to solve them, obtaining the Pareto optimal solution set.

[0041] Step S2.1: Design two indicators for the image test subset: uncertainty and diversity, and formally express the two optimization objectives.

[0042] First, multi-objective optimization problems belong to the field of multidimensional decision-making, and are mathematical problems involving the simultaneous optimization of multiple objective functions. The purpose of multi-objective optimization is to balance two or more conflicting objectives and make the optimal decision. The formal definition of multi-objective optimization is as follows:

[0043]

[0044] Where x is an S-dimensional decision vector in the search space, f m (x) is the m-th objective to be optimized (m = 1, 2, ..., M), where M is the number of objectives to be optimized.

[0045] Domination relationships exist between individuals in a population. In multi-objective optimization problems, a domination relationship means that if one individual is no worse than another individual on all objective functions and is better than another individual on at least one objective function, then that individual is said to dominate the other individual.

[0046] In general, the optimal solution in a multi-objective optimization problem is called the Pareto optimal solution. The general description of a Pareto optimal solution is: given a multi-objective optimization problem maxf(x), if x... * ∈Ω, and there are no other x∈Ω such that f i (x * )≤f i If (x)(i=1,2,…,m) holds true, and at least one of them is a strict inequality, then x *It is the Pareto optimal solution of maxf(x). Moreover, in most cases, there is more than one Pareto optimal solution, and all Pareto optimal solutions form the Pareto optimal solution set.

[0047] For the first optimization objective, which is to maximize the uncertainty of the test image subset, this embodiment uses entropy, an information-theoretic metric, to calculate the uncertainty of the test images. This metric is based on the output of the model's softmax layer and takes into account the predicted probability value of each category, making the uncertainty assessment of the test images more comprehensive and accurate.

[0048] The formal expression for maximizing the uncertainty of the image test subset is as follows:

[0049]

[0050] Among them, uncertainty(X) s ) represents the image test subset X s Uncertainty; x i It is a subset X of the image test s C is any test image; C is the number of categories that the deep neural network model can classify. Indicates test image x i The probability of belonging to category c.

[0051] For the second optimization objective, namely maximizing the diversity of the image test subset, this embodiment uses a distance metric method—Euclidean distance—to reflect the diversity of the image test subset by calculating the distance between test images in the image test subset.

[0052] The formal expression for maximizing the diversity of image test subsets is as follows:

[0053]

[0054] Among them, diversity(X) s ) represents the image test subset X s diversity; x i and x j It is a subset X of the image test s Any two different test images; ||x i -x j ||2 represents the test image x i and x j The Euclidean distance.

[0055] The specific execution steps of the NSGA-II multi-objective optimization algorithm in step S2 include:

[0056] Define the representation of individuals in the population. Each individual in the population is an n-dimensional vector (n is the size of the image test set). Each dimension of the individual vector uses a real number from 0 to 100 to represent the weight of the test image selected at the corresponding position.

[0057] An initial population is randomly generated, with each individual in the population corresponding to a subset of test images. The first generation of offspring is obtained through the three basic operations of selection, crossover, and mutation.

[0058] Starting from the second generation, the parent and offspring populations are merged. Individuals in the population are ranked using a non-dominated sorting method, classifying them according to their dominance relationships in the optimization objective values ​​(i.e., the values ​​of both uncertainty and diversity objectives). This results in a set of solutions arranged in ascending order of non-dominance level. A lower non-dominance level indicates a solution closer to the Pareto optimum. Then, a crowding value is calculated based on each individual's distance from other individuals in the objective space, ranking individuals within the same level. A higher crowding value indicates a higher likelihood of that solution being selected for the next generation. Finally, suitable individuals are selected based on the non-dominance relationships and their crowding values ​​to form a new parent population.

[0059] New offspring are generated through basic operations; the iteration process ends and returns a Pareto optimal solution set after a specified number of iterations or other specified termination conditions are met. The obtained Pareto optimal solution set includes one or more Pareto optimal solutions, each of which is the optimal solution to the problem to be solved.

[0060] Step S3, Model Retraining Phase:

[0061] A subset of images is sampled and labeled based on the Pareto optimal solution. The model is then retrained based on the pre-trained model, and the change in accuracy before and after retraining is calculated.

[0062] First, for each Pareto optimal solution in the Pareto optimal solution set, the elements are sorted from largest to smallest. Then, a certain number of test images at corresponding positions are selected to form a test image subset.

[0063] Then, using the selected image test subset, the pre-trained model is retrained for 5 epochs using the same optimizer as in the training phase to obtain the retrained model. Furthermore, to reduce the impact of randomness during retraining, this embodiment repeats the retraining process 10 times, and the average value is used as the final result.

[0064] Finally, the Average Accuracy Improvement (AAI) metric is used to calculate the improvement in model accuracy before and after retraining. Its calculation method is as follows:

[0065]

[0066] Among them, M r Let M0 represent the retrained model, D represent the original image test set, and Acc(M,D) represent the accuracy of model M on dataset D.

[0067] To verify the effectiveness of the proposed method, two existing test image selection methods, SA and DeepGini, were selected. SA is a distance-based test image selection method that uses a distance metric to evaluate the difference between each test image and the training image set, and then selects test images with larger differences to form a test subset. DeepGini, from a statistical perspective, uses a metric to measure the probability that each test image will be incorrectly predicted by the model, and then selects test images based on the probability.

[0068] We selected the publicly available image test set CIFAR10 as our experimental data and used four adversarial attack methods—FGSM, JSMA, BIM-A, and BIM-B—to generate different test datasets to simulate different testing environments. Furthermore, we set two different sampling rates, 1% and 3%, to evaluate the improvement in model accuracy after retraining the model with test subsets selected by different methods under different sampling rates and testing environments. The results are shown in Table 1.

[0069] Table 1 Comparison of accuracy improvement (%) of the model after retraining

[0070]

[0071] Experimental results show that, compared with existing methods, retraining the model using the test images selected in this embodiment is more effective.

[0072] Based on the same inventive concept, embodiments of the present invention provide a deep neural network-based test image selection system with multi-objective optimization, comprising:

[0073] The preprocessing module is used to collect image test data from different application scenarios as the original image test set for pre-trained image recognition models;

[0074] The image test subset selection module generates an initial population for the multi-objective optimization algorithm NSGA-II based on the size n of the image test set. Each individual in the population is an n-dimensional vector, and each dimension of the individual vector represents the weight of the test image selected at the corresponding position using real numbers. Each individual corresponds to an image test subset. After sorting the elements in the individual from largest to smallest, a preset number of test images at the corresponding positions are selected to form the image test subset. With the goal of maximizing the uncertainty and diversity of the image test subset, the NSGA-II algorithm is used to iteratively solve the population problem to obtain the Pareto optimal solution set. Each Pareto optimal solution is an n-dimensional vector, and each dimension represents the weight of the test image selected at the corresponding position using real numbers.

[0075] In addition, there is a retraining module, which is used to retrain the pre-trained image recognition model based on the optimal image test subset corresponding to the Pareto optimal solution set.

[0076] Based on the same inventive concept, embodiments of the present invention provide a computer system, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the computer program is loaded onto the processor, it implements the steps of the deep neural network test image selection method based on multi-objective optimization.

[0077] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A method for selecting test images for deep neural networks based on multi-objective optimization, characterized in that, The method comprises the following steps: Collecting picture test data under different application scenarios as an original picture test set of a pre-trained image recognition model; According to the size of the picture test set , an initial population of a multi-objective optimization algorithm NSGA-II is generated, each individual in the population is a dimensional vector, and each dimension of the individual vector is represented by a real number indicating the weight of the test picture at the corresponding position; wherein each individual corresponds to a picture test subset, and after sorting the elements in the individual from large to small, a preset number of test pictures at the corresponding positions are selected to form the picture test subset; To maximize the uncertainty and diversity of the picture test subset, the NSGA-II algorithm is used for population iteration solution, and a Pareto optimal solution set is obtained, each Pareto optimal solution is a dimensional vector, and the real number of each dimension represents the weight of the test picture selected at the corresponding position. According to an optimal picture test subset corresponding to a Pareto optimal solution set, retraining the pre-trained image recognition model; A formalized representation of uncertainty of the picture test subset is: ； wherein, denotes a picture test subset of uncertainty; is a picture test subset of any one test picture, is the number of classes classified by the image recognition model; denotes a test picture belongs to class with probability A formalized representation of diversity of the picture test subset is: ； wherein denotes a picture test subset of a plurality of pictures; and denotes any two different test pictures of the picture test subset ; denotes the Euclidean distance between the test pictures and . 2.The method of claim 1, wherein, Adversarial sample pictures are generated for the original picture test set, and the adversarial sample pictures and the original picture test set are mixed to simulate different application scenarios. 3.The method of claim 1, wherein, The specific execution steps of using a multi-objective optimization algorithm NSGA-II to solve a multi-objective problem to obtain a Pareto optimal solution set include: Randomly generating an initial population, each individual in the population corresponding to a picture test subset; obtaining a first generation of offspring population through selection, crossover and mutation three basic operation operators; From the second generation, the parent population and the offspring population are combined, and in the newly generated population, the uncertainty and diversity of the picture test subset are sorted according to the two objectives, and the individuals in each non-dominated layer are calculated according to the crowding degree, and the individuals are selected according to the non-dominated relationship and the crowding degree of the individuals to form a new parent population; New offspring are generated through basic operation operators; when the iteration reaches the specified termination condition, the iteration process ends and returns the Pareto optimal solution set.

4. The method of claim 1, wherein, For each solution in the Pareto optimal solution set, the elements are sorted from large to small, and then the test pictures corresponding to the preset number of positions are selected to form an optimal picture test subset.

5. The method of claim 1, wherein, The method for retraining the image recognition model is to use the selected picture test subset to retrain the pre-trained model for a number of epochs to obtain the retrained model.

6. A multi-objective optimization based deep neural network test picture selection system, characterized by, It comprises: a preprocessing module for collecting picture test data under different application scenarios as an original picture test set of a pre-trained image recognition model; The picture test subset selection module is configured to select a picture test subset according to a size of the picture test set , generate an initial population of a multi-objective optimization algorithm NSGA-II, each individual in the population is a dimensional vector, each dimension of the individual vector is represented by a real number to indicate the weight of the test picture at the corresponding position; wherein each individual corresponds to a picture test subset, and after sorting the elements in the individual from large to small, a preset number of test pictures at the corresponding positions are selected to form the picture test subset; To maximize the uncertainty and diversity of the picture test subset, the NSGA-II algorithm is used for population iteration solution, and a Pareto optimal solution set is obtained, each Pareto optimal solution is a dimensional vector, and the real number of each dimension represents the weight of the test picture selected at the corresponding position. and a retraining module for retraining the pre-trained image recognition model according to an optimal picture test subset corresponding to a Pareto optimal solution set; A formalized representation of uncertainty of the picture test subset is: ； wherein, denotes a picture test subset of uncertainty; is a picture test subset of any one test picture, is the number of classes classified by the image recognition model; denotes a test picture belongs to class ; a formalized representation of the diversity of the picture test subset is: ； wherein denotes a picture test subset of diversity; and denotes any two different test pictures of the picture test subset . denotes the Euclidean distance between the test pictures and .

7. A computer system comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, The computer program is loaded into the processor to implement the steps of the multi-objective optimization-based deep neural network test picture selection method according to any one of claims 1-5.

Images