A training sample selection method for DNN model based on semantic features

Abstract

The present invention provides a training sample selection method for a DNN model based on semantic features. The method comprises: extracting semantic features of each sample image using a saliency detection method based on frequency domain; clustering and sampling the semantic features of all sample images, dividing the sample images into multiple categories with different semantic features, and obtaining sample images for testing and retraining a deep neural network DNN model. The present invention extracts semantic features of test cases by using a saliency detection method based on frequency domain, which can capture the essential features of data consistent with human intuition, and cluster them into different categories in the frequency domain feature space. The saliency detection method based on frequency domain is used to extract semantic features of test cases, which can more effectively detect various errors of DNN and improve the adversarial robustness of the DNN model.

Description

Technical Field

[0001] The present invention relates to the technical field of sample selection, and in particular to a training sample selection method for a DNN (Deep Neural Networks) model based on semantic features. Background Art

[0002] Artificial intelligence refers to the intelligence displayed by machines created by humans. With the rapid development of technologies such as big data, cloud computing, and the Internet, deep learning, as a representative technology of artificial intelligence, has made remarkable achievements in computer vision, speech recognition, and natural language processing, and has had a profound impact on the economy and lifestyle of human society.

[0003] Although deep learning has performed well in various fields, it still faces serious problems of insufficient robustness in real scenarios. Robustness refers to the ability of deep learning models to maintain their performance and reliability when encountering small changes in input data or unforeseen situations, which is crucial to ensuring its security. Studies have shown that the most advanced deep learning models are still vulnerable to maliciously designed adversarial samples and perturbed samples in natural environments, giving wrong or overconfident predictions on perturbed inputs. This instability poses a huge security risk to the application of deep learning models in real scenarios.

[0004] Deep learning models use a data-driven programming paradigm and make decisions through non-deterministic logic, so a large number of test cases need to be generated during testing to achieve adequate evaluation. In addition, traditional software failures usually originate from specific functions or lines of code, while DNN failures usually manifest as model errors, which are usually related to data, learning algorithms, or training processes. Therefore, errors in DNN models cannot be located in specific components (such as neurons or layers) and fixed directly. It is necessary to analyze the errors and select appropriate samples to retrain the model to improve the robustness of the model on faulty data.

[0005] In order to reduce testing overhead and annotation costs, and improve model robustness after testing, a test sample selection method for a DNN model in the prior art includes: selecting representative test cases based on specific rules (such as uncertainty and robustness) to test and retrain the DNN model. This method usually focuses on the number of error detections, but ignores the diversity of errors. In real scenarios, the test cases obtained by this method lack diversity, resulting in an inability to fully cover the input space of the model. In addition, studies have shown that using these test cases to retrain the DNN model is not effective in improving the robustness of the DNN model. Summary of the invention

[0006] An embodiment of the present invention provides a method for selecting training samples for a DNN model based on semantic features, so as to achieve efficient acquisition of training samples for the DNN model.

[0007] In order to achieve the above object, the present invention adopts the following technical scheme.

[0008] A method for selecting test and training samples based on semantic features, comprising:

[0009] The semantic features of each sample image are extracted using a frequency-domain-based saliency detection method;

[0010] The semantic features of all sample images are clustered and sampled, and the sample images are divided into multiple categories with different semantic features to obtain sample images for testing and retraining deep neural network (DNN) models.

[0011] Preferably, the method of extracting semantic features of each sample image using a frequency domain-based saliency detection method includes:

[0012] A sample image set is obtained, each sample image in the sample image set is selected in turn, the sample image is converted from the spatial domain to the frequency domain using a frequency domain-based saliency detection method, and semantic features are extracted from the sample image in the frequency domain, the semantic features including color, texture, shape and edge information.

[0013] Preferably, the frequency domain-based saliency detection method comprises: Fourier phase transform FPT, high pass filter HPF, residual spectrum SR and quaternary Fourier transform QFT.

[0014] Preferably, when the frequency domain-based saliency detection method adopts Fourier phase transform FPT, the method of extracting semantic features of each sample image using the frequency domain-based saliency detection method includes:

[0015] Given an image matrix g with M rows and N columns, f(mn) represents the pixel value of the mth row and nth column. The Fourier transform is used to transform the input image g from the spatial domain to the frequency domain. The Fourier transform F(u,v) of f(mn) is shown in formula (5-1):

[0016]

[0017] The input image is converted from the frequency domain back to the spatial domain through the inverse fast Fourier transform IFFT, as shown in formula (5-2):

[0018]

[0019] Based on FFT and IFFT, the saliency image of the input image is calculated using Fourier phase transform FPT, high pass filter HPF, residual spectrum SR and quaternion Fourier transform QFT respectively:

[0020] (1) Fourier phase transform FPT: The phase spectrum of the image is calculated by taking the phase part through Fourier transform, as shown in formula (5-3).

[0021]

[0022] Among them, Im[F(u,v)] and Re[F(u,v] are the imaginary and real parts of F(u,v) respectively. By calculating IFFT on P(u,v), the saliency map S can be obtained. a (m,n), saliency image S a (m,n) mainly contains the frequency domain structure and features of the original image. a (m,n) can extract the semantic features of the pixel value f(mn).

[0023] Preferably, when the frequency-domain-based saliency detection method adopts a high-pass filter HPF, the method of extracting semantic features of each sample image using the frequency-domain-based saliency detection method includes:

[0024] Set the Sobel high-pass filter to include the edge matrix G for detecting horizontal x and the vertical edge matrix G y , as shown in formula (5-4):

[0025]

[0026] The horizontal gradient value V of the pixel value f(mn) x (m,n) and the vertical gradient value V y (m,n) is calculated by the convolution operation in formula (5-5):

[0027] V x (m,n)=f(m,n)*G x , V y (m,n)=f(m,n)*G y (5-5)

[0028] Calculate the saliency image S using formula (5-6) a (m,n):

[0029]

[0030] The saliency image S a (m,n) is used as the semantic feature of the pixel value f(mn).

[0031] Preferably, the said clustering and sampling of the semantic features of all sample images, dividing the sample images into multiple categories with different semantic features, and obtaining sample images for testing and retraining the DNN model, includes:

[0032] Extract semantic features from each image, where the semantic features are one of the attributes of color, texture, shape and edge information, form a feature space with the semantic features of all images, randomly select an element from the feature space as the first cluster center of the k-means++ algorithm, that is, take the image corresponding to the semantic features of the selected element as the first centroid, calculate the shortest distance from each element in the feature space to the existing cluster center, use the square of this distance as the probability of selecting the element as the new centroid, select the element corresponding to the maximum probability as the new centroid, repeat the above process until all centroids are selected, and divide the sample image into multiple categories with different semantic features;

[0033] The uniform sampling (US) strategy is used to uniformly sample sample images of each category to obtain test case sample images for testing and retraining the DNN model.

[0034] It can be seen from the technical solution provided by the above-mentioned embodiments of the present invention that the present invention provides a method for extracting semantic features of test cases using a frequency-domain-based saliency detection method, which can capture the essential features of data consistent with human intuition and cluster them into different categories in the frequency-domain feature space. Using a frequency-domain-based saliency detection method to extract semantic features of test cases can more effectively detect various errors in DNNs. In addition, the adversarial robustness and OOD robustness of the model are effectively improved after retraining.

[0035] Additional aspects and advantages of the present invention will be given in part in the following description, which will become obvious from the following description, or may be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative work.

[0037] Figure 1 A schematic diagram of a method for selecting training samples for a DNN model based on semantic features provided in an embodiment of the present invention;

[0038] Figure 2A schematic diagram of an algorithm code for implementing a method for selecting test and training samples based on semantic features provided in an embodiment of the present invention;

[0039] Figure 3 The data set, model and attack result diagram provided by the embodiment of the present invention. DETAILED DESCRIPTION

[0040] The embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and cannot be interpreted as limiting the present invention.

[0041] It will be understood by those skilled in the art that, unless expressly stated, the singular forms "one", "said", and "the" used herein may also include plural forms. It should be further understood that the term "comprising" used in the specification of the present invention refers to the presence of the features, integers, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or groups thereof. It should be understood that when we refer to an element as being "connected" or "coupled" to another element, it may be directly connected or coupled to the other element, or there may be intermediate elements. In addition, the "connection" or "coupling" used herein may include wireless connection or coupling. The term "and / or" used herein includes any unit and all combinations of one or more associated listed items.

[0042] It will be understood by those skilled in the art that, unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as those generally understood by those skilled in the art in the art to which the present invention belongs. It should also be understood that terms such as those defined in common dictionaries should be understood to have meanings consistent with the meanings in the context of the prior art, and will not be interpreted with idealized or overly formal meanings unless defined as herein.

[0043] To facilitate understanding of the embodiments of the present invention, several specific embodiments will be further explained below with reference to the accompanying drawings, and each embodiment does not constitute a limitation on the embodiments of the present invention.

[0044] Frequency domain analysis involves the study of the frequency components of a signal. Frequency domain analysis techniques have a wide range of applications, from time series data to images. The Fourier transform is the basis of frequency domain analysis, which converts a signal in the time or spatial domain into its frequency domain representation. Frequency domain analysis has been widely used in traditional image processing, especially in saliency detection. Frequency domain analysis methods usually use Discrete Fourier Transform (DFT) to convert an image from the spatial domain to the frequency domain. Frequency domain information can represent semantic features of data that are consistent with human perception, such as structure, edges, and texture.

[0045] The processing flow of a method for selecting training samples of a DNN model based on semantic features provided by an embodiment of the present invention is as follows: Figure 1 As shown, the processing steps include the following:

[0046] Step S1: extracting semantic features of each sample image using a frequency domain-based saliency detection method.

[0047] This method is applied to 1,000 images selected from the public dataset ImageNet, and all images constitute a sample image set.

[0048] Each image in the sample image set is selected in turn, and the sample image is converted from the spatial domain to the frequency domain using a saliency detection method based on the frequency domain, and semantic features are extracted from the sample image, and the semantic features include edges and textures.

[0049] Step S2: clustering and sampling the semantic features of the sample images, dividing the sample images into multiple categories with different semantic features, and obtaining the sample images finally selected for testing and retraining the DNN.

[0050] The Kmeans++ algorithm is used to cluster the semantic features of the samples into different categories in the frequency domain feature space. Then, the US (Uniform Strategy) strategy semantic feature classification can be used to divide the test cases into multiple categories with similar semantic features.

[0051] Specifically, the frequency domain-based saliency detection method in the above step S1 includes: Fourier Phase Transform (FPT), High-pass Filter (HPF), Residual Spectrum (SR) and Quaternion Fourier Transform (QFT).

[0052] Semantic feature extraction operation: Using the above four frequency-domain-based saliency detection methods: convert the sample image from the spatial domain to the frequency domain, and extract the semantic features of the sample image in the frequency domain, such as edges and textures.

[0053] In order to extract the semantic features of the sample image, the present invention uses Fourier transform to convert the input image from the spatial domain to the frequency domain, that is, given an image matrix g with M rows and N columns, f(mn) represents the pixel value of the mth row and nth column, and the Fourier transform F(u,v) of f(mn) is shown in formula (5-1):

[0054]

[0055] For an image8 matrix with M rows and N columns, the Fourier transform theoretically requires (MN)2 operations, which is very time-consuming. For this reason, the Fast Fourier Transform (FFT) can be used to reduce the computational complexity from O((MN)2) to O(MNIog(M*N)) by decomposing the DFT into a series of smaller DFTs.

[0056] In addition, the input image can be converted from the frequency domain back to the spatial domain through the Inverse Fast Fourier Transform (IFFT), as shown in formula (5-2).

[0057]

[0058] Based on FFT and IFFT, the above four saliency detection methods can be used to extract the semantic information of the test case. The calculation details are as follows:

[0059] (1) FPT: The phase spectrum of the image is calculated by taking the phase part through Fourier transform, as shown in formula (5-3).

[0060]

[0061] Among them, Im[F(u,v)] and Re[F(u,v] are the imaginary and real parts of F(u,v) respectively. By calculating IFFT on P(u,v), the saliency map S can be obtained. a (m,n).

[0062] (2) HPF: High-pass filter is mainly used to enhance the high-frequency components of an image, thereby highlighting edges and details.

[0063] The present invention uses a Sobel filter to calculate HPF. The Sobel filter is a discrete difference filter used for edge detection, which highlights the edge by calculating the gradient strength of the image. The Sobel filter contains two 3*3 matrices, as shown in formula (5-4), which are used to detect edges in the horizontal and vertical directions respectively.

[0064]

[0065] For each pixel in the image, its gradient values ​​in the horizontal and vertical directions are calculated through the convolution operation in formula (5-5).

[0066] V x (m,n)=f(m,n)*G x , V y (m,n)=f(m,n)*G y (5-5)

[0067] Then the total gradient strength is calculated by formula (5-6) to obtain the saliency image S a (m,n), the saliency image S a (m,n) is used as the semantic feature of the pixel value f(mn).

[0068]

[0069] The saliency image S obtained after processing a (m,n) highlights the edge area in the original image and can be regarded as the result of HPF processing.

[0070] (3) SR: This method estimates the saliency through the residual of the amplitude spectrum of the image. After performing FFT on a given image, the amplitude spectrum A(u,v) = |F(u,v)| and the phase spectrum P(u,v) are calculated. Then the average amplitude spectrum log(A(u,v)) and its smoothed version Sm(u,v) are calculated. The spectral residual is calculated by R(u,v) = log(A(u,v))Sm(u,v). The Fourier transform of the saliency map is S a (u,v)=e R(u,v) ×e iP(u,v) Finally, IFFT is used to obtain the saliency image S a (m,n). Where R(u,v) is the spectral residual and P(u,v) is its phase spectrum.

[0071] (4) QFT: The saliency test based on QFT shows that in the human visual system, the four feature channels of motion feature M, brightness feature 1, red-green neuron RG, and blue-yellow neuron BY are almost independent. Therefore, the above four features of the image can be represented by quaternion q(m,n.), and the weighted image quaternion representation is defined as follows:

[0072] q(m,n)=M(m,n)+RG(m,n)*μ 1 +BY(m,n)*μ 2 +I(m,n)*μ 3 (5-7)

[0073] Among them, μ i , i=1,2,3 satisfies μ i 2 = -1,μ 1 ⊥μ 2 ,μ 2 ⊥μ 3 ,μ 1 ⊥μ 3 ,μ 3 =μ 1 μ 2 The QFT transforms an image from the spatial domain to the frequency domain as follows:

[0074]

[0075] In the above calculation, the 2D FFT concept is actually extended to accommodate quaternion data. Then, in the frequency domain, the phase information of the basis Q(u,v) is significantly detected, that is, S a (u,v)=|Q(u,v)|. Finally, the saliency map S is obtained by performing inverse QFT on Sa(u,v) a (m,n).

[0076] Specifically, the Kmeans++ clustering algorithm in step S2 is an optimized version of the Kmeans clustering algorithm. The initial centroids are selected through a specific randomization process to reduce the impact of randomization, and then weighted random sampling is used to select the remaining centroids to divide all sample images into k clusters.

[0077] The k-means++ algorithm can be used to cluster images or regions in images based on their semantic features. Here, "semantic features" usually refer to features that can represent the content or structure of an image, such as color, texture, shape, and edge information. When using these features to cluster images, the steps for selecting the initial centroid and the remaining centroid of the k-means++ algorithm are as follows:

[0078] Extract features: First, extract semantic features from each image. The semantic feature can be one of the attributes of color, texture, shape, and edge information, depending on the image attribute you want the algorithm to focus on.

[0079] Randomly select the first centroid: Randomly select a point (i.e. a set of features) from the feature space as the first cluster center. This is equivalent to randomly selecting the features of an image (or image region) from the image set as the first centroid.

[0080] Select the remaining centroids based on distance: For each point in the feature space (i.e., the set of features for each image or image region), calculate its shortest distance to the existing centroids (this is usually the Euclidean distance or other suitable distance metric). Then, use the square of this distance as the probability of selecting the point as the new centroid. This means that images (or image regions) that are farther away from the existing centroids have a higher probability of being selected as the new centroid.

[0081] Repeat step 3: Repeat the above process until k centroids are selected.

[0082] Use the standard k-means algorithm: Once all k centroids are selected, use the standard k-means algorithm to iterate and cluster all images (or image regions) until the clustering results no longer change.

[0083] Use the US strategy to sample sample images of each category to obtain test case sample images used to test and retrain the DNN model. Using the US strategy helps ensure that the features of each category are evenly and fairly represented, especially when dealing with multi-category image data:

[0084] (1) Maintaining category balance. By uniformly sampling images of each category, the US strategy ensures that each category has the same number of samples in the final dataset. This helps prevent the model from being biased towards categories with a larger number of samples, thereby improving the performance and generalization ability of the model on all categories.

[0085] (2) Improve feature diversity. Uniform sampling can improve the feature diversity within each category. Even if some categories have low sample diversity under natural conditions, the US strategy ensures that different images are selected from these categories, thereby capturing a wider range of feature variations.

[0086] (3) Avoid overfitting: Sampling evenly across each category helps prevent the model from overfitting to specific images or features of a particular category. This is because the model needs to learn to recognize multiple representations of each category rather than relying solely on recurring, larger samples).

[0087] After adding different perturbations to the sample images, the semantic features extracted by the above four frequency-domain-based saliency detection methods on adversarial samples and OOD (Out of distribution) samples show the saliency maps of different extraction methods. From left to right are images in the spatial domain, and saliency maps extracted by FPT, HPF, SR, and QFT. Although adversarial attacks and OOD damage cause the original samples to lose some semantic information, the four frequency-domain-based methods all effectively extract the semantic information of adversarial samples and OOD samples. In addition, FPT performs best among the four saliency detection methods and can effectively detect image semantic features in all types of perturbation data.

[0088] The above US strategy can maximize the diversity of the selected test cases. The US strategy ensures that the samples selected from each cluster follow a uniform distribution and does not bias the sample selection towards any specific area, which helps to improve the diversity and coverage of the samples. The test provides a wider and more diverse set of test cases, which is very important for deep neural network testing.

[0089] In order to classify sample images into multiple categories with similar semantic features, after extracting the semantic features of the sample images, the clustering method is used to cluster the semantic features of the sample images into different categories in the frequency domain feature space. Based on the consideration of convergence speed and clustering performance, the Kmeans++ clustering algorithm is used to divide all samples into k clusters. Then, the US strategy is adopted to maximize the diversity of the selected test cases. Kmeans++ and US strategies can be easily extended or adapted to different data sets and application scenarios, and provide an efficient, high-quality and unbiased strategy for DNN testing, ensuring the diversity and wide coverage of image samples.

[0090] Figure 2 The following is a schematic diagram of an algorithm code for implementing a method for selecting a test and training sample based on semantic features provided by an embodiment of the present invention. Figure 2As shown in the figure, in the algorithm, a frequency-domain-based saliency detection method is used to extract semantic features of sample images (row 4). These methods are independent of the DNN model and avoid bias from feature extractors similar to DNNs. Frequency-domain-based methods can filter out imperceptible high-frequency noise and retain semantic information. Then, clustering methods are used to further explore potential diverse categories based on these semantic features (row 8). Finally, the US strategy is used to uniformly sample from each category (rows 9-16) to obtain the final selected sample image. After that, the final selected sample image can be used to test and retrain the DNN model. Through the above process, test cases with semantic diversity can be effectively identified and selected. These test cases can discover diverse model errors and help improve the robustness of DNNs during retraining, thereby improving the security of DNNs in real applications;

[0091] In order to evaluate the effectiveness of testing and retraining. Two data sets and six pre-trained models are used in the embodiment: CIFAR-10 data set (ResNet, VGG, DenseNet121 (Dens) and SVHN data set (ResNet50). In addition, the advanced adversarial attack technology PGD is also applied to generate candidate sets for test and retraining sample selection. A data set, model and attack result schematic diagram provided by the present invention are shown in FIG. Figure 3 shown.

[0092] Among them, DenseNetl21 is a neural network architecture that introduces dense connections between layers. DenseNet121 is a variant with 121 layers, which includes dense blocks in which each layer is connected to other layers in a feed-forward manner.

[0093] Furthermore, the parameters for generating adversarial samples and the parameters involved in the retraining process are analyzed in the embodiment. In order to generate adversarial samples on CIFAR-10 and SVHN, this embodiment uses the Torchattacks library to implement the PGD attack. For the CIFAR-10 dataset and the SVHN dataset, the maximum perturbation is set to eps=4 / 255 and the number of iterations is set to steps=10.

[0094] In order to select a suitable sampling rate to select samples to retrain the DNN model, the present invention sets different sampling rates for SVHN (ResNet50) and CIFAR-10 (ResNet18), and observes the changes in the robustness of the retrained models on adversarial data.

[0095] The method of the present invention more effectively improves robustness on both CIFAR-10 and SVHN datasets. However, increasing the number of training samples will increase training overhead. This embodiment sets sampling rates of 0.2 and 0.1 for the CIFAR-10 and SVHN datasets to balance performance and time overhead in subsequent experiments. In addition, in subsequent evaluations, in order to reduce the impact of randomness in sampling, this embodiment runs five experiments for each configuration and then reports the average value of the results. In addition, the present invention adjusts other hyperparameters to achieve the best effect.

[0096] The embodiment of the present invention implements a test and training sample selection method based on semantic features, uses a frequency domain-based saliency detection method to extract semantic features of test cases, and clusters the semantic features into different categories in the frequency domain feature space, which can more effectively detect various errors of DNN. In addition, after retraining, the adversarial robustness and OOD robustness of the model are effectively improved.

[0097] In summary, the embodiment of the present invention uses a test and training sample selection method based on semantic features and a significance detection method based on the frequency domain to extract the semantic features of the test case, which can capture the essential characteristics of the data consistent with human intuition and cluster them into different categories in the frequency domain feature space. The use of a significance detection method based on the frequency domain to extract the semantic features of the test case can more effectively detect various errors of DNN. In addition, after retraining, the adversarial robustness and OOD robustness of the model are effectively improved.

[0098] The results of the empirical study show that the frequency-domain-based saliency detection method of the present invention can effectively extract semantic information from perturbation samples. These results also emphasize the importance of frequency-domain analysis in saliency detection and semantic feature extraction. The results of saliency detection are not only helpful for understanding image content, but also provide valuable basis for DNN testing.

[0099] Those skilled in the art can understand that the accompanying drawings are only schematic diagrams of an embodiment, and the modules or processes in the accompanying drawings are not necessarily required to implement the present invention.

[0100] It can be known from the description of the above implementation methods that those skilled in the art can clearly understand that the present invention can be implemented by means of software plus a necessary general hardware platform. Based on such an understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which can be stored in a storage medium such as ROM / RAM, a magnetic disk, an optical disk, etc., and includes a number of instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in the various embodiments of the present invention or certain parts of the embodiments.

[0101] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device or system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment. The device and system embodiments described above are merely schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0102] The above is only a preferred specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed by the present invention should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

Claims

1. A method for selecting test and training samples based on semantic features, characterized in that: include: The semantic features of each sample image are extracted using a frequency-domain-based saliency detection method; Cluster and sample the semantic features of all sample images, divide the sample images into multiple categories with different semantic features, and obtain sample images for testing and retraining deep neural network DNN models; When the frequency domain-based saliency detection method adopts Fourier phase transform FPT, the method of extracting semantic features of each sample image using the frequency domain-based saliency detection method includes: Given an image matrix g with M rows and N columns, f(m, n) represents the pixel value of the mth row and nth column. The Fourier transform is used to transform the input image g from the spatial domain to the frequency domain. The Fourier transform F(u, v) of f(m, n) is shown in formula (5-1): The input image is converted from the frequency domain back to the spatial domain through the inverse fast Fourier transform IFFT, as shown in formula (5-2): Based on the fast Fourier transform FFT and IFFT, the Fourier phase transform FPT, high-pass filter HPF, residual spectrum SR and quaternion Fourier transform QFT are used to calculate the saliency image of the input image: (1) Fourier phase transform FPT: The phase spectrum of the image is calculated by taking the phase part through Fourier transform, as shown in formula (5-3); Among them, Im[F(u,v)] and Re[F(u,v] are the imaginary and real parts of F(u,v) respectively. By calculating IFFT on P(u,v), the saliency map S can be obtained. a (m,n), according to the saliency image S a (m,n) extracts the semantic features of the pixel value f(m,n).

2. The method according to claim 1, characterized in that The method of extracting semantic features of each sample image using a frequency domain-based saliency detection method includes: A sample image set is obtained, each sample image in the sample image set is selected in turn, the sample image is converted from the spatial domain to the frequency domain using a frequency domain-based saliency detection method, and semantic features are extracted from the sample image in the frequency domain, the semantic features including color, texture, shape and edge information.

3. The method according to claim 2, characterized in that The frequency domain-based saliency detection method includes: Fourier phase transform FPT, high pass filter HPF, residual spectrum SR and quaternary Fourier transform QFT.

4. The method according to claim 3, characterized in that: When the frequency domain-based saliency detection method uses a high-pass filter HPF, the method of extracting semantic features of each sample image using the frequency domain-based saliency detection method includes: Set the Sobel high-pass filter to include the edge matrix G for detecting horizontal x and the vertical edge matrix G y , as shown in formula (5-4): The horizontal gradient value V of the pixel value f(m,n) x (m,n) and the vertical gradient value V y (m,n) is calculated by the convolution operation in formula (5-5): V x (m,n)=f(m,n)*G x ,V y (m,n)=f(m,n)*G y (5-5) Calculate the saliency image S using formula (5-6) a (m,n): The saliency image S a (m,n) is used as the semantic feature of the pixel value f(m,n).

5. The method according to any one of claims 1 to 4, characterized in that: The method of clustering and sampling the semantic features of all sample images, dividing the sample images into multiple categories with different semantic features, and obtaining sample images for testing and retraining the DNN model includes: Extract semantic features from each image, where the semantic features are one of the attributes of color, texture, shape and edge information, form a feature space with the semantic features of all images, randomly select an element from the feature space as the first cluster center of the k-means++ algorithm, that is, take the image corresponding to the semantic features of the selected element as the first centroid, calculate the shortest distance from each element in the feature space to the existing cluster center, use the square of this distance as the probability of selecting the element as the new centroid, select the element corresponding to the maximum probability as the new centroid, repeat the above process until all centroids are selected, and divide the sample image into multiple categories with different semantic features; The uniform sampling (US) strategy is used to uniformly sample sample images of each category to obtain test case sample images for testing and retraining the DNN model.

Images