A ship image adversarial sample generation method based on neural network coverage

Abstract

This invention relates to the field of image adversarial example technology, and particularly to a method for generating adversarial examples of ship images based on neural network coverage, comprising the following steps: S1: Prepare ship satellite remote sensing image samples and obtain corresponding label information; S2: Select the first remote sensing image sample with obvious ship features, input it into a deep neural network model, and confirm the active activation paths of the intermediate layers of the model; S3: Select the original image sample without ship features, add ship features to form ship image samples, and extract neuron activation values; S4: Perform random adversarial attacks on the ship image samples to generate adversarial examples of ship images; S5: Obtain the input sample similarity index, the intermediate layer feature map difference index, and the neural network coverage index, and construct a multi-objective optimization search space; S6: Add the ship image adversarial examples to the multi-objective optimization search space and obtain the non-dominated solution set. This invention is of great significance for improving the recognition accuracy and robustness of ship image samples.

Description

Technical Field

[0001] This invention relates to the field of image adversarial sample technology, and in particular to a method for generating adversarial samples for ship images based on neural network coverage. Background Technology

[0002] Big data and artificial intelligence technologies have benefited from the rapid development of hardware technology, achieving revolutionary impacts in numerous technological fields such as autonomous driving, intelligent manufacturing, and image recognition. Deep learning, as a crucial component of artificial intelligence, frequently employs deep neural networks to solve many tasks in the field of intelligence, achieving remarkable results. Identifying and monitoring ship targets using remote sensing images is a key technology in the field of satellite imagery. With the establishment of satellite internet constellations and the increase in the number of satellites in orbit, the acquisition of remote sensing data is trending towards multi-platform, multi-angle, and multi-dimensional approaches, making intelligent methods based on deep neural networks particularly important in the field of ship identification using satellite imagery.

[0003] The implementation mechanism of deep neural networks differs from that of traditional software. They abstract, statistically analyze, and interpret the features of a dataset to identify key characteristics and match them with target labels. Due to their "black-box" nature, deep neural networks are highly vulnerable to both active and passive security threats. In image recognition, even subtle perturbations imperceptible to humans can cause deep neural networks to misjudge and output incorrect decisions. These image samples, yielding results contrary to expectations, are called adversarial examples, which can potentially lead to security risks in deep learning models. Furthermore, blind pixels or flash pixels in the sensor itself, or image data contamination caused by electromagnetic interference in the transmission link, can trigger unexpected results, causing satellite malfunctions or decision-making anomalies. Therefore, in-depth research into the generation principles and methods of adversarial examples is crucial for understanding the decision-making principles of deep neural network models, revealing the mechanisms by which adversarial attacks induce inherent vulnerabilities in models, and inspiring the construction of adversarial defense mechanisms.

[0004] To date, numerous research results have been achieved regarding adversarial example generation techniques in deep neural networks. Application CN116011558A, entitled "A Highly Transferable Adversarial Example Generation Method and System," includes the following steps: acquiring feature maps of the original image across multiple intermediate layers of a specified deep learning network; dividing all feature maps into positive and negative feature regions based on the activation values ​​and gradients of the intermediate layers; using a weighted sum of different regions from all specified intermediate layers as a loss function; processing the loss function using the momentum method to obtain adversarial examples for each iteration; and generating highly transferable adversarial examples after a preset number of iterations. The proposed highly transferable adversarial example generation method optimizes the reuse efficiency of adversarial examples in different application domains.

[0005] Application CN117274658A, entitled "A Method and Apparatus for Generating Adversarial Examples," discloses the following method: The correct category of a target image is obtained through a deep convolutional neural network; a first, second, and third sensitive semantic domains are obtained based on generalized class activation map methods, saliency map methods, and modified backpropagation methods; key semantic fusion domains are obtained by fusing coarse-grained and fine-grained key semantic pixels using a union method; an adversarial attack optimization model is obtained based on the key semantic fusion domain; and the optimal adversarial example of the optimized adversarial attack model is obtained using the Adam gradient descent algorithm. This effectively solves the problems of high computational cost and low efficiency. Although these inventions have significant effects in their respective fields, they mainly analyze and process the feature maps of the intermediate layers of the neural network, without deeply analyzing the activation states of neurons. Summary of the Invention

[0006] The purpose of this invention is to address the shortcomings of existing technologies by providing a method for generating adversarial examples for ship images based on neural network coverage. This method goes beyond simply analyzing the feature maps of the intermediate layers of the network; it delves deeper into the activation states of neurons. Using neural network coverage, image sample difference rate, and the feature map difference rate of the intermediate layers as guiding indicators, it constructs a search space for adversarial examples. Through a multi-objective optimization algorithm, this invention not only effectively obtains Pareto optimal solutions for adversarial examples but also ensures the optimality of the resulting set of adversarial example solutions. The method includes the following steps: S1: Prepare satellite remote sensing image samples of ships and obtain the label information corresponding to the remote sensing image samples; S2: Based on the label information, select the first remote sensing image sample with obvious ship features from the remote sensing image samples, input the first remote sensing image sample into the deep neural network model, and confirm the active activation paths of the intermediate layers of the model: S3: Select an original image sample from the remote sensing image samples that does not contain ship features, add several ship features to the original image sample to form a ship image sample, and extract the neuron activation values ​​under the active activation path; S4: Perform random adversarial attacks on the ship image samples to weaken ship features and generate several adversarial ship image samples. S5: Calculate the input sample similarity index, intermediate layer feature map difference index, and neural network coverage index based on the ship adversarial examples and the ship image samples. Construct a multi-objective optimization search space based on the input sample similarity index, the intermediate layer feature map difference index, and the neural network coverage index. S6: The adversarial sample of the ship image is added to the multi-objective optimization search space. The non-dominated solution set is obtained according to the Pareto optimality theory. If the non-dominated solution set is not satisfied, the process returns to step S4 and continues to execute. Otherwise, the non-dominated solution set is output.

[0007] Preferably, step S2 further includes: S21: Modify the source code of the target deep neural network model to obtain the deep neural network model; S22: The first remote sensing image sample is input into the deep neural network model to obtain the activation data of each neuron in the intermediate layer of the deep neural network model; S23: Obtain the number of intermediate layers and the size of the intermediate layers of the deep neural network model, and set the proportion of active neurons in each layer; S24: Read the neuron activation data under the first remote sensing image sample condition, and count the neuron information content of each neuron; S25: Determine the neuron activation path of each layer in the intermediate layer of the model based on the ratio of active neurons and the neuron information content.

[0008] Preferably, the step of statistically analyzing the neuronal information content of each neuron further includes: The formula for calculating the information content of a neuron is shown below: , in, This represents the set of all activation values ​​of a neuron under the conditions of the first remote sensing image sample. Represents neurons The information content, i.e., information entropy. This represents the activation value of a neuron under the conditions of the first remote sensing image sample, i.e. yes Any activation value in the list, This indicates that The probability function of activation values.

[0009] Preferably, step S3 further includes: S31: Add several ship features to the original image sample to form a ship image sample; S32: The ship image sample is input into the target depth model. Based on the recognition result, it is determined whether the ship image sample is successfully generated. If the recognition result is the same as the original image sample, the ship image sample generation fails and returns to step S31. If the recognition result is opposite to the original image sample, the ship image sample generation is successful. S33: Obtain the neuron activation values ​​of the ship image sample and the original image sample under the neuron activation path.

[0010] Preferably, in step S5, the calculation of the input sample similarity index further includes: S511: Load a ship image sample from the ship image samples and a ship image adversarial sample from the ship image adversarial samples; S512: Calculate the sample similarity between the certain ship image sample and the certain ship image adversarial sample. The formula for calculating the sample similarity is as follows: , in, For the image sample of a certain ship, This is an adversarial example of a certain ship image. The pixel value of a certain ship image sample. The pixel value of the adversarial sample in the image of a certain ship; S513: Record the sample similarity, i.e., input the sample similarity index.

[0011] Preferably, in step S5, the intermediate layer feature map difference index further includes: S521: Load a certain image sample from the ship image samples and a certain ship image adversarial sample from the ship image adversarial samples; S522: Generate an intermediate layer feature map based on the neuron activation values ​​under the adversarial sample conditions of the ship image; S523: Calculate the sample difference in neuron activation values ​​between a certain ship image sample and a certain ship image adversarial sample. The formula for calculating the sample difference is as follows:

[0012] in, For a certain ship image sample, This is an adversarial example of a certain ship image. This indicates the number of neurons in the neuron activation path. express and The degree of sample difference under the activation path The set of pixel values ​​for a given ship image sample. This is a set of adversarial examples for a specific ship image; S524: Record the sample difference degree, i.e., the difference degree index of the intermediate layer feature map.

[0013] Preferably, in step S5, the neural network coverage metric further includes: S531: Load the neuron coverage rate CoverRate of the current non-dominated solution set NDSet. Set the neuron coverage rate CoverRate to 0 for the first read. S532: Load the neuron activation values ​​of the ship image sample and the current non-dominated solution set NDSet; S533: Calculate the surprise sufficiency of the current non-dominated solution set NDSet under the ship image sample conditions using the sum-density estimation method. The formula for calculating the surprise value is as follows:

[0014] in, The number of neurons. To input samples under the current non-dominated solution set NDSet conditions Neuron activation values ​​in the layer For the ship image samples in The activation values ​​of neurons in the layer.

[0015] S534: To ensure that the surprise value of the current non-dominated solution set NDSet increases as the probability density decreases, the probability density is converted into sparseness. LSA is defined as the negative form of the logarithm of the density function, as shown in the following formula: .

[0016] S535: Based on the set of surprise values ​​obtained under the current non-dominated solution set NDSet, given an upper limit U, Divide into n SA segments, that is Then, the neuron coverage index of the current non-dominated solution set NDSet is calculated using the following formula: .

[0017] Preferably, in step S5, constructing the multi-objective optimization search space further includes: S541: The input sample similarity index, the intermediate layer feature map difference index, and the neural network coverage index are set as the three optimization objectives of the multi-objective search space, and the overall optimization function is:

[0018] in, It is a weighting coefficient, and , and ; S542: Define the first sub-objective function as the input similarity function of the samples, i.e. , The second sub-objective function is defined as the intermediate feature difference function of the samples, i.e. , The third sub-objective function is defined as the current coverage function of the samples, i.e. , S543: Substitute the first sub-objective function, the second sub-objective function, and the third sub-objective function into the optimization function, the calculation formula of which is: , In the constructed multi-objective optimization space, each of the first, second, and third sub-objective functions corresponds to an optimization direction in one dimension.

[0019] Preferably, step S6 includes: S61: Read the current non-dominated solution set NDSet. If this is the first time reading, the current non-dominated solution set NDSet is an empty set. S62: Read the adversarial sample of the ship image , causing the population to gather and set the construction set ; S63: From the constructed set Take an individual from Let the constructed set And make the dominating set ; S64: Construct the set China The dominant individuals are successively incorporated into the dominance set. In, and let the constructed set ; S65: If there exists an individual Belonging to the set of constructions And make If so, return to step S63 to continue execution; otherwise, It is incorporated into the non-dominated solution set NDSet.

[0020] Preferably, step S6 further includes: S66: If the constructed set If the set is empty, continue execution; otherwise, return to step S93. S67: If the termination condition of the multi-objective optimization is met, continue execution; otherwise, return to step S4. S68: Output the current non-dominated solution set NDSet.

[0021] Compared with existing technologies, this invention provides a more comprehensive and in-depth solution at both the theoretical and practical levels, providing a solid guarantee for the security and robustness of deep learning models.

[0022] Compared with the prior art, the beneficial effects of the present invention are: This invention uses neuron coverage as a core indicator to effectively guide the generation of high-quality adversarial examples, and is particularly suitable for complex satellite remote sensing environments.

[0023] This invention uses three objective optimization functions: input sample similarity, intermediate layer difference, and neuron coverage to ensure that adversarial examples maintain a high degree of similarity to the original input while maximizing the significant difference in the intermediate layer and the neuron coverage. This invention uses Pareto optimality theory as its theoretical basis to perform multi-objective optimization processing, accurately balancing various optimization objectives, thereby obtaining the optimal set of adversarial sample solutions. Attached Figure Description

[0024] Various other advantages and benefits will become apparent to those skilled in the art upon reading the following detailed description of preferred embodiments. The accompanying drawings are for illustrative purposes only and are not intended to limit the invention.

[0025] Figure 1 This is a flowchart of a method for generating adversarial examples for ship images based on neural network coverage, according to the present invention. Figure 2 This is a framework diagram of a method for generating adversarial examples for ship images based on neural network coverage, according to the present invention. Figure 3 This is a flowchart of the neuron activation path in a method for generating adversarial examples from ship images based on neural network coverage, according to the present invention. Figure 4 This is a flowchart of the activation values ​​under the neuron activation path in a method for generating adversarial examples of ship images based on neural network coverage according to the present invention. Figure 5 This is a flowchart illustrating the calculation of the coverage of the adversarial sample set in a method for generating adversarial samples from ship images based on neural network coverage, according to the present invention. Figure 6 This is a flowchart illustrating the acquisition of the non-dominated solution set of adversarial examples in a ship image adversarial example generation method based on neural network coverage according to the present invention. Detailed Implementation

[0026] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention. Obviously, the described embodiments are only some, not all, of the embodiments described in this application. All other embodiments obtained by those skilled in the art based on the embodiments in this application without creative effort are within the scope of protection of this application.

[0027] Those skilled in the art will understand that, unless otherwise stated, the singular forms “a” and “an” used herein, and “the”, may also include the plural forms. It should be further understood that the term “comprising” as used in this specification means the presence of the stated features, integers, steps, operations, elements, and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0028] Example Deep neural networks typically consist of multiple network layers, weight matrices, and convolutional kernels. They process and transmit information through numerous neurons. Different ship image samples will produce different neuron activation states, resulting in different decision outcomes. Adversarial examples, on the other hand, arise from minute differences in the input image, which are amplified by intermediate layers, causing discrepancies between the intermediate layer's feature map and the original feature map, leading to errors in the final result. Therefore, the adversarial example set generated by the method of this invention integrates features from different neural network coverage rates, greatly enriching the multidimensional features of the adversarial examples. Furthermore, the adversarial example set provides a Pareto optimal solution set, allowing for the selection of the most suitable adversarial examples from different perspectives.

[0029] Please see Figure 1 and Figure 2 As shown, in order to fully utilize the characteristics of neural network coverage in deep neural networks and explore efficient methods for generating adversarial examples, this embodiment provides a method for generating adversarial examples from ship images based on neural network coverage, including the following steps: S1: Prepare satellite remote sensing image samples of ships and obtain the corresponding tag information for the remote sensing image samples. Specifically, in this embodiment, the satellite remote sensing image samples of ships are real satellite remote sensing images, and the image sample set needs to include image samples with ship features and image samples without ship features. The tag information marks whether the satellite remote sensing image contains ship features and information such as the coordinates of the ship features and the type of ship. This tag information needs to be completed manually to ensure that the tag information is true and reliable. S2: Based on the label information, select the first remote sensing image sample with obvious ship features from the remote sensing image samples, input the first remote sensing image sample into the deep neural network model, and confirm the active activation paths of the intermediate layers of the model: S3: Select original image samples from remote sensing image samples that do not contain ship features, add several ship features to the original image samples to form ship image samples, and extract the activation values ​​of neurons under active activation paths; S4: Perform random adversarial attacks on ship image samples to weaken ship features and generate several adversarial ship image samples. S5: Based on the ship adversarial examples and ship image samples, calculate the input sample similarity index, intermediate layer feature map difference index, and neural network coverage index. Construct a multi-objective optimization search space based on the input sample similarity index, intermediate layer feature map difference index, and neural network coverage index. S6: The adversarial examples of ship images are added to the multi-objective optimization search space. The non-dominated solution set is obtained according to the Pareto optimality theory. If the non-dominated solution set is not satisfied, the process returns to step S4 and continues to execute. Otherwise, the non-dominated solution set is output.

[0030] Preferably, step S1 further includes: S11: Prepare a dataset of 100 images, including 90 satellite remote sensing images with ship features and 10 satellite remote sensing images without ship features. S12: Manually label the content of the satellite remote sensing image. The labeling information includes whether each sample contains a ship, the ship's location coordinates in the image, and the type of ship. S13: Place the labeled dataset in a fixed path for use in subsequent steps.

[0031] Furthermore, deep neural network models typically consist of multiple intermediate layers with a large number of neurons. However, the neurons containing the main feature information are generally relatively concentrated. To minimize computational complexity, step S2 identifies the active activation paths of the deep neural network model under typical ship satellite remote sensing image conditions. The specific definition of the neuron activation path is as follows: Given a deep neural network (DNN) and the original training sample set and a group of neurons The original training sample set consists of ship image samples, among which In the input sample In this case, neurons The activation value is For ordered subsets of neurons , express The neuron activation value vector in the data. The modulus equals At this time, it is called yes exist The neuronal activation trajectory on the [data / path]. For a set of inputs... ,definition As a group in Upper neuron activation trajectory, .

[0032] Please see Figure 3 As shown, step S2 further includes: S21: Modify the source code of the target deep neural network model to obtain the deep neural network model. Specifically, in this embodiment, the source code of the YOLOv5s model is modified, and a hook function is registered at the forward propagation function. The specific content includes obtaining the activation values ​​of neurons in each network layer during the forward propagation process and saving the activation values ​​to a specified path for subsequent calculation and analysis. The activation value data of each sample is saved as a list of multidimensional arrays, where each element in the list is the activation value of a network layer. This element is a multidimensional array that stores the activation values ​​of multiple channels of the network layer. S22: Input the first remote sensing image sample into the deep neural network model to obtain the activation data of each neuron in the intermediate layer of the deep neural network model. Specifically, in this embodiment, select 50 satellite remote sensing image samples with obvious ship features and input them into the YOLOv5s model in step S21 to obtain the neuron activation values ​​of the ship satellite remote sensing image under the YOLOv5s model. S23: Obtain the number of intermediate layers and the size of the intermediate layers of the deep neural network model, and set the proportion of active neurons in each layer. Specifically, in this embodiment, the number of intermediate network layers and the size of each intermediate layer of the YOLOv5 model are determined by the list length and the size of each list element. Based on the YOLOv5s input size and the number of intermediate layers, and taking into account the computational cost, the proportion of active neurons in each layer is set to 20%. S24: Read the neuron activation data under the first remote sensing image sample condition, and count the neuron information content of each neuron. Specifically, in this embodiment, read the first remote sensing image sample... Neuron activation data under specific conditions; S25: Based on the proportion of active neurons and the information content of neurons, determine the neuron activation path of each layer in the intermediate layer of the model. Specifically, in this embodiment, the information content of each neuron under all sample conditions is calculated according to the formula for calculating the information content of neurons, and the result is stored in a list variable.

[0033] Preferably, the neuronal information content of each neuron is statistically analyzed, further including: The formula for calculating the information content of a neuron is shown below: , in, This represents the set of all activation values ​​of a neuron under the conditions of the first remote sensing image sample. Represents neurons The information content, i.e., information entropy. This represents the activation value of a neuron under the conditions of the first remote sensing image sample. yes Any activation value in the list, This indicates that The probability function for activation values, specifically in this embodiment, involves statistically analyzing the indices of the 20% of neurons with the highest information content in each intermediate layer of the network, and then placing all active neurons in the intermediate layers into a new list variable to generate activation paths for the intermediate layer neurons.

[0034] Having followed the steps described above, the neuron activation paths in the target deep neural network model have been determined using satellite remote sensing images of the ship. These activation paths contain most of the information from the input sample images analyzed by the neural network model. Subsequent steps will then use these activation paths as a basis to analyze the differences between adversarial examples and the original samples, and select the optimal set of adversarial examples.

[0035] Please see Figure 4 As shown, step S3 further includes: S31: Several ship features are added to the original image sample to form a ship image sample. Specifically, in this embodiment, one satellite remote sensing image sample without ship features is arbitrarily selected as the original image sample. Select one prominent ship feature and add it to the original image sample. In the process, new image samples are generated. That is, satellite remote sensing image samples of ships; S32: Input the ship image sample into the target depth model. Determine whether the ship image sample was successfully generated based on the recognition result. If the recognition result is the same as the original image sample, the ship image sample generation fails, and return to step S31. If the recognition result is the opposite of the original image sample, the ship image sample generation is successful. Specifically, in this embodiment, the original image sample... Samples of satellite remote sensing images of ships Input the data into the YOLOv5s model and check if the recognition results are opposite. That is, the original image sample is recognized as having no ships, while the satellite remote sensing image sample of ships is recognized as having ships. If the recognition results are the same, it means that the ship sample is not well added and the ship features need to be selected again and added again until the recognition results of the original image sample and the satellite remote sensing image sample of ships are opposite. Record and save the current sample. S33: Obtain the neuron activation values ​​of the ship image samples and the original image samples under the neuron activation path. Specifically, in this embodiment, obtain the neuron activation values ​​of the current sample generated in step S32 under the YOLOv5s model, extract the neuron activation values ​​under the activation path, and repeat the above steps to generate a set of 10 sets of ship satellite remote sensing image samples.

[0036] The objective of this invention in generating adversarial examples is to maximize the difference in intermediate layer feature maps and the coverage of neurons, while minimizing input differences. Based on this idea, an adversarial example evaluation system is designed with "input sample similarity," "intermediate layer feature map difference," and "neural network coverage" as evaluation dimensions.

[0037] Furthermore, in step S4, the ship features in the ship satellite remote sensing image samples are weakened, and several temporary adversarial samples are randomly generated. The temporary image adversarial examples are ship image adversarial examples. Specifically, in this embodiment, 10 temporary adversarial examples are generated. This temporary adversarial example is a search element in the multi-objective optimization space.

[0038] Please see Figure 5 As shown, in step S5, the input sample similarity index is calculated, which further includes: S511: Load a ship image sample from the ship image sample set and a ship image adversarial sample from the ship image adversarial samples set. Specifically, in this embodiment, the temporary adversarial sample will be loaded. Input the target deep neural network model to obtain neuron activation data; S512: Calculate the sample similarity between a certain ship image sample and a certain ship image adversarial sample. The formula for calculating the sample similarity is as follows: , in, For a sample image of a certain ship, For an adversarial example of a certain ship image, For a sample image of a ship, the pixel value is... For a specific adversarial example of a ship image, the pixel values ​​are... ; S513: Record sample similarity, i.e., input sample similarity index. Specifically, in this embodiment, the sample similarity value in step S512 is recorded as the value of the first sub-objective function of multi-objective optimization.

[0039] Preferably, in step S5, the intermediate layer feature map difference index further includes: S521: Load a ship image sample from the ship image sample and a ship image adversarial sample from the ship image adversarial sample; S522: Generate intermediate layer feature maps based on the neuron activation values ​​under certain ship image sample and adversarial sample conditions. Specifically, in this embodiment, intermediate layer feature maps FeaR and FeaA are generated. S523: Calculate the sample difference in neuron activation values ​​between a sample of a ship image and an adversarial sample of a ship image. The formula for calculating the sample difference is as follows:

[0040] in, For a sample image of a certain ship, For an adversarial example of a certain ship image, This indicates the number of neurons in the set along the neuron activation path. express and The degree of sample difference under the activation path A set of pixel values ​​for a sample image of a ship. A set of adversarial examples for a specific ship image; S524: Record the sample difference degree, i.e. the difference degree index of the intermediate layer feature map. Specifically, in this embodiment, the value of the sample difference degree in step S523 is recorded as the value of the sub-objective function of the second multi-objective optimization.

[0041] Preferably, in step S5, the neural network coverage metric further includes: S531: Load the neuron coverage rate CoverRate of the current non-dominated solution set NDSet. Set the neuron coverage rate CoverRate read for the first time to be 0. Specifically, in this embodiment, since it is the first time, the current neuron coverage rate CoverRate is set to 0. If it is not the first time, load the neuron coverage rate of the previous cycle.

[0042] S532: Load the neuron activation values ​​of the ship image samples and the current non-dominated solution set NDSet. Specifically, in this embodiment, the ship image sample set is loaded for the first time. and adversarial sample set Obtain the neuron activation values ​​under the activation path in step S521.

[0043] S533: Calculate the surprise sufficiency of the current non-dominated solution set NDSet under the ship image sample conditions using the sum-density estimation method. The formula for calculating the surprise value is as follows:

[0044] in, The number of neurons. Given the current non-dominated solution set NDSet, the input sample is in Neuron activation pathway values ​​in the layer Given a sample of ship images, The activation values ​​of neurons in the layer.

[0045] S534: To ensure that the surprise value of the current non-dominated solution set NDSet increases as the probability density decreases, the probability density is converted into sparseness. The LSA is defined as the negative form of the logarithm of the density function, as shown in the following formula: .

[0046] S535: Based on the set of surprise values ​​obtained under the current non-dominated solution set NDSet, given an upper limit U, Divide into n SA segments, that is Then, the neuron coverage index for the current non-dominated solution set NDSet is calculated using the following formula: .

[0047] The core of multi-objective optimization problems is the multi-objective function. In adversarial example generation tasks, the corresponding objective function should be established according to the actual needs of the problem, namely the similarity of input samples, the difference of intermediate features, and the neuron coverage target.

[0048] Decomposition-based multi-objective optimization algorithms are the basic approach to solving multi-objective optimization problems. The core idea of ​​this method is to decompose the multi-objective optimization problem into a set of single-objective sub-problems or multiple multi-objective sub-problems, and then use the neighborhood relationship between the sub-problems to optimize all the sub-problems simultaneously through cooperation, thereby finding an approximation of the entire Pareto front.

[0049] Preferably, in step S5, constructing a multi-objective optimization search space further includes: S541: The input sample similarity index, the intermediate layer feature map difference index, and the neural network coverage index are set as the three optimization objectives in the multi-objective search space. The overall optimization function is:

[0050] in, It is a weighting coefficient, and , and .

[0051] S542: Define the first sub-objective function as the input similarity function of the samples, i.e. , The second sub-objective function is defined as the intermediate feature difference function of the samples, i.e. , The third sub-objective function is defined as the current coverage function of the samples, i.e. .

[0052] S543: Substitute the first, second, and third sub-objective functions into the optimization function. The formula for calculating the optimization function is as follows: , In the constructed multi-objective optimization space, each of the first, second, and third sub-objective functions corresponds to an optimization direction in one dimension. Specifically, in this embodiment, the index results of several temporary adversarial examples from the above steps are added to the multi-objective optimization set for subsequent steps to select the optimal solution set. 10 temporary adversarial examples Calculate the values ​​of the three sub-objective functions respectively. The calculated sub-objective function values ​​of the 10 temporary adversarial examples are added to the multi-objective optimization set.

[0053] The core idea of ​​Pareto optimality theory is to ensure that, without any loss of resources, all parties can increase their resources through reasonable allocation. This theory is based on the concept of dominance.

[0054] Wherein, dominance is defined as follows: Assuming a multi-objective optimization problem has a total of [number] constraints that can be satisfied. One feasible solution ,in and yes Two distinct solutions among a set of feasible solutions are considered to be mutually exclusive if they satisfy the following two conditions. Dominate ,use express.

[0055] (1) For all sub-objectives, Not compared Difference, that is ; (2) There exists at least one sub-objective such that Compare Okay. That is... ,make .

[0056] At this time, it is called It is either non-dominated or dominant; Dominated. Dominate The concept is represented by symbols. express.

[0057] Pareto theory is used to obtain the optimal set of adversarial sample solutions from a multi-objective optimization space. For individuals in the population undergoing optimization, To construct a set, under the initial conditions NDSet is a non-dominant set, initially empty, and its dominant set is... The initial conditions are empty. From Randomly select an individual and compare it with other individuals in turn. The individuals controlled by that individual will be removed from the list. If an individual is not dominated by any other individual, it is considered non-dominated and is added to the non-dominated set NDSet; otherwise, it is removed directly from the set. Delete the individual from the set, and so on, until a set is constructed. Empty.

[0058] Please see Figure 6 As shown, step S6 includes: S61: Read the current non-dominated solution set NDSet. If this is the first time reading, the current non-dominated solution set NDSet is empty. Specifically, in this embodiment, since this is the first time, the non-dominated solution set NDSet is set to empty. If this is not the first time reading, the non-dominated solution set is set to empty. ; S62: Adversarial Examples for Reading Ship Images , causing the population to gather and set the construction set Specifically, in this embodiment, the set of function values ​​of the temporary adversarial sample is read. and set the constructor set and ; S63: Constructing Sets Take an individual from Let the construct set And make the dominating set ; S64: Construct a set China Dominant individuals are successively incorporated into the dominance set. In, and let construct a set ; S65: If there exists an individual Belongs to the constructed set And make If so, return to step S63 to continue execution; otherwise, It is incorporated into the non-dominated solution set NDSet.

[0059] Preferably, step S6 further includes: S66: If construct a set If the set is empty, continue execution; otherwise, return to step S63. S67: If the termination condition of multi-objective optimization is met, continue execution; otherwise, return to step S54. S68: Output the current non-dominated solution set NDSet.

[0060] The above description is merely a preferred embodiment of the present invention. The scope of protection of the present invention is not limited to the above embodiments. All technical solutions falling within the scope of the present invention's concept are within the scope of protection of the present invention. It should be noted that for those skilled in the art, any improvements and modifications made without departing from the principles of the present invention should also be considered within the scope of protection of the present invention.

Claims

1. A method for generating adversarial examples for ship images based on neural network coverage, characterized in that, Includes the following steps: S1: Prepare satellite remote sensing image samples of ships and obtain the label information corresponding to the remote sensing image samples; S2: Based on the label information, select the first remote sensing image sample with obvious ship features from the remote sensing image samples, input the first remote sensing image sample into the deep neural network model, and confirm the active activation paths of the intermediate layers of the model: S3: Select an original image sample from the remote sensing image samples that does not contain ship features, add several ship features to the original image sample to form a ship image sample, and extract the neuron activation values ​​under the active activation path; S4: Perform random adversarial attacks on the ship image samples to weaken ship features and generate several adversarial ship image samples. S5: Calculate the input sample similarity index, intermediate layer feature map difference index, and neural network coverage index based on the ship adversarial examples and the ship image samples. Construct a multi-objective optimization search space based on the input sample similarity index, the intermediate layer feature map difference index, and the neural network coverage index. S6: The adversarial sample of the ship image is added to the multi-objective optimization search space, and a non-dominated solution set is obtained according to Pareto optimality theory. If the non-dominated solution set is not satisfied, the process returns to step S4 to continue execution; otherwise, the non-dominated solution set is output. Step S6 includes: S61: Read the current non-dominated solution set NDSet. If this is the first time reading, the current non-dominated solution set NDSet is an empty set. S62: Read the adversarial sample of the ship image , causing the population to gather and set the construction set ; S63: From the constructed set Take an individual from Let the constructed set And make the dominating set ; S64: Construct the set China The dominant individuals are successively incorporated into the dominance set. In, and let the constructed set ; S65: If there exists an individual Belonging to the set of constructions And make If so, return to step S63 to continue execution; otherwise, Incorporate it into the non-dominated solution set NDSet; Step S6 further includes: S66: If the constructed set If the set is empty, continue execution; otherwise, return to step S63. S67: If the termination condition of the multi-objective optimization is met, continue execution; otherwise, return to step S4. S68: Output the current non-dominated solution set NDSet.

2. The method for generating adversarial examples for ship images based on neural network coverage according to claim 1, characterized in that, Step S2 further includes: S21: Modify the source code of the target deep neural network model to obtain the deep neural network model; S22: The first remote sensing image sample is input into the deep neural network model to obtain the activation data of each neuron in the intermediate layer of the deep neural network model; S23: Obtain the number of intermediate layers and the size of the intermediate layers of the deep neural network model, and set the proportion of active neurons in each layer; S24: Read the neuron activation data under the first remote sensing image sample condition, and count the neuron information content of each neuron; S25: Determine the neuron activation path of each layer in the intermediate layer of the model based on the ratio of active neurons and the neuron information content.

3. The method for generating adversarial examples for ship images based on neural network coverage according to claim 2, characterized in that, The statistical analysis of the neuronal information content of each neuron further includes: The formula for calculating the information content of a neuron is shown below: ， in, This represents the set of all activation values ​​of a neuron under the conditions of the first remote sensing image sample. Represents neurons The information content, i.e., information entropy. This represents the activation value of a neuron under a certain sample condition in the first remote sensing image sample, i.e. yes Any activation value in the list, This indicates that The probability function of activation values.

4. The method for generating adversarial examples for ship images based on neural network coverage according to claim 3, characterized in that, Step S3 further includes: S31: Add several ship features to the original image sample to form a ship image sample; S32: The ship image sample is input into the target depth model. Based on the recognition result, it is determined whether the ship image sample is successfully generated. If the recognition result is the same as the original image sample, the ship image sample generation fails and returns to step S31. If the recognition result is opposite to the original image sample, the ship image sample generation is successful. S33: Obtain the neuron activation values ​​of the ship image sample and the original image sample under the neuron activation path.

5. The method for generating adversarial examples for ship images based on neural network coverage according to claim 4, characterized in that, In step S5, the calculation of the input sample similarity index further includes: S511: Load a ship image sample from the ship image samples and a ship image adversarial sample from the ship image adversarial samples; S512: Calculate the sample similarity between the certain ship image sample and the certain ship image adversarial sample. The formula for calculating the sample similarity is as follows: ， in, For a certain ship image sample, This is an adversarial example of a certain ship image. The pixel value of a certain ship image sample. The pixel value of the adversarial sample in the image of a certain ship; S513: Record the sample similarity, i.e., input the sample similarity index.

6. The method for generating adversarial examples for ship images based on neural network coverage according to claim 5, characterized in that, In step S5, the intermediate layer feature map difference index further includes: S521: Load a ship image sample from the ship image samples and a ship image adversarial sample from the ship image adversarial samples; S522: Generate an intermediate layer feature map based on the neuron activation values ​​under the adversarial sample conditions of the ship image; S523: Calculate the sample difference in neuron activation values ​​between a certain ship image sample and a certain ship image adversarial sample. The formula for calculating the sample difference is as follows: in, For a certain ship image sample, This is an adversarial example of a certain ship image. This indicates the number of neurons in the neuron activation path. express and The degree of sample difference under the activation path The set of pixel values ​​for a given ship image sample. This is a set of adversarial examples for a specific ship image; S524: Record the sample difference degree, i.e., the difference degree index of the intermediate layer feature map.

7. The method for generating adversarial examples for ship images based on neural network coverage according to claim 6, characterized in that, In step S5, the neural network coverage metric further includes: S531: Load the neuron coverage rate CoverRate of the current non-dominated solution set NDSet. Set the neuron coverage rate CoverRate to 0 for the first read. S532: Load the neuron activation values ​​of the ship image sample and the current non-dominated solution set NDSet; S533: Calculate the surprise sufficiency of the current non-dominated solution set NDSet under the ship image sample conditions using the sum-density estimation method. The formula for calculating the surprise value is as follows: in, The number of neurons. To input samples under the current non-dominated solution set NDSet conditions Neuron activation values ​​in the layer For a certain sample under the condition of the ship image sample, Neuron activation values ​​in the layer; S534: To ensure that the surprise value of the current non-dominated solution set NDSet increases as the probability density decreases, the probability density is converted into sparseness. LSA is defined as the negative form of the logarithm of the density function, as shown in the following formula: ； S535: Based on the set of surprise values ​​obtained under the current non-dominated solution set NDSet, given an upper limit U, Divide into n SA segments, that is Then, the neuron coverage index of the current non-dominated solution set NDSet is calculated using the following formula: 。 8. The method for generating adversarial examples for ship images based on neural network coverage according to claim 7, characterized in that, In step S5, the construction of the multi-objective optimization search space further includes: S541: The input sample similarity index, the intermediate layer feature map difference index, and the neural network coverage index are set as the three optimization objectives of the multi-objective search space, and the overall optimization function is: in, It is a weighting coefficient, and , and ; S542: Define the first sub-objective function as the input similarity function of the samples, i.e. ， The second sub-objective function is defined as the intermediate feature difference function of the samples, i.e. ， The third sub-objective function is defined as the current coverage function of the samples, i.e. ， S543: Substitute the first sub-objective function, the second sub-objective function, and the third sub-objective function into the optimization function, the calculation formula of which is: ， In the constructed multi-objective optimization space, each of the first, second, and third sub-objective functions corresponds to an optimization direction in one dimension.

Images