A method and system for generating image adversarial samples based on joint loss function

Abstract

The present invention discloses a method and system for generating image adversarial samples based on a joint loss function. The method comprises obtaining image samples and random noise samples; inputting the image samples into a generative network to obtain adversarial samples and constructing a first loss function; inputting the image samples and adversarial samples into a discriminative network to construct a second loss function; inputting the adversarial samples into a model to be attacked to construct a third loss function based on a detection frame; inputting the image samples into the model to be attacked to obtain attention weights, obtaining feature tensors of adversarial samples and random noise samples at different scales through an image pyramid, and constructing a fourth loss function based on them; constructing a joint loss function based on the aforementioned loss functions to update the weights of the generative network and the discriminative network; repeating the above steps until the optimal generative network is obtained, and inputting the image samples into it to obtain the final adversarial sample. The present invention does not require obtaining the structure of the model to be attacked, and has the advantages of a short adversarial sample generation cycle and strong attack migration.

Description

Technical Field

[0001] The present invention relates to the technical field of image recognition data processing, and in particular to a method and system for generating image adversarial samples based on a joint loss function. Background Art

[0002] In the field of object detection based on convolutional neural networks, adversarial examples are image samples that are subtly perturbed by the original input, making them difficult for the human eye to recognize but causing the corresponding object detection model to generate a high-confidence error output. In practice, these adversarial examples can make the corresponding object detection models extremely vulnerable.

[0003] Researching how to generate adversarial examples and use them to train object detection models is an important approach to improving the stability of object detection models. Existing adversarial examples are primarily generated through the following process: First, the internal structure of the target detection model to be attacked must be obtained. Next, the image output is obtained, and a loss function is calculated using perturbation constraints. Backpropagation of the target detection model's gradients is performed, and pixel gradients are calculated and iteratively updated to generate adversarial examples.

[0004] However, the above-mentioned method of generating adversarial samples still has the following obvious defects in actual use: First, the generation of the adversarial samples needs to be based on the internal structure of the model to be attacked. In most practical cases, the internal structure of the model to be attacked is not easy to obtain. This makes it difficult to generate the adversarial samples. Secondly, this type of adversarial sample obtained by obtaining the internal structure of the model to be attacked in advance often only has an attack effect on the model, and often has the problem of low attack effectiveness when performing transfer attacks. Furthermore, the gradient backpropagation method requires multiple iterations when calculating the gradient changes of image pixels, which leads to a long generation cycle of adversarial samples and the inability to process real-time data streams. Summary of the Invention

[0005] The purpose of the present invention is to provide a method and system for generating image adversarial samples based on a joint loss function, so as to improve the technical problems that the existing image adversarial sample generation method needs to be based on the internal structure of the model to be attacked; the generation process has a long cycle and cannot process real-time data; and the generated adversarial samples have poor attack transferability.

[0006] To achieve the above objectives, the present invention proposes the following technical solutions:

[0007] A method for generating image adversarial samples based on a joint loss function, comprising:

[0008] Obtain random noise samples and several image samples of the model to be attacked;

[0009] The image sample is input into the generative network to generate an adversarial sample, and a first loss function is constructed based on the difference between the adversarial sample and the image sample; then, the image sample and the adversarial sample are respectively input into the discriminative network to construct a second loss function based on the output prediction result and the corresponding true label;

[0010] Inputting the adversarial sample into the model to be attacked to construct a third loss function based on the output label information; inputting the image sample into the model to be attacked to obtain the corresponding attention weight, and obtaining the feature tensors of the adversarial sample and the random noise sample at different scales through the image pyramid, so as to construct a fourth loss function based on the attention weight, the feature tensors of the adversarial sample at different scales, and the feature tensors of the random noise sample at different scales;

[0011] Constructing a joint loss function by using the first loss function, the second loss function, the third loss function, and the fourth loss function to update the weight parameters of the generator network and the discriminator network respectively;

[0012] The above steps are repeated until the optimized generative network is obtained, and the image sample is input into it to obtain the final adversarial sample.

[0013] Furthermore, the expression of the fourth loss function is:

[0014]

[0015] Among them, E I represents the expected value, M represents the total number of feature layers of different scales in the image pyramid, m = 1, 2, ..., M represents the feature layers of different scales in turn; X m R represents the feature tensor calculated on the mth feature layer of the adversarial sample; m A represents the feature tensor calculated by the random noise sample on the mth feature layer; m is the attention weight of the image sample, where the weight of the pixels inside the detection box is greater than the weight of the pixels outside the detection box.

[0016] Furthermore, the attention weight is obtained through the following steps:

[0017] Get all the detection frames in any of the image samples, and define the confidence level of the kth detection frame containing the target to be identified as S k , and assign it to the weight of the kth pixel in the detection frame; correspondingly, the weight of the kth pixel outside the detection frame is 0;

[0018] Accumulate the weights of all pixels within the detection frames; when there is a detection frame intersection, take the maximum weight as the weight of the pixel;

[0019] The accumulated weights are normalized to obtain the attention weights.

[0020] Furthermore, the expression of the joint loss function is:

[0021]

[0022] in,

[0023] is the first loss function, and its expression is:

[0024]

[0025] Among them, |p n | represents the pixel distance between the image sample and the n-th pixel in the adversarial sample;

[0026] L D is the second loss function, and its expression is:

[0027]

[0028] Where t is the true label of the image sample or the adversarial sample, the image sample is a positive sample, recorded as 1, and the adversarial sample is a negative sample, recorded as 0; is the probability that the discriminator outputs a positive sample;

[0029] L dect is the third loss function, and its expression is:

[0030] L dect =λ obj L obj (Z|O,σ)+λ loc L loc (Z|O,σ)+λ prob L prob (Z|O,σ);

[0031] Among them, L obj is the confidence loss term, λ obj is the confidence loss coefficient; L loc is the detection box position loss term, λ loc is the detection frame position loss coefficient; L prob is the detection category loss, λ prob is the detection category loss coefficient; Z is the adversarial sample, σ is the parameter of the model to be attacked; O = {o z |P z =0, 1≤z≤S} represents artificially preset false labels, where z represents the ID of the detection box, o zRepresents the coordinate position and category probability of the detection box z, P z =0 indicates that the confidence level of the detection box z in correctly marking the target to be identified is 0, and S is the total number of targets to be identified.

[0032] Furthermore, constructing a second loss function based on the output prediction result and the true label includes:

[0033] Inputting the image sample and the adversarial sample into the discriminant network; wherein the discriminant network includes five convolutional layers in sequence, and a BatchNorm layer is added to the middle three convolutional layers;

[0034] Perform layer-by-layer downsampling forward propagation based on each of the convolutional layers, and output the discrimination results of the image sample and the adversarial sample through a fully connected layer to form a prediction result; wherein the first four convolutional layers use the LeakyReLU activation function, and the last convolutional layer uses the Sigmoid activation function;

[0035] A second loss function is constructed based on the predicted results and the corresponding true labels.

[0036] Furthermore, inputting the image sample into a generative network to generate an adversarial sample includes:

[0037] Performing low-dimensional feature representation on the image sample through an encoder, and performing feature extraction based on each convolutional layer;

[0038] The extracted features are mapped to the sample label space through the decoder to generate perturbations;

[0039] The generation network includes eight convolutional layers, and the first seven convolutional layers use the LeakyReLU activation function, and the last convolutional layer uses the Tanh activation function;

[0040] The perturbation amount is added to the image sample to obtain the adversarial sample.

[0041] A system for generating image adversarial samples based on a joint loss function, comprising:

[0042] The first acquisition module is used to obtain random noise samples and a number of image samples of the model to be attacked;

[0043] A generative adversarial module is configured to input the image sample into a generative network to generate an adversarial sample, and construct a first loss function based on the difference between the adversarial sample and the image sample; and input the image sample and the adversarial sample into a discriminative network respectively to construct a second loss function based on the output prediction result and the true label;

[0044] A first construction module is configured to input the adversarial sample into the model to be attacked to construct a third loss function based on the output detection box; input the image sample into the model to be attacked to obtain the corresponding attention weight, and obtain the feature tensors of the adversarial sample and the random noise sample at different scales through the image pyramid to simultaneously construct a fourth loss function based on them;

[0045] A second construction module is used to construct a joint loss function through the first loss function, the second loss function, the third loss function and the fourth loss function, so as to update the weight parameters of the generator network and the discriminator network respectively;

[0046] The iterative optimization module is used to repeatedly call the above modules until the optimized generation network is obtained, and input the image sample into it to obtain the final adversarial sample.

[0047] Furthermore, the first building block includes:

[0048] The second acquisition module is used to obtain all detection frames in any of the image samples, and define the confidence level of the kth detection frame containing the target to be identified as S k , and assign it to the weight of the kth pixel in the detection frame; correspondingly, the weight of the kth pixel outside the detection frame is 0;

[0049] A weight accumulation module, configured to accumulate the weights of all pixels within the detection frames; wherein, when there is an intersection of the detection frames, the maximum weight is taken as the weight of the pixel;

[0050] A normalization processing module is used to normalize the accumulated weights to obtain the attention weights.

[0051] Furthermore, the generative adversarial module includes a generation module, and the generation module includes:

[0052] A first extraction module, configured to perform low-dimensional feature representation on the image sample through an encoder and perform feature extraction based on each convolutional layer;

[0053] A perturbation generation module is used to map the extracted features into the sample label space through the decoder to generate perturbations;

[0054] A first output module is configured to add the perturbation amount to the image sample to obtain the adversarial sample.

[0055] Furthermore, the generation adversarial module includes an adversarial module, and the adversarial module includes:

[0056] A second extraction module is configured to input the image sample and the adversarial sample into the discriminant network, wherein the discriminant network includes five convolutional layers in sequence, and a BatchNorm layer is added to the middle three convolutional layers;

[0057] A discrimination module, configured to perform layer-by-layer downsampling forward propagation based on each of the convolutional layers, and output the discrimination results of the image sample and the adversarial sample through a fully connected layer to form a prediction result; wherein the first four convolutional layers use a LeakyReLU activation function, and the last convolutional layer uses a Sigmoid activation function;

[0058] The second output module is used to construct a second loss function based on the prediction result and the corresponding true label.

[0059] Beneficial effects:

[0060] It can be seen from the above technical solutions that the technical solution of the present invention provides an image adversarial sample generation method based on a joint loss function to improve the various defects existing in the existing adversarial sample generation process.

[0061] The method includes: first, obtaining image samples of the model to be attacked and random noise samples. Then, an adversarial sample is obtained based on the generative network. Based on the adversarial sample, the generative network, the discriminative network, and the model to be attacked sequentially obtain a first loss function, a second loss function, a third loss function, and a fourth loss function. Furthermore, a joint loss function is constructed based on the above loss functions to update the generative network and the discriminative network. Finally, the above process is repeated to optimize the generative network and obtain the final adversarial sample.

[0062] The method uses the model to be attacked as a participant in the entire adversarial sample generation process and combines it with the generative network and the discriminative network. The loss functions corresponding to the generative network, the discriminative network, and the model to be attacked are obtained respectively, and then a joint loss function is constructed. At this time, when generating adversarial samples, it is only necessary to iteratively optimize the generative network gradually with the goal of minimizing the joint loss function. Compared with the existing technology, it is no longer necessary to obtain the internal structure of the model to be attacked, making the entire method more feasible. Moreover, only one forward propagation is required in each round of iteration to generate the adversarial sample for that round. Compared with the existing technology, there is no need for repeated iterative gradient backpropagation calculations, thereby saving the generation cycle of each round of adversarial samples and having the ability to process real-time data streams.

[0063] And because the joint loss function is related to the adversarial sample generation stage, the adversarial sample discrimination stage, and the adversarial sample recognition stage, when the generative network is iteratively trained based on the joint loss function, the iterative error can always be minimized, thereby achieving optimization faster.

[0064] Specifically, for the first loss function, since the perturbation amount will gradually decrease in continuous training iterations, the modification of the adversarial sample relative to the image sample can be gradually reduced, ensuring the smoothness of the adversarial sample. For the second loss function, the optimization process of the generative network is also related to the mutual adversarial iteration between it and the discriminant network, thereby promoting the optimization of the generative network. Since the image features of the image samples are extracted based on the convolutional layers in the convolutional neural network, their size cannot be processed. However, when the target detection model performs target detection, it is based on both high-resolution low-level features and high-level features with high semantic information.

[0065] The third loss function, when generating the adversarial example, uses an image pyramid to process features at different scales. This allows the generation of the adversarial example to take both of these factors into account, thereby increasing the adversarial example's effectiveness against the target model. The third loss function is constructed based on the detection box information of the first recognition example, effectively suppressing the appearance of the detection box, thereby reducing the probability of the adversarial example being recognized in the final recognition phase.

[0066] The fourth loss function introduces random noise samples and processes them at various scales based on the image pyramid, making the adversarial examples approach random noise in both low-level and high-level features. Attention weights are generated based on the first recognition samples at different scales and constructed by combining feature tensors of the adversarial samples and random noise samples at different scales. This alters the texture of the target to be identified in the adversarial example, making it as close to the background image as possible. This improves the success rate of the generated adversarial example against unknown models and increases the attack portability of the adversarial example, making it highly effective against any target detection model.

[0067] It should be appreciated that all combinations of the foregoing concepts, as well as additional concepts described in greater detail below, to the extent such concepts are not mutually inconsistent, can be considered to be part of the inventive subject matter of this disclosure.

[0068] The foregoing and other aspects, embodiments, and features of the present invention will be more fully understood from the following description in conjunction with the accompanying drawings. Other additional aspects of the present invention, such as features and / or beneficial effects of the exemplary embodiments, will become apparent from the following description or through practice of specific embodiments according to the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0069] The accompanying drawings are not intended to be drawn to scale. In the drawings, each identical or nearly identical component shown in various figures may be represented by the same reference numeral. For clarity, not every component is labeled in every figure. Embodiments of various aspects of the present invention will now be described by way of example and with reference to the accompanying drawings, in which:

[0070] Figure 1 This is a flowchart of the adversarial sample generation method according to the present invention;

[0071] Figure 2 for Figure 1 Flowchart of the generation of adversarial samples in

[15] ;

[0072] Figure 3 for Figure 1 Flowchart for constructing the first loss function;

[0073] Figure 4 for Figure 1 Flowchart for constructing the attention weight in the fourth loss function described in . DETAILED DESCRIPTION

[0074] To further clarify the objectives, technical solutions, and advantages of the embodiments of the present invention, the technical solutions of the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the described embodiments are only a portion of the embodiments of the present invention, not all of them. Based on the described embodiments of the present invention, all other embodiments derived by persons of ordinary skill in the art without requiring creative effort are within the scope of protection of the present invention. Unless otherwise defined, technical or scientific terms used herein shall have the ordinary meanings understood by persons of ordinary skill in the field to which the present invention pertains.

[0075] The words "first", "second" and similar terms used in the patent application specification and claims of the present invention do not indicate any order, quantity or importance, but are only used to distinguish different components. Similarly, unless the context clearly indicates otherwise, the singular forms of "a", "an" or "the" and similar terms do not indicate a quantitative limitation, but rather indicate the presence of at least one. Words such as "include" or "comprise" mean that the elements or objects appearing before "include" or "comprise" include the features, wholes, steps, operations, elements and / or components listed after "include" or "comprise", and do not exclude the existence or addition of one or more other features, wholes, steps, operations, elements, components and / or their collections. "Up", "down", "left", "right" and the like are only used to indicate relative positional relationships. When the absolute position of the described object changes, the relative positional relationship may also change accordingly.

[0076] The existing adversarial example generation process requires prior knowledge of the internal structure of the model to be attacked and employs multiple backpropagation calculations during each update. This results in a long adversarial example generation cycle, difficulty processing real-time data streams, and inability to perform transfer attacks. Therefore, the present invention aims to provide a method for generating image adversarial examples based on a joint loss function to address these shortcomings of existing adversarial example generation.

[0077] Example 1

[0078] like Figure 1 As shown, the method includes:

[0079] Step S102: Obtain several image samples of the model to be attacked and construct random noise samples.

[0080] In a specific implementation, the image samples are images of different types and sizes corresponding to the target scene, and can be obtained by artificial creation or from the input database of the model to be attacked.

[0081] The random noise sample is a randomly generated noise image.

[0082] Step S104: input the image sample into the generative network to generate an adversarial sample, and construct a first loss function based on the difference between the adversarial sample and the image sample; input the image sample and the adversarial sample into the discriminative network respectively to construct a second loss function based on the output prediction result and the true label.

[0083] In practice, the generative network was implemented using the deep learning framework Pytorch. The optimization process was performed using a GTX 3090 graphics processing unit (GPU) with an initial learning rate of 0.0001. Because the convolution kernels of the generative network are fully convolutional and trained on image patches, the size of the adversarial examples is arbitrary.

[0084] like Figure 2 As shown in FIG, as a specific implementation, the generation of the adversarial sample includes:

[0085] Step S202: Perform low-dimensional feature representation on the image sample through an encoder, and perform layer-by-layer downsampling and forward propagation based on each convolutional layer to perform feature extraction.

[0086] In this step, the feature tensor size after each convolutional layer is the same.

[0087] Step S204: Map the extracted features into the sample label space through the decoder to generate a perturbation amount.

[0088] The generative network includes eight convolutional layers, the first seven of which use the LeakyReLU activation function, and the last convolutional layer uses the Tanh activation function. The LeakyReLU activation function is used to increase the convergence speed during iterative optimization.

[0089] Step S206: Add the perturbation amount to the image sample to obtain the adversarial sample.

[0090] In this step, the first loss function is used to reduce the modification amount of the adversarial sample relative to the image sample in continuous training iterations, thereby ensuring the smoothness of the adversarial sample.

[0091] In this embodiment, the first loss function is specifically similarity loss. Common similarity losses include l0 loss, l1 loss, l2 loss and l ∞ Considering the convenience of updating the joint loss function described later, the l2 loss is specifically adopted.

[0092] As a specific implementation method, the expression of the first loss function is:

[0093]

[0094] Among them, |p n | represents the pixel distance between the image sample and the nth pixel in the adversarial sample.

[0095] In this step, the optimization process of the generative network is also related to the mutual antagonism promotion between the generative network and the discriminative network, thereby promoting the optimization of the generative network.

[0096] During the optimization iteration, the Adam gradient descent algorithm is performed twice on the discriminant network, and its initial learning rate is set to 0.0002.

[0097] Since the output layer activation functions of the generator network and the discriminator network are the Tanh function and the Sigmoid function, respectively, both of which are S-shaped functions, there is a drawback that the convergence rate is reduced during the subsequent iterative optimization based on the backpropagation algorithm. Based on this, as a specific embodiment, the second loss function is set to a binary cross entropy loss to solve the gradient vanishing problem caused by the Sigmoid function.

[0098] At this time, the expression of the second loss function is:

[0099]

[0100] Wherein, t is the prediction result of the image sample or the adversarial sample, is the probability that the predicted result is the same as the true label.

[0101] like Figure 3 As shown, as a specific implementation, the second loss function includes:

[0102] Step S402: input the image sample and the adversarial sample into the discriminant network; wherein the discriminant network includes five convolutional layers in sequence, and a BatchNorm layer is added to the middle three convolutional layers.

[0103] In this step, the BatchNorm layer is used to perform batch normalization of the data, and the length and width of the feature tensor obtained by the image sample after passing through the first three convolutional layers are halved.

[0104] Step S404: Perform layer-by-layer downsampling forward propagation based on each of the convolutional layers, and output the discrimination results of the image sample and the adversarial sample through a fully connected layer to form a prediction result; wherein, the first four convolutional layers use the LeakyReLU activation function, and the last convolutional layer uses the Sigmoid activation function.

[0105] In this step, since the first four convolutional layers use the LeakyReLU activation function, it is beneficial to accelerate the convergence speed during iterative optimization and prevent gradient sparseness.

[0106] Step S406: Construct a second loss function based on the prediction result and the true label of the corresponding image sample or adversarial sample.

[0107] Step S106: Input the adversarial sample into the model to be attacked to construct a third loss function based on the output detection box; input the image sample into the model to be attacked to obtain the corresponding attention weight, and obtain the feature tensors of the adversarial sample and the random noise sample at different scales through the image pyramid to simultaneously construct a fourth loss function based on them.

[0108] Generally, the generation of adversarial samples is divided into two types: untargeted and targeted. The specific type targeted in this embodiment is the targeted type. For the targeted type, the category output by the generated adversarial sample after forward propagation of the target detection model is the specified category, and the position of the output detection box can also be controlled. Based on this, as a specific implementation method, the purpose of the adversarial sample is to reduce the success rate of target detection. Therefore, this embodiment specifically sets a third loss function to make all detection boxes invalid.

[0109] At this time, the expression of the third loss function is:

[0110] L dect =λ obj Lobj (z|O,σ)+λ loc L loc (z|O,σ)+λ prob L prob (z|O,σ);

[0111] Among them, L obj is the confidence loss term, λ obj is the confidence loss coefficient; L loc is the detection box position loss term, λ loc is the detection frame position loss coefficient; L prob is the detection category loss, λ prob is the detection category loss coefficient; Z is the adversarial sample, σ is the parameter of the model to be attacked; O = {o z |P z =0, 1≤z≤S} represents artificially preset false labels, where z represents the ID of the detection box, o z Represents the coordinate position and category probability of the detection box z, P z =0 indicates that the confidence level of the detection frame z correctly marking the target to be identified is 0, and S is the total number of targets to be identified. In this embodiment, the false label is used to make the target to be identified be identified as background.

[0112] As another specific implementation, in order to modify image samples at feature layers of different scales so that the generated adversarial samples differ from the image samples at feature layers of each size, the fourth loss function is set to a multi-scale attention feature loss.

[0113] At this time, the expression of the fourth loss function is:

[0114]

[0115] Among them, E I represents the expected value, M represents the total number of feature layers of different scales in the image pyramid, m = 1, 2, ..., M represents the feature layers of different scales in turn; X m R represents the feature tensor calculated on the mth feature layer of the adversarial sample; m A represents the feature tensor calculated by the random noise sample on the mth feature layer; m is the attention weight of the image sample, where the weight of the pixels inside the detection box is greater than the weight of the pixels outside the detection box.

[0116] At this time, the fourth loss function will cause the feature tensor X of the adversarial sample to mThis approach is close to random perturbation, thereby changing the characteristic texture of the target to be identified. At this point, because the adversarial examples approach random noise in both high-resolution low-level features and high-level features with high semantic information, the subsequent joint loss function is more aggressive against the target model to be attacked and has good transferability to other object detection models.

[0117] like Figure 4 As shown in FIG, as a specific implementation, the attention weight is obtained through the following steps:

[0118] Step S602: Get all detection boxes in the adversarial sample and set the weight of the kth pixel in the detection box to S k , the weight of the k-th pixel outside the detection frame is 0;

[0119] Step S604: Accumulate the weights of the pixels within the detection frame; when there is an intersection of the detection frames, take the largest weight as the weight of the pixel;

[0120] Step S606: Normalize the accumulated weights to obtain the attention weights.

[0121] Step S108: construct a joint loss function based on the first loss function, the second loss function, the third loss function and the fourth loss function, and update the weight parameters of the generative network and the adversarial network based on the joint loss function.

[0122] As a specific implementation, the expression of the joint loss function is:

[0123]

[0124] And because the joint loss function is related to the adversarial sample generation stage, the adversarial sample discrimination stage, and the adversarial sample recognition stage, when the generative network is iteratively trained based on the joint loss function, it can always be consistent with the actual iteration situation to the greatest extent, reducing the iteration error; thereby achieving optimization faster.

[0125] Step S110: Repeat the above steps until the optimized generation network is obtained, and input the image sample into the optimized generation network to obtain the final adversarial sample.

[0126] In this step, the total number of repetitions is set to 10,000 rounds.

[0127] The above method process can be executed in a processor or stored in a memory (or computer-readable medium). Computer-readable media includes permanent and non-permanent, removable and non-removable media that can implement information storage using any algorithm or technology. Information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable media does not include transitory computer-readable media such as modulated data signals and carrier waves.

[0128] These computer programs can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for executing on the computer or other programmable device to implement the process. Figure 1 a process or multiple processes and / or boxes Figure 1 The steps of the functions specified in one or more blocks can be implemented by different modules corresponding to different steps.

[0129] Example 2

[0130] In target detection, mean average precision (mAP) is an important indicator to measure detection performance. A target detection model usually detects multiple objects. At this time, a PR curve can be drawn for each category, and the corresponding average precision value can be calculated; and the average precision is defined by averaging the average precision values ​​of multiple categories. In order to prove the attack effectiveness of the method described in Example 1, 100 images were extracted from a set of 1000 infrared images as image samples, and adversarial samples were generated for the 100 image samples by the method described in Example 1, and then the target detection model was used to perform attack detection. In order to improve the stability of the results, the above process was repeated 100 times to record the average mAP, and the attack efficiency was measured based on the minimum time, maximum time and average time during the attack process. The corresponding results are shown in Table 1 below:

[0131] Table 1 Comparison of success rate and efficiency of different attack methods (time: seconds)

[0132]

[0133] As can be seen from the above table, the method described in Example 1 performs well in terms of both attack effectiveness and attack efficiency.

[0134] Example 3

[0135] To demonstrate the attack transferability of the method described in Example 1, we used the adversarial samples obtained in Example 2 to attack single detectors and collective detectors. The specific results are as follows:

[0136] (1) Single detector migration performance test

[0137] When the single detectors are YOLOv3, Faster-RNN, and YOLOv5, the corresponding detection results are shown in Table 2 below:

[0138] Table 2 Single detector migration performance test results (time: seconds)

[0139]

[0140] (2) Integrated detector migration performance test

[0141] When the integrated detectors are YOLOv3+YOLOv5, YOLOv3+Faster-RNN, YOLOv5+Faster-RNN, and YOLOv3+YOLOv5+Faster-RNN, the corresponding detection results are shown in Table 3 below:

[0142] Table 3 Multi-detector migration performance test results

[0143]

[0144] It can be seen from the results in Table 2 and Table 3 that the adversarial samples obtained by the method described in Example 1 have good transfer aggressiveness to various target detectors.

[0145] Example 4

[0146] This embodiment provides an image adversarial sample generation system based on a joint loss function, which is built based on the method described in Example 1. It includes:

[0147] The first acquisition module is used to obtain random noise samples and a number of image samples of the model to be attacked;

[0148] A generative adversarial module is configured to input the image sample into a generative network to generate an adversarial sample, and construct a first loss function based on the difference between the adversarial sample and the image sample; and input the image sample and the adversarial sample into a discriminative network respectively to construct a second loss function based on the output prediction result and the true label;

[0149] A first construction module is configured to input the adversarial sample into the model to be attacked to construct a third loss function based on the output detection box; input the image sample into the model to be attacked to obtain the corresponding attention weight, and obtain the feature tensors of the adversarial sample and the random noise sample at different scales through the image pyramid to simultaneously construct a fourth loss function based on them;

[0150] A second construction module is used to construct a joint loss function through the first loss function, the second loss function, the third loss function and the fourth loss function, so as to update the weight parameters of the generator network and the discriminator network respectively;

[0151] The iterative optimization module is used to repeatedly call the above modules until the optimized generation network is obtained, and input the image sample into it to obtain the final adversarial sample.

[0152] In a specific implementation, in order to calculate the attention weight, the first building block further includes:

[0153] The second acquisition module is used to obtain all detection frames in any of the image samples, and define the confidence level of the kth detection frame containing the target to be identified as S k , and assign it to the weight of the kth pixel in the detection frame; correspondingly, the weight of the kth pixel outside the detection frame is 0;

[0154] A weight accumulation module, configured to accumulate the weights of all pixels within the detection frames; wherein, when there is an intersection of the detection frames, the maximum weight is taken as the weight of the pixel;

[0155] A normalization processing module is used to normalize the accumulated weights to obtain the attention weights.

[0156] In order to generate adversarial samples, the generation adversarial module further includes a generation module. Specifically, the generation module includes:

[0157] A first extraction module, configured to perform low-dimensional feature representation on the image sample through an encoder and perform feature extraction based on each convolutional layer;

[0158] A perturbation generation module is used to map the extracted features into the sample label space through the decoder to generate perturbations;

[0159] A first output module is configured to add the perturbation amount to the image sample to obtain the adversarial sample.

[0160] In order to perform adversarial optimization on adversarial samples, the generative adversarial module is configured to further include an adversarial module. Specifically, the adversarial module includes:

[0161] A second extraction module is configured to input the image sample and the adversarial sample into the discriminant network, wherein the discriminant network includes five convolutional layers in sequence, and a BatchNorm layer is added to the middle three convolutional layers;

[0162] A discrimination module, configured to perform layer-by-layer downsampling forward propagation based on each of the convolutional layers, and output the discrimination results of the image sample and the adversarial sample through a fully connected layer to form a prediction result; wherein the first four convolutional layers use a LeakyReLU activation function, and the last convolutional layer uses a Sigmoid activation function;

[0163] The second output module is used to construct a second loss function based on the prediction result and the corresponding true label.

[0164] While the present invention has been disclosed above with reference to preferred embodiments, this is not intended to limit the present invention. Persons skilled in the art will readily appreciate that various modifications and variations can be made without departing from the spirit and scope of the present invention. Therefore, the scope of protection of the present invention shall be determined by the claims.

Claims

1. A method for generating image adversarial samples based on a joint loss function, characterized in that: include: Obtain random noise samples and several image samples of the model to be attacked; The image sample is input into the generative network to generate an adversarial sample, and a first loss function is constructed based on the difference between the adversarial sample and the image sample; then, the image sample and the adversarial sample are respectively input into the discriminative network to construct a second loss function based on the output prediction result and the corresponding true label; Among them, the first loss function is: Among them, |p n | represents the pixel distance between the image sample and the n-th pixel in the adversarial sample; The second loss function is: Where t is the true label of the image sample or the adversarial sample, the image sample is a positive sample, recorded as 1, and the adversarial sample is a negative sample, recorded as 0; is the probability that the discriminant network output is a positive sample; Inputting the adversarial sample into the model to be attacked to construct a third loss function based on the output label information; inputting the image sample into the model to be attacked to obtain the corresponding attention weight, and obtaining the feature tensors of the adversarial sample and the random noise sample at different scales through the image pyramid, so as to construct a fourth loss function based on the attention weight, the feature tensors of the adversarial sample at different scales, and the feature tensors of the random noise sample at different scales; Wherein, the third loss function is: L dect =λ obj L obj (Z|Ο,σ)+λ loc L loc (Z|Ο,σ)+λ prob L prob (Z|Ο,σ); Among them, L obj is the confidence loss term, λ obj is the confidence loss coefficient; L loc is the detection box position loss term, λ loc is the detection frame position loss coefficient; L prob is the detection category loss, λ prob is the detection category loss coefficient; Z is the adversarial sample, σ is the parameter of the model to be attacked; O = { z |P z =0, 1≤z≤S} represents a false label preset by humans, where z represents the ID of the detection frame, z Represents the coordinate position and category probability of the detection box z, P z =0 means that the confidence level of the detection box z correctly marking the target to be identified is 0, and S is the total number of targets to be identified; The fourth loss function is: Among them, E I represents the expected value, M represents the total number of feature layers of different scales in the image pyramid, m = 1, 2, ..., M represents the feature layers of different scales in turn; X m R represents the feature tensor calculated on the mth feature layer of the adversarial sample; m A represents the feature tensor calculated by the random noise sample on the mth feature layer; m is the attention weight of the image sample, where the weight of the pixels inside the detection box is greater than the weight of the pixels outside the detection box; Constructing a joint loss function by using the first loss function, the second loss function, the third loss function, and the fourth loss function to update the weight parameters of the generator network and the discriminator network respectively; Wherein, the joint loss function is: The above steps are repeated until the optimized generative network is obtained, and the image sample is input into it to obtain the final adversarial sample.

2. The method for generating image adversarial samples based on a joint loss function according to claim 1, characterized in that: The attention weight is obtained by the following steps: Get all the detection frames in any of the image samples, and define the confidence level of the kth detection frame containing the target to be identified as S k , and assign it to the weight of the kth pixel in the detection frame; correspondingly, the weight of the kth pixel outside the detection frame is 0; Accumulate the weights of all pixels within the detection frames; when there is a detection frame intersection, take the maximum weight as the weight of the pixel; The accumulated weights are normalized to obtain the attention weights.

3. The method for generating image adversarial samples based on a joint loss function according to claim 1, characterized in that: The second loss function is constructed based on the output prediction result and the true label, including: Inputting the image sample and the adversarial sample into the discriminant network; wherein the discriminant network includes five convolutional layers in sequence, and a BatchNorm layer is added to the middle three convolutional layers; Perform layer-by-layer downsampling forward propagation based on each of the convolutional layers, and output the discrimination results of the image sample and the adversarial sample through a fully connected layer to form a prediction result; wherein the first four convolutional layers use the LeakyReLU activation function, and the last convolutional layer uses the Sigmoid activation function; A second loss function is constructed based on the predicted results and the corresponding true labels.

4. The method for generating image adversarial samples based on a joint loss function according to claim 1, wherein: Inputting the image sample into a generative network to generate an adversarial sample comprises: Performing low-dimensional feature representation on the image sample through an encoder, and performing feature extraction based on each convolutional layer; The extracted features are mapped to the sample label space through the decoder to generate perturbations; The generation network includes eight convolutional layers, and the first seven convolutional layers use the LeakyReLU activation function, and the last convolutional layer uses the Tanh activation function; The perturbation amount is added to the image sample to obtain the adversarial sample.

5. A system for generating image adversarial samples based on a joint loss function, characterized in that: The method according to claim 1 includes: The first acquisition module is used to obtain random noise samples and a number of image samples of the model to be attacked; A generative adversarial module is configured to input the image sample into a generative network to generate an adversarial sample, and construct a first loss function based on the difference between the adversarial sample and the image sample; and input the image sample and the adversarial sample into a discriminative network respectively to construct a second loss function based on the output prediction result and the true label; A first construction module is configured to input the adversarial sample into the model to be attacked to construct a third loss function based on the output detection box; input the image sample into the model to be attacked to obtain the corresponding attention weight, and obtain the feature tensors of the adversarial sample and the random noise sample at different scales through the image pyramid to simultaneously construct a fourth loss function based on them; A second construction module is used to construct a joint loss function through the first loss function, the second loss function, the third loss function and the fourth loss function, so as to update the weight parameters of the generator network and the discriminator network respectively; The iterative optimization module is used to repeatedly call the above modules until the optimized generation network is obtained, and input the image sample into it to obtain the final adversarial sample.

6. The image adversarial sample generation system based on the joint loss function according to claim 5, characterized in that The first building block includes: The second acquisition module is used to obtain all detection frames in any of the image samples, and define the confidence level of the kth detection frame containing the target to be identified as S k , and assign it to the weight of the kth pixel in the detection frame; correspondingly, the weight of the kth pixel outside the detection frame is 0; A weight accumulation module, configured to accumulate the weights of all pixels within the detection frames; wherein, when there is an intersection of the detection frames, the maximum weight is taken as the weight of the pixel; A normalization processing module is used to normalize the accumulated weights to obtain the attention weights.

7. The image adversarial sample generation system based on the joint loss function according to claim 5, characterized in that The generative adversarial module includes a generation module, and the generation module includes: A first extraction module, configured to perform low-dimensional feature representation on the image sample through an encoder and perform feature extraction based on each convolutional layer; A perturbation generation module is used to map the extracted features into the sample label space through the decoder to generate perturbations; A first output module is configured to add the perturbation amount to the image sample to obtain the adversarial sample.

8. The image adversarial sample generation system based on the joint loss function according to claim 5, characterized in that: The generation adversarial module includes an adversarial module, and the adversarial module includes: A second extraction module is configured to input the image sample and the adversarial sample into the discriminant network, wherein the discriminant network includes five convolutional layers in sequence, and a BatchNorm layer is added to the middle three convolutional layers; A discrimination module, configured to perform layer-by-layer downsampling forward propagation based on each of the convolutional layers, and output the discrimination results of the image sample and the adversarial sample through a fully connected layer to form a prediction result; wherein the first four convolutional layers use a LeakyReLU activation function, and the last convolutional layer uses a Sigmoid activation function; The second output module is used to construct a second loss function based on the prediction result and the corresponding true label.

Images