An optical remote sensing physical adversarial sample generation method based on multi-task loss function optimization

Abstract

The application provides an optical remote sensing physical adversarial sample generation method based on a multi-task loss function optimization, and the method comprises the following steps: step one, data and model preparation; step two, arranging a parameterized physical adversarial sample in a training set image; step three, training the parameterized physical adversarial sample; step four, arranging the optimized parameterized physical adversarial sample in a test set image, and testing the adversarial performance by using a test model; and output: a physical adversarial sample design drawing; a series of physical adversarial samples are prepared according to the proposed generation method, and experiments are conducted in a data space and a physical real world to test the interference ability of the physical adversarial samples; the experimental results fully prove the effectiveness of the proposed generation method, and the generation method has wide application value.

Description

Technical Field

[0001] This invention relates to a method for generating physical adversarial samples for optical remote sensing based on multi-task loss function optimization, belonging to the fields of image generation, remote sensing, and optical intelligent counter-reconnaissance technology. Background Technology

[0002] The task of remote sensing intelligent target detection is to return the category and bounding box coordinates of one or more specific targets in a given remote sensing image. In recent years, with the maturity of remote sensing technology and the significant increase in the use of drones, the application of remote sensing technology or drones for military reconnaissance, area surveillance, and tracking of key targets of interest has become increasingly common. A large number of optical images can be collected during surveillance by remote sensing platforms or drone platforms; however, manually examining image data is a massive and inefficient undertaking. With the advancement of computer vision and the significant increase in computing power, deep learning-based intelligent target detection algorithms have gained widespread favor due to their high detection accuracy, generalization performance, and end-to-end automated processing characteristics, and are gradually becoming popular in the field of remote sensing. Therefore, it can be expected that image analysis will become increasingly automated and intelligent in the near future. In the field of remote sensing, a large number of targeted end-to-end target detection algorithms have been proposed. These algorithms, using deep neural network models trained on massive aerial remote sensing datasets, can automatically perform key object detection, such as aircraft, vehicles, and buildings. The results of this automated analysis step are then presented to the operator as clues for further analysis. Many of these excellent algorithms have been used in real-world scenarios and achieved quite good practical performance. Therefore, exploring optical intelligent counter-reconnaissance technology in remote sensing aerial scenarios is an extremely meaningful research endeavor.

[0003] Current research has explored methods for attacking deep network models. Szegedy et al. first discovered in 2013 that current machine learning models, including neural networks, are vulnerable to adversarial examples. They proposed an algorithm in their paper to generate adversarial examples in the data space for attacking target classification neural networks (see: Szegedy et al., Interesting Properties of Neural Networks [J], arXiv Preprint arXiv: 1312.6199, 2013. (Szegedy C, Zaremba W, Sutskever I, et al. Intriguing Properties of Neural Networks [J]. arXiv Preprint) arXiv:1312.6199,2013.)); Xie et al. explored the vulnerability of deep learning models for object detection and semantic segmentation tasks (see: Xie et al., Adversarial Examples for Semantic Segmentation and Object Detection [C] / / Proceedings of the IEEE International Conference on Computer Vision, 2017:1369-1378.). This suggests that while intelligent systems perform exceptionally well on a wide range of tasks, they are highly susceptible to targeted attacks that could render them ineffective.

[0004] In remote sensing scenarios, Czaja et al. explored the attack effects on classification convolutional neural networks on the fMoW dataset, which contains 53,000 frames of remote sensing image sequences taken by satellites (see: Cai et al., Adversarial Examples in Remote Sensing [C] / / Proceedings of the 26th ACM SIGSPATIAL International Conference on Advances in Geographic Information Systems, 2018:408-411.). Xu et al. proposed using a combination of multiple attack methods to generate general adversarial perturbations to interfere with the decision results of classification convolutional neural networks (see: Xu et al., General Adversarial Examples in Remote Sensing: Methods and Benchmarks [J], IEEE Earth Sciences & Remote Sensing, 2022, 60:1-15.). Y, Ghamisi P. Universal Adversarial Examples in Remote Sensing: Methodology and Benchmark[J].IEEE Transactions on Geoscience and Remote Sensing,2022,60:1-15.)) Du et al. used a learning-based model to generate adversarial perturbations on SAR images and achieved certain results (see reference: Du et al., A Fast Adversarial Attack Algorithm to Fool SAR Target Recognition with Deep Convolutional Neural Networks[J].IEEE Geoscience and Remote Sensing Letters,2021,19:1-5.(Du C,Huo C,Zhang L,etal.Fast C&W:A Fast Adversarial Attack Algorithm to Fool SAR Target Recognition with Deep Convolutional Neural Networks[J].IEEE Geoscience and Remote Sensing Letters,2021,19:1-5.)). Although the above work has confirmed the possibility of interfering with remote sensing intelligent target detection systems, most of the experiments have stopped at the data space, and the resulting perturbations are not feasible in real remote sensing scenarios. To date, research on physically achievable counter-reconnaissance units in the field of remote sensing is still relatively lacking.

[0005] This invention investigates the vulnerability of intelligent systems and proposes a method for generating physical adversarial examples for intelligent target detection systems targeting key ground objects in the field of optical remote sensing. First, to address the stealth requirement of key ground objects, a concealment blinding module is designed to blind the intelligent target detection system. Second, to address the misleading requirement of key ground object categories, a reverse visual interpretation orientation misleading module is designed to induce false alarms in the intelligent target detection system. For targets with frequent movement requirements, a reverse visual interpretation blinding camouflage coating is designed to conceal the corresponding targets. This invention uses three representative intelligent target detection algorithms—Fast-RCNN, YOLOv5, and Swin Transformer—to test the performance of the designed concealment blinding module, the reverse visual interpretation orientation misleading module, and the reverse visual interpretation blinding camouflage coating on a large number of remote sensing images. Experiments show that the application of the designed physical adversarial examples reduces the average accuracy of intelligent target detection by nearly 35%. Finally, blinding and misleading experiments were conducted on a simulated sand table targeting key ground object categories such as aircraft and oil tanks to verify the effectiveness of the designed physical adversarial examples. Summary of the Invention

[0006] 1. Objective: The objective of this invention is to provide a method for generating physical adversarial examples in optical remote sensing based on multi-task loss function optimization. This method addresses the need for stealth of key ground objects by designing a concealment blinding module for blinding the intelligent target detection system; secondly, addressing the need for misleading the category of key ground objects by designing a reverse visual interpretation orientation misleading module to induce false alarms in the intelligent target detection system; and thirdly, for targets with frequent movement needs, designing a reverse visual interpretation blinding camouflage coating for the corresponding target stealth.

[0007] 2. Technical Method: This invention is achieved through the following technical methods:

[0008] This invention is a method for generating physical adversarial examples in optical remote sensing based on multi-task loss function optimization, mainly implemented through deep learning, image processing, and optimization. The specific steps of this method are as follows:

[0009] Step 1: Data and Model Preparation.

[0010] This invention first parameterizes physical adversarial examples, with initial values ​​being randomly generated random numbers or predefined image pixel values. The initial values ​​and original size can be determined according to design needs; the default is randomly generated random numbers. Second, it imports pre-prepared camouflage-style images as style reference images for style constraints. Third, it imports a Vgg-19 convolutional neural network model pre-trained on the ImageNet dataset to extract style feature maps.

[0011] This invention further constructs a series of intelligent target detection systems. After sufficient training on a large number of remote sensing images, these systems are used for training and testing against physical adversarial examples. Specifically, one intelligent target detection system each is constructed using Fast R-CNN, YOLOv5, and Swin Transformer as detection models (hereinafter referred to as the Fast R-CNN model, YOLOv5 model, and Swin Transformer model). The three models have randomized initial weights and are independently trained iteratively on the remote sensing images used as the training set. The models are then validated on a test set, and their detection accuracy (mAP@.5) for each category of targets in the remote sensing images of the test set is statistically analyzed. One of these models is randomly selected as the intelligent target detection system for training against physical adversarial examples (hereinafter referred to as the auxiliary model), and the other two models are used as the intelligent target detection systems for testing against physical adversarial examples (hereinafter referred to as the test models). The selected remote sensing images can be obtained from various publicly available datasets in the field of computer vision, such as DOTA and iSAID.

[0012] Step 2: Arrange the parameterized physical adversarial examples in the training set images.

[0013] Based on user needs, this invention provides three types of parameterized physical adversarial samples: a concealed blinding module, a directional misleading module that reverses visual interpretation, and a blinding camouflage coating that reverses visual interpretation.

[0014] Three different arrangements were used to place the concealment blinding module, the orientation misleading module of anti-visual interpretation, and the blinding camouflage paint into the training set to obtain perturbed images for training.

[0015] Step 3: Train parameterized physical adversarial examples.

[0016] The perturbed image I adv The input is fed into the auxiliary model, and then the loss function is calculated based on the output of the auxiliary model. The parameterized physical adversarial examples are then optimized using gradient descent. The specific process is as follows:

[0017] For each input image I, which contains N targets, each target t n Assign a category label l n ∈C and rotated label box p n Where C represents the total number of key target categories contained in the remote sensing image data; the task of the intelligent target detection system F is to successfully locate the specific location coordinates r of these N targets. n And accurately classify. Define F in the output of the intelligent target detection system. c (I,t) n )∈R C F o (I,t) n)∈R、 The intelligent target detection system detects target t in image I. n The calculated classification score vector contains confidence scores and predicted location coordinates; the final predicted category of the intelligent detection system is the label corresponding to the element with the highest score in the classification score vector, i.e., max. id (F c (I,t) n The loss function Loss is defined as follows in this invention:

[0018] Loss=α1L cls +α2L obj +α3L TV +α4L nps +α5L style (1)

[0019] Among them, L cls Classification confidence loss is used to determine whether the output category of the intelligent target detection system matches the category label. n Consistent, L obj The invention employs a confidence loss to determine whether a target has been detected; L is designed to address this issue. TV The invention also designs a smoothing loss function to constrain the smoothness of parameterized physical adversarial examples and reduce high-frequency noise; L is used in this invention. nps The non-printable score loss function is used to score the degree to which the parameterized physical adversarial sample ε can be accurately printed and manufactured, ensuring that the generated parameterized physical adversarial sample can ultimately be manufactured as accurately as possible to play a role. Furthermore, this invention designs L in the loss function Loss. style , where L is the style loss function, enabling the physical adversarial examples to possess a certain degree of concealment and anti-visual interpretation capability; the style of the parameterized physical adversarial examples is compared with the style of the input camouflage image to ensure style consistency; α1, α2, α3, α4, and α5 are predefined hyperparameters. When training the parameterized physical adversarial examples, α1 and α2 take negative values; α3, α4, and α5 take positive values; where L cls and L obj The calculation formula is as follows:

[0020] L cls =CrossEntropy(F c (I,t) n ), l n (2)

[0021] L obj =CrossEntropy(F o (I,t) n ), 1(r n(3)

[0022]

[0023] IoU is the intersection-union ratio function, 1(r n The indicator function is defined as follows: if the cross-union ratio (CURRR) calculated using the cross-union function between the predicted location coordinates of the intelligent target detection system and the actual location coordinates in the label is greater than a certain threshold γ, then the intelligent target detection system is considered to have correctly located the target, and the indicator function is set to 1; otherwise, it is set to 0. CrossEntorpy is the cross-entropy loss. For parameterized physical adversarial examples ε, L... TV The calculation formula is as follows:

[0024]

[0025] i and j are the x and y coordinate indices of v, respectively; ε(i, j) is the pixel value at x-coordinate i and y-coordinate j in ε; m and n are the width and height of ε, respectively; constraints are imposed on the smoothness of ε; L nps The calculation formula is as follows:

[0026]

[0027] In the formula, P is the set of all RGB color triplets p that can be produced. This is a pixel value of a parameterized physical adversarial sample ε.

[0028] L style The calculation is as follows: Using the Vgg-19 network Φ, features of the parameterized physical adversarial sample ε and the input camouflage-style image m are extracted. For each image x input to the Vgg-19 network Φ, let φ... j (x) is the feature image of the j-th layer of network Φ, with shape C. j ×H j ×W j Define Gram matrix C j ×C j The elements of the matrix with x-coordinate c and y-coordinate c′ are:

[0029]

[0030] Where, φ j (x) c，h，w Let c be the first-dimensional index of the feature image of the j-th layer of network Φ, h be the second-dimensional index, and w be the third-dimensional index; style loss L style The Frobenius norm is the squared difference between the Gram matrices of the output image and the target image.

[0031]

[0032] The perturbed image I adv In the input auxiliary model, the corresponding loss function Loss is calculated according to the above formulas (1) to (8), and the parameterized physical adversarial sample is iteratively updated using gradient descent. In addition, during the optimization iteration process, since the value of each pixel of the optical image is in the range of 0 to 1, it is necessary to crop the pixel value of the parameterized physical adversarial sample after each gradient descent update. The specific optimization iteration formula using gradient descent is as follows:

[0033]

[0034] Where lr is the learning rate, cilp 0-1 This is the cropping function, which sets pixel values ​​less than 0 to 0, pixel values ​​greater than 1 to 1, and other pixel values ​​to remain unchanged. After 15 iterations, a parameterized physical adversarial example ε with good performance can be obtained. * .

[0035] Step 4: Arrange the optimized parameterized physical adversarial examples on the test set images and use the test model to test the adversarial performance.

[0036] The optimized parameterized physical adversarial sample ε * The optimized and parameterized physical adversarial examples are then placed in the test set images using the same arrangement as in step two. Subsequently, for each image in the test set after optimization, the test model is used to perform target detection, and the detection accuracy (mAP@.5) for each category of targets in the test set is calculated. Parameterized physical adversarial examples that reduce the test model's detection accuracy (mAP@.5) by 50% or more in a certain category are retained, and the corresponding physical adversarial example design diagram is obtained by using the OpenCV library.

[0037] 3. Advantages and effects:

[0038] This invention presents a method for generating physical adversarial examples for optical remote sensing based on multi-task loss function optimization. Its advantages include: targeting widely used intelligent target detection systems, it designs three types of optical physical adversarial examples based on multi-task loss function optimization. Compared to traditional optical concealment techniques such as ordinary camouflage, the optical physical adversarial examples designed in this invention can not only blind and mislead intelligent target detection systems, but also possess good concealment, resistance to visual interpretation, portability, and support for various application scenarios. The physical adversarial examples generated based on the proposed method have undergone performance testing in both data space and the physical real world, demonstrating high interference capability against intelligent target detection systems. Attached Figure Description

[0039] Figure 1 This is a flowchart of the overall process for generating physical adversarial examples in optical remote sensing based on multi-task loss function optimization.

[0040] Figure 2 This is a schematic diagram of the arrangement of concealed blinding modules.

[0041] Figure 3 A schematic diagram of the orientation misleading module layout for anti-visual interpretation.

[0042] Figure 4 A schematic diagram of a blinding camouflage pattern layout for reverse visual interpretation.

[0043] Figure 5a , Figure 5b The color difference between the result displayed on the digital display and the finished product from the printing equipment.

[0044] Figure 6a , Figure 6b Comparison of remote sensing image detection results with and without concealed blinding modules.

[0045] Figure 7a , Figure 7b Comparison of remote sensing image detection results with and without the deployment of a directional misleading module for anti-visual interpretation.

[0046] Figure 8a , Figure 8b Design drawings for both standard camouflage and blinding camouflage patterns with reverse visual interpretation.

[0047] Figure 9a , Figure 9b , Figure 9c Comparison of remote sensing image detection results with and without camouflage paint schemes designed for visual interpretation.

[0048] Figure 10a , Figure 10b The results of aircraft category detection were obtained by taking pictures of simulated sand table images with and without physical confrontation samples.

[0049] Figure 11a , Figure 11b The results of the oil tank category inspection were obtained by taking pictures of simulated sand table images, showing whether or not physical confrontation samples were arranged. Detailed Implementation

[0050] To better understand the technical method of the present invention, the embodiments of the present invention will be further described below with reference to the accompanying drawings:

[0051] This invention is implemented using Python within the PyTorch framework. First, data and model preparation is performed, parameterized physical adversarial examples are generated, and relevant parameters and models for training and testing are set. Next, training data is read in, and the parameterized physical adversarial examples are placed within it. Then, an intelligent target detection system, used as the training physical adversarial examples, is used to iteratively train the parameterized physical adversarial examples. After 15 iterations on the entire training set, the parameterized physical adversarial examples possess a certain level of interference capability. Finally, the trained parameterized physical adversarial examples are placed in the test set images, and the adversarial performance is tested using a test model. The entire process is as follows: Figure 1 As shown, specifically, the optical remote sensing physical adversarial example generation method based on multi-task loss function optimization of the present invention includes the following steps:

[0052] Step 1: Data and Model Preparation.

[0053] This invention first parameterizes physical adversarial examples, with initial values ​​being randomly generated random numbers or predefined image pixel values. The initial values ​​and original size can be determined according to design needs; the default is randomly generated random numbers. Secondly, this invention imports pre-prepared camouflage-style images as style reference images for style constraints. Thirdly, this invention imports a Vgg-19 convolutional neural network model pre-trained on the ImageNet dataset to extract style feature maps.

[0054] This invention further constructs a series of intelligent target detection systems. After sufficient training on a large number of remote sensing images, these systems are used for training and testing against physical adversarial examples. Specifically, one intelligent target detection system each is constructed using Fast R-CNN, YOLOv5, and Swin Transformer as detection models (hereinafter referred to as the Fast R-CNN model, YOLOv5 model, and Swin Transformer model). The three models have randomized initial weights and are independently trained iteratively on the remote sensing images used as the training set. The models are then validated on a test set, and their detection accuracy (mAP@.5) for each category of targets in the remote sensing images of the test set is statistically analyzed. One of these models is randomly selected as the intelligent target detection system for training against physical adversarial examples (hereinafter referred to as the auxiliary model), and the other two models are used as the intelligent target detection systems for testing against physical adversarial examples (hereinafter referred to as the test models). The selected remote sensing images can be obtained from various publicly available datasets in the field of computer vision, such as DOTA and iSAID.

[0055] Step 2: Arrange the parameterized physical adversarial examples in the training set images.

[0056] Based on user needs, this invention provides three types of parameterized physical adversarial samples: a concealed blinding module, a directional misleading module that reverses visual interpretation, and a blinding camouflage coating that reverses visual interpretation.

[0057] For the concealed blinding module, its original size is 64×64; its arrangement is such that for each target t in a training set image I... n Get the annotation area p of its rotated frame. n Subsequently, this invention selects the labeled region p. n The concealed blinding modules are placed in suitable locations within the area, specifically as follows: For critical target categories (such as aircraft) where it is difficult to completely cover the surface with the module in a real-world scenario, the concealed blinding modules are placed in the marked area p. n The four vertex rectangles, each with a width and height equal to the labeled area p. n 1 / 3; In real-world scenarios, it is permissible to completely cover key target categories on the surface with the module (such as small vehicles, oil tankers), and the concealed blinding module is placed in the marked area p. n The central rectangular area, and the width and height of each vertex rectangular area are the dimensions of the labeled area p. n 1 / 3; such as Figure 2 As shown;

[0058] For the orientation misleading module in anti-visual interpretation, its original size is 64×64; its arrangement is as follows: for a training image I, 4 to 8 regions p are randomly generated. i Region p i The width, height, and angle are each taken as random numbers from 1 to 256 independently. For region p i The orientation misleading module of the anti-visual interpretation is adjusted to size and region p through linear interpolation. i The same size and rotated to the corresponding angle, so that according to region p i The adjusted anti-visual interpretation orientation misdirection module can precisely match the region p i Overlap, then region p i Each pixel in the middle is replaced according to the region p i The pixel values ​​that overlap with the adjusted orientation misleading module in the reverse visual interpretation are then used. Applying this arrangement to all training set images completes the arrangement of the orientation misleading module for reverse visual interpretation. For example... Figure 3 As shown.

[0059] For blinding camouflage patterns interpreted in reverse visual mode ε camouflage Its original size is 64×64; its arrangement is as follows: for a training set image I, from the corresponding label file lab i Obtain each target t from n Rotating label box pn And obtain the semantic segmentation mask image. i The blinding camouflage pattern interpreted by reverse vision ε camouflage Linear interpolation to p n The same size, rotated to the corresponding angle, and then compared with the mask image. i Multiply to obtain the adjusted product with respect to the target t. n Blinding camouflage pattern ε′ of the same size and angle camouflage Considering that directly replacing pixels would modify some structural outlines and textures of the original target, which does not conform to the use of camouflage paint in the physical world, the adjusted blinding camouflage paint ε′ was modified accordingly. camouflage The present invention arranges the target t inside the rotating annotation frame in a hybrid manner. n The formula is as follows:

[0060] d n =t n ·(1-eps)+ε′ camouflage ·eps (10)

[0061] In the formula d n To determine the target area after coating, eps is the mixing coefficient, which is set to 0.5 in this invention; for each target t n After performing blinding camouflage layout for anti-visual interpretation, the perturbed image I is obtained. adv ,like Figure 4 As shown.

[0062] Step 3: Train parameterized physical adversarial examples.

[0063] The perturbed image I adv The input is fed into the auxiliary model, and then the loss function is calculated based on the output of the auxiliary model. The parameterized physical adversarial examples are then optimized using gradient descent. The specific process is as follows:

[0064] For each input image I, which contains N targets, each target t n Assign a category label l n Rotated label box p of ∈C n Where C represents the total number of key target categories contained in the remote sensing image data; the task of the intelligent target detection system F is to successfully locate the specific location coordinates r of these N targets. n And accurately classify. Define F in the output of the intelligent target detection system. c (I,t) n )∈ R C F o (I,t) n )∈R, The intelligent detection system detects image I and targets t respectively. n The calculated classification score vector contains confidence scores and predicted location coordinates; the final predicted category of the intelligent detection system is the label corresponding to the element with the highest score in the classification score vector, i.e., max. id (F c (I,t) n The loss function Loss is defined as follows in this invention:

[0065] Loss=α1L cls +α2L obj +α3L TV +α4L nps +α5L style (1) Where L cls Classification confidence loss is used to determine whether the output category of the intelligent target detection system matches the category label. n Consistent, L obj A confidence loss is introduced for the target, used to determine whether the target has been detected. Previous studies, while demonstrating that generated perturbations can lead to incorrect decisions in intelligent systems, have primarily relied on high-frequency noise, such as... Figure 5a and Figure 5b As shown; in remote sensing applications with large sensing distances, physical adversarial examples based on high-frequency noise are prone to losing details due to resolution changes caused by height variations, thus affecting their interference performance. Therefore, this invention designs L TV The smoothing loss function is used to constrain the smoothness of parameterized physical adversarial examples and reduce high-frequency noise. In the physical world, due to the color space differences between printing equipment and digital displays, printing equipment often cannot accurately reproduce the colors displayed on a digital display. Figure 5a and Figure 5b As shown; in order to ensure that the final physical adversarial sample does not produce color deviation and thus affect the interference performance, this invention designs L nps The non-printable scoring loss function is used to score the degree to which the parameterized physical adversarial sample ε can be accurately printed and manufactured, ensuring that the generated parameterized physical adversarial sample can ultimately be manufactured as accurately as possible to play a role. Furthermore, the special nature of remote sensing scenarios often requires key targets and corresponding equipment to have a certain degree of concealment, making them difficult for operators to detect. Therefore, this invention designs L in the loss function Loss. style, where is the style loss function, enabling physical adversarial examples to possess a certain degree of concealment and anti-visual interpretation capability; the style of the parameterized physical adversarial examples is compared with the style of the input camouflage image to ensure style consistency; α1, α2, α3, α4, and α5 are predefined hyperparameters. When training the parameterized physical adversarial examples, α1 and α2 are negative values ​​to enable the parameterized physical adversarial examples to continuously interfere with the auxiliary model; α3, α4, and α5 are positive values ​​to enable the parameterized physical adversarial examples to continuously possess smoothing characteristics, improve their ability to be accurately printed and made to have a camouflage style, and enhance concealment and anti-visual interpretation capability. In this invention, α1, α2, α3, α4, and α5 are taken as -10, -1, 2.5, 0.5, and 5, respectively; where L cls and L obj The calculation formula is as follows:

[0066] L cls =CrossEntropy(F c (I,t) n ), l n (2)

[0067] L obj =CrossEntropy(F o (I,t) n ), 1(r n (3)

[0068]

[0069] IoU is the intersection-union ratio function, 1(r n The cross-entropy ratio (COR) is an indicator function. If the COR between the predicted location coordinates of the intelligent target detection system and the specific location coordinates in the label, calculated using the cross-entropy ratio function, is greater than a certain threshold γ, then the intelligent target detection system is considered to have correctly located the target position, and the indicator function is set to 1; otherwise, it is set to 0. In this invention, the threshold γ is set to 0.5. CrossEntropy is the cross-entropy loss. For parameterized physical adversarial samples ε, L... TV The calculation formula is as follows:

[0070]

[0071] i and j are the x and y coordinate indices of v, respectively; ε(i, j) is the pixel value at x-coordinate i and y-coordinate j in ε; m and n are the width and height of ε, respectively; constraints are imposed on the smoothness of ε; L nps The calculation formula is as follows:

[0072]

[0073] In the formula, P is the set of all RGB color triplets p that can be produced. This is a pixel value of a parameterized physical adversarial sample ε.

[0074] L style The calculation is as follows: Using the Vgg-19 network Φ, features of the parameterized physical adversarial sample ε and the input camouflage-style image m are extracted. For each image x input to the Vgg-19 network Φ, let φ... j (x) is the feature image of the j-th layer of network Φ, with shape C. j ×H j ×W j Define Gram matrix C j ×C j The elements of the matrix with x-coordinate c and y-coordinate c′ are:

[0075]

[0076] Style loss L style The Frobenius norm is the squared difference between the Gram matrices of the output image and the target image.

[0077]

[0078] The perturbed image I adv The input is used to obtain the output F in the auxiliary model. c (I adv , t n ), F o (I adv , t n )and Then, calculate the corresponding L according to formulas (2) to (4). cls and L obj Substitute the parameterized physical adversarial sample ε into formulas (5) to (6) to calculate the corresponding L. TV and L nps Substitute the parameterized physical adversarial sample ε and the pre-prepared camouflage-style image m into formula (8) and take j as 16 to calculate L. style Finally, the obtained L cls L obj L TV L nps and L styleSubstitute the values ​​into formula (1) to calculate the corresponding loss function Loss, and use gradient descent to iteratively update the parameterized physical adversarial samples. In addition, during the optimization iteration process, since the value of each pixel in the optical image is in the range of 0 to 1, it is necessary to crop the pixel values ​​of the parameterized physical adversarial samples after each gradient descent update. The specific optimization iteration formula using gradient descent is as follows:

[0079]

[0080] Where lr is the learning rate, which is set to 0.005 in this paper; cilp 0-1 This is the cropping function, which sets pixel values ​​less than 0 to 0, pixel values ​​greater than 1 to 1, and other pixel values ​​to remain unchanged. After 15 iterations, a parameterized physical adversarial example ε with good performance can be obtained. * .

[0081] Step 4: Arrange the optimized parameterized physical adversarial examples on the test set images and use the test model to test the adversarial performance. The specific process is as follows:

[0082] The optimized parameterized physical adversarial sample ε * The optimized and parameterized physical adversarial examples are then placed in the test set images using the same arrangement as in step two. Subsequently, for each image in the test set after optimization, the test model is used to perform target detection, and the detection accuracy (mAP@.5) for each category of targets in the test set is calculated. Parameterized physical adversarial examples that reduce the test model's detection accuracy (mAP@.5) by 50% or more in a certain category are retained, and the corresponding physical adversarial example design diagram is obtained by using the OpenCV library.

[0083] Output: Physical adversarial sample design diagram.

[0084] Experimental results: From Figures 6a to 9c It can be seen that the physical adversarial examples generated by the optical remote sensing physical adversarial example generation method based on multi-task loss function optimization proposed in this invention have good interference performance in the data space. Figure 6a The image shows a remote sensing image without any concealed blinding modules, in which the aircraft is detected. Figure 6b The image contains a remote sensing image with concealed blinding modules for different aircraft types, in which the aircraft are not detected. Figure 7a For remote sensing images without a directional misleading module for anti-visual interpretation, Figure 7b The remote sensing image contains a misleading module for anti-visual interpretation of aircraft types, in which some background areas mislead the intelligent detection system into detecting the aircraft; Figure 8a and Figure 8bThese are design drawings for both standard camouflage and a blinding camouflage pattern interpreted visually. Figure 9a The image is a remote sensing image without camouflage markings. Figure 9b To deploy remote sensing images with ordinary camouflage paint, Figure 9c To deploy remote sensing images of blinding camouflage patterns for visual interpretation, Figure 9a , Figure 9b All aircraft were detected, and Figure 9c Not detected; furthermore, this invention actually fabricated physical countermeasure samples targeting aircraft, oil tanks, etc., and tested their interference performance on a simulated sand table. Figures 10a to 11b It can be seen that the created physical adversarial examples can effectively interfere with the detection of key targets by intelligent target detection systems. Figure 10a and Figure 11a All images were taken without any physical adversarial examples being deployed. Figure 10b and Figure 11b All images are taken of physical adversarial examples. Figure 10b and Figure 11b Most key targets within the image were not detected; the experimental results fully demonstrate the effectiveness of the generation method proposed in this invention and have broad application value.

Claims

1. A method for generating physical adversarial examples in optical remote sensing based on multi-task loss function optimization, characterized in that: Includes the following steps: Step 1: Data and Model Preparation; Parametric physical adversarial examples are generated using either randomly generated random numbers or predefined image pixel values ​​as initial values. The initial values ​​and original size are determined based on design requirements, with randomly generated random numbers being the default. A pre-prepared camouflage-style image is imported to serve as a style reference image for style constraints. A Vgg-19 convolutional neural network model pre-trained on the ImageNet dataset is imported to extract style feature maps. Step 2: Arrange parameterized physical adversarial examples in the training set images; Based on user needs, three types of parameterized physical adversarial examples are provided: concealment blinding modules, orientation misleading modules with reverse visual interpretation, and blinding camouflage paint schemes with reverse visual interpretation; these are respectively arranged in the training set to obtain perturbed images for training. Step 3: Train parameterized physical adversarial examples; Perturbed image The input is fed into the auxiliary model, and then the loss function is calculated based on the output of the auxiliary model. The parameterized physical adversarial sample is then optimized using the gradient descent method. Step 4: Arrange the optimized parameterized physical adversarial examples in the test set images and use the test model to test the adversarial performance; Here, the loss function is defined. as follows: ； in, Classification confidence loss is used to determine whether the output category of the intelligent target detection system matches the category label. Consistent The target has a confidence loss, which is used to determine whether the target has been detected. For smoothing loss function, For non-printable score loss function; In the loss function Chinese design , which is the style loss function, enables physical adversarial examples to have concealment and anti-visual interpretation capabilities; the style of the parameterized physical adversarial examples is compared with the style of the input camouflage image to ensure style consistency; , , , , For predefined hyperparameters, when training on parameterized physical adversarial examples, , The value can be negative. , , Take positive numbers; where and The calculation formula is as follows: ； ； ； Let be the intersection-union ratio function. As an indicator function, if the cross-union ratio (CURRR) calculated using the intersection-union function of the predicted location coordinates and the actual location coordinates in the label of the intelligent target detection system is greater than a certain threshold... If the value is 1, the intelligent target detection system is considered to have correctly located the target position, and the indicator function is set to 1; otherwise, it is set to 0. Cross-entropy loss is used for parameterized physical adversarial examples. , The calculation formula is as follows: ； , They are respectively The horizontal and vertical indexes, for The horizontal axis is The vertical axis is Pixel value at; , They are respectively The dimensions are width and height; for The smoothness creates constraints; The calculation formula is as follows: ； In the formula For all manufacturable RGB color triplets gather, For parameterized physical adversarial samples A certain pixel value.

2. The optical remote sensing physics adversarial example generation method based on multi-task loss function optimization according to claim 1, characterized in that: In step one, an intelligent target detection system is built. After sufficient training on a large number of remote sensing images, it is used for training and testing physical adversarial examples. Specifically, a Fast R-CNN model, a YOLOv5 model, and a SwinTransformer model are built. The initial weights of the three models are randomized and they are independently trained iteratively on the remote sensing images used as the training set. The models are then tested on the test set, and the detection accuracy mAP@.5 for each category of targets in the remote sensing images of the test set is statistically analyzed. One of these models is randomly selected as the intelligent target detection system for training physical adversarial examples (hereinafter referred to as the auxiliary model), and the other two models are used as the intelligent target detection system for testing physical adversarial examples (hereinafter referred to as the test models).

3. The optical remote sensing physics adversarial example generation method based on multi-task loss function optimization according to claim 1 or 2, characterized in that: In step one, the selected remote sensing images are from publicly available datasets in the field of computer vision, including DOTA and iSAID.

4. The optical remote sensing physics adversarial example generation method based on multi-task loss function optimization according to claim 1, characterized in that: In step three, for each input image It contains One goal, each goal The assignment has a category label and rotating annotation box ,in The total number of key target categories contained in remote sensing image data; intelligent target detection system The task is to successfully locate this The specific location coordinates of the target And accurately classify them.

5. A method for generating physical adversarial examples in optical remote sensing based on multi-task loss function optimization according to claim 1 or 4, characterized in that: In the definition of the output of the intelligent target detection system, , , The intelligent target detection system performs image processing. During the detection, for the target The calculated classification score vector contains confidence scores and predicted location coordinates; the final predicted category of the intelligent detection system is the label corresponding to the element with the highest score in the classification score vector, i.e. .

6. The optical remote sensing physics adversarial example generation method based on multi-task loss function optimization according to claim 1, characterized in that: The calculation is as follows: using a Vgg-19 network Extracting parameterized physical adversarial examples and input camouflage style images The characteristics of the Vgg-19 network for each input Image ,set up For the network The The shape of the layer feature image is Define Gram matrix for The matrix has x-coordinates of . The vertical axis is The elements are: ； in, For the network The The first dimension index of the layer feature image is The second dimension index is The third dimension index is Elements; style loss The Frobenius norm is the squared difference between the Gram matrices of the output image and the target image. ； Perturbed image In the input auxiliary model, the corresponding loss function is calculated according to the above formulas (1) to (8). The parameterized physical adversarial examples are iteratively updated using gradient descent.

7. The optical remote sensing physics adversarial example generation method based on multi-task loss function optimization according to claim 6, characterized in that: During the optimization iteration process, since the value of each pixel in the optical image ranges from 0 to 1, it is necessary to crop the pixel values ​​of the parameterized physical adversarial examples after each gradient descent update. The specific optimization iteration formula using gradient descent is as follows: ； in, For learning rate, This is a cropping function that sets pixel values ​​less than 0 to 0, pixel values ​​greater than 1 to 1, and other pixel values ​​to remain unchanged.

8. The method for generating physical adversarial examples in optical remote sensing based on multi-task loss function optimization according to claim 1, characterized in that: In step four, the optimized parameterized physical adversarial samples are... Arrange them in the test set images using the same arrangement as in step two; Subsequently, for the optimized parameterized physical adversarial sample set, the test model was used to perform target detection on each image, and the detection accuracy mAP@.5 for each category of targets in the test set was calculated. The parameterized physical adversarial samples that could reduce the detection accuracy mAP@.5 of the test model by 50% or more in a certain category were retained, and the corresponding physical adversarial sample design map was obtained by using the OpenCV library.

Images