Generative adversarial attack method of multi-load remote sensing target detection model

Abstract

The invention relates to a generative adversarial attack method for a multi-load remote sensing target detection model, and the method comprises the following steps: inputting a multi-load remote sensing image into a disturbance generator, and generating a corresponding adversarial sample; inputting the adversarial sample into a target detection model, and obtaining a feature layer output and detection result; multi-modal characteristics guide feature weighting and loss optimization, and a disturbance generator is updated; and outputting an adversarial disturbance generator oriented to the multi-load remote sensing image. According to the method, the output result of the multi-scale intermediate layer of the target detection network and the final detection result are obtained, the physical characteristics of each mode are dynamically combined, the multi-mode adaptive feature weighting strategy is designed, and the key features of each mode are specifically destroyed, so that high-hiding and high-success-rate counterattack of the multi-load remote sensing target detection model is realized.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence security and remote sensing information processing technology, specifically to a generative adversarial attack method for a multi-payload remote sensing target detection model. Background Technology

[0002] With the collaborative application of multi-source remote sensing payloads (such as optical, synthetic aperture radar SAR, and infrared), multi-payload remote sensing target detection models have significantly improved target recognition capabilities in complex scenarios by fusing information from different bands, playing a crucial role in fields such as national defense reconnaissance and disaster monitoring. However, the security of these models faces serious challenges—attackers can construct malicious inputs to induce incorrect detection results without altering the model structure, thereby affecting the reliability of decision-making.

[0003] Compared to traditional single-modal remote sensing data, multi-payload remote sensing data exhibits significantly different characteristics: optical images rely on texture and color information from the RGB three channels, making them sensitive to target edges and object materials; SAR images, through microwave scattering imaging, are sensitive to geometric structures and edge contours; and infrared images characterize the thermal radiation intensity of ground objects, making them sensitive to temperature differences and thermal targets. Different payloads represent targets in independent and complementary dimensions, but the model needs to extract key features for each payload's physical characteristics during detection. Multimodal fusion is not simply a superposition of cross-modal information, but rather a comprehensive decision based on independent analysis of multiple characteristics.

[0004] This characteristic exposes existing attack methods to a key flaw: traditional adversarial attacks are designed for natural images or single-modal remote sensing, and the perturbations they generate are usually globally uniform noise, failing to consider the differences in characteristics among multiple payloads. Specifically, the key features of a single modality are not specifically disrupted, resulting in a low success rate of attack detection on SAR / infrared images. At the same time, existing methods rely on fixed template perturbations and cannot dynamically adjust attack strategies based on the characteristic responses of different payloads such as optical, SAR, and infrared, which limits their effectiveness, especially in weak texture / low contrast scenes such as SAR and infrared.

[0005] However, current research on adversarial attacks against remote sensing image target detection models mainly focuses on visible light images or single-modal scenes, lacking in-depth utilization of the differences in key modal characteristics such as SAR and infrared. Existing attack methods are mostly designed based on natural images or general remote sensing data, failing to fully consider the geometric edge sensitivity of SAR images and the thermal target contrast characteristics of infrared images. This results in a low success rate of attacks on SAR and infrared modalities in multimodal remote sensing data, making it difficult to effectively reveal the model's vulnerability in these key modalities, thus limiting the comprehensiveness and reliability of security assessments for remote sensing target detection models. Summary of the Invention

[0006] In view of the above-mentioned technical problems, this invention proposes a generative adversarial attack method for multi-payload remote sensing target detection models. By acquiring the multi-scale intermediate layer output results and the final detection results of the target detection network, and dynamically combining the physical characteristics of each modality, a multi-modal adaptive feature weighting strategy is designed to specifically destroy the key features of each modality, thereby achieving a highly covert and high-success-rate adversarial attack on the multi-payload remote sensing target detection model.

[0007] The technical solution to the technical problem of this invention is: a generative adversarial attack method for a multi-payload remote sensing target detection model, comprising the following steps:

[0008] Step S1: Input the original remote sensing image containing at least two different load modes into the perturbation generator to generate adversarial perturbations, and superimpose the adversarial perturbations onto the original remote sensing image to obtain adversarial samples;

[0009] Step S2: Input the adversarial sample into the pre-trained target detection model, and obtain the output feature map of at least one intermediate feature layer of the target detection model and the final target detection result;

[0010] Step S3: Based on the output feature map and combined with the physical characteristics of different load modes, execute a modality-adaptive feature weighting strategy to generate a feature weight matrix corresponding to each mode; based on the feature weight matrix, the output feature map and the target detection result, construct a total loss function, and use the total loss function to update the parameters of the perturbation generator through backpropagation;

[0011] Step S4: Iterate through steps S1 to S3 until the perturbation generator meets the training termination condition, and output the trained perturbation generator to generate adversarial examples for the multi-load remote sensing target detection model.

[0012] According to one technical solution of the present invention, the at least two different load modes include at least two of the following: visible light mode, synthetic aperture radar mode, and infrared mode.

[0013] According to one technical solution of the present invention, in step S1, the perturbation generator is a generator in a generative adversarial network based on an encoder-decoder structure;

[0014] The generator receives a normalized multimodal image and sequentially passes it through an initial convolutional block, a downsampling coding layer, a residual block modulation layer, an upsampling decoding layer, and an output convolutional layer to generate range-limited adversarial perturbations.

[0015] According to one technical solution of the present invention, the adversarial perturbation generated by the generator needs to satisfy a preset perturbation strength constraint, specifically including:

[0016] Computational counter-disturbance Find the pixel-wise maximum absolute norm and limit it to a preset range. To ensure its visual imperceptibility, the following applies: Where x is the original clean sample, x' is the adversarial sample, and clip() is the clipping function that restricts all element values ​​in the tensor to the interval [ ]. ]Inside, The disturbance intensity.

[0017] According to a technical solution of the present invention, in step S2, the output feature map is extracted from the shallow layer of the backbone network and the feature pyramid fusion layer of the target detection model by registering a forward hook;

[0018] The target detection results include the predicted target category, confidence level, and bounding box coordinates.

[0019] According to a technical solution of the present invention, in step S3, the modal adaptive feature weighting strategy employs different weight matrix generation methods for different load modes:

[0020] For the visible light mode, the first weight matrix is ​​generated based on the spatial gradient magnitude of its output feature map;

[0021] For SAR modes, a second weighting matrix is ​​generated based on the distribution of strong scattering points in their output feature map;

[0022] For the infrared mode, a third weight matrix is ​​generated based on the thermal target region segmentation of its output feature map.

[0023] According to one technical solution of the present invention, step S3 includes:

[0024] For the visible light mode, the first weight matrix Generate in the following way:

[0025]

[0026] in, and These represent the gradients of the feature map in the horizontal and vertical directions, respectively. This indicates a normalization operation;

[0027] For SAR modes, the second weighting matrix Generate in the following way:

[0028]

[0029] in, This is the output feature map of the SAR mode. Threshold parameters for strong dispersion points, For indicator functions;

[0030] For the infrared mode, the third weight matrix Generate in the following way:

[0031] in, This is the output feature map of the infrared mode. Let F(IR) be the mean of F(IR). It is a linear rectification function.

[0032] According to a technical solution of the present invention, in step S3, the total loss function Loss due to feature differences Classification loss and bounding box regression loss The weighted summation consists of:

[0033] in, , , This is a hyperparameter.

[0034] According to one technical solution of the present invention, the feature difference loss The calculation formula is:

[0035]

[0036] in, , , These represent the number of channels, height, and width of the feature map, respectively. For the feature weights of the corresponding modes, For the corresponding modal intermediate feature layer, Represents the square of the Euclidean distance.

[0037] According to one technical solution of the present invention, the classification loss The cross-entropy loss is expressed as:

[0038]

[0039] in Labels for the real category, Predict probabilities for the model;

[0040] The bounding box regression loss The intersection-union ratio (IUU) between the predicted bounding box and the ground truth bounding box is calculated and expressed as follows:

[0041]

[0042] in, For the true bounding box, To predict the bounding box, This is for intersection, union, and ratio operations.

[0043] Compared with the prior art, the present invention has the following beneficial effects:

[0044] This invention provides a generative adversarial attack method for a multi-payload remote sensing target detection model. Through the dynamic generation mechanism of generative adversarial attacks, this invention overcomes the poor applicability of traditional adversarial attack methods in synthetic aperture radar or infrared modes due to physical property mismatch and gradient failure. It can achieve high success rate and high concealment adversarial attacks in visible light, SAR, infrared single-mode and multi-mode fusion scenarios.

[0045] The modality-adaptive feature weighting mechanism designed in this invention can accurately identify the key features that different remote sensing payload images rely on in target detection tasks, and assign differentiated weights to the corresponding channels in the intermediate layer feature maps of the YOLO model. This "targeted" weighting allows the generator to prioritize destroying the most critical features for detection during adversarial training, rather than blindly perturbing all regions, thereby significantly improving the accuracy and modality-specificity of adversarial perturbations and avoiding the waste of computational resources caused by ineffective perturbations.

[0046] This invention ensures that the generated adversarial examples are highly similar to real examples in both visual and physical characteristics through multimodal consistency constraints and concealment losses. For example, perturbations in the SAR mode do not introduce strong noise that does not conform to the laws of microwave scattering, perturbations in the optical mode do not cause color abrupt changes or texture distortion, and perturbations in the infrared mode do not disrupt the temperature distribution logic of thermal targets. This physical plausibility and visual consistency allows adversarial examples to pass conventional preprocessing detection, and their concealment is significantly better than traditional random noise or fixed template perturbation methods.

[0047] This invention offers a flexible and scalable generation method, adaptable to various scenario requirements. It supports adversarial attacks on specific payload remote sensing images and multi-payload combinations. Furthermore, the GAN's generation architecture allows for flexible expansion of modal branches, exhibiting strong technical compatibility and scenario generalization capabilities. Attached Figure Description

[0048] Figure 1 This is a flowchart of a generative adversarial attack method for a multi-payload remote sensing target detection model in one embodiment of the present invention;

[0049] Figure 2 This is a flowchart illustrating a generative adversarial attack method for a multi-payload remote sensing target detection model, as described in one embodiment of the present invention.

[0050] Figure 3This is the network structure of the perturbation generator in one embodiment of the present invention;

[0051] Figure 4 This is a schematic diagram of the composition of a generative adversarial attack device for a multi-payload remote sensing target detection model in one embodiment of the present invention. Detailed Implementation

[0052] To more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the accompanying drawings used in the embodiments will be briefly described below. Obviously, the drawings described below are merely some embodiments of the present invention, and those skilled in the art can obtain other drawings based on these drawings without any creative effort.

[0053] The present invention will now be described in detail with reference to the accompanying drawings and specific embodiments. The embodiments cannot be described in detail here, but the embodiments of the present invention are not limited to the following embodiments.

[0054] like Figures 1 to 3 As shown, this invention provides a generative adversarial attack method for a multi-payload remote sensing target detection model. This method trains a perturbation generator G, which receives multi-payload remote sensing images and outputs small perturbations matching the characteristics of each modality. These perturbations are then superimposed to form adversarial examples that can effectively deceive the target detection model f. The method includes the following steps:

[0055] Step S1: Input the original remote sensing image containing at least two different load modes into the perturbation generator to generate adversarial perturbations, and superimpose the adversarial perturbations onto the original remote sensing image to obtain adversarial samples;

[0056] A perturbation generator employing an encoder-decoder structure applies controllable, minute perturbations to the original multi-payload remote sensing images. Generate visually natural adversarial examples that can deceive object detection models. The core of the generator is to learn the "vulnerable feature regions" of remote sensing images through multi-level feature modulation (the encoder extracts multi-scale features, and the decoder reconstructs the perturbation map), and to optimize the perturbation strategy through adversarial training.

[0057] Payload modes typically include at least two of the following: visible light (Optical, denoted X_O), synthetic aperture radar (SAR, denoted X_S), and infrared (X_IR). These images require preprocessing such as spatial registration and normalization.

[0058] The preprocessed multimodal image is input into a pre-built perturbation generator, the structure of which is as follows: Figure 3As shown, the perturbation generator is a generator in a generative adversarial network based on an encoder-decoder structure. The perturbation generator receives a normalized multimodal image and sequentially passes it through an initial convolutional block, a downsampling coding layer, a residual block modulation layer, an upsampling decoding layer, and an output convolutional layer to generate range-limited adversarial perturbations. Combined with instance normalization and the ReLU activation function, it possesses the following structural characteristics:

[0059] (1) Input layer and initial convolutional block: The input is a normalized multimodal image. ,in For visible light (Optical) remote sensing images, For synthetic aperture radar (SAR) remote sensing images, This is an infrared remote sensing image. Initial features are extracted using reflection padding and a 7×7 convolutional layer, as shown in the formula:

[0060] ,

[0061] in, As the baseline number of channels, These represent the kernel size, stride, and padding, respectively. Reflection padding avoids the loss of boundary information and enhances the global perception capability of features.

[0062] (2) Downsampling encoding: The feature map is downsampled stepwise through two sets of 3×3 convolutional layers (stride=2), doubling the number of channels. → → Spatial resolution halved ( The formula is:

[0063] .

[0064] This process captures multi-level features of multimodal images, providing global contextual information for perturbation generation.

[0065] (3) Residual Fast Modulation: Six residual blocks are introduced to enhance features on the low-resolution feature map output by the encoder. Each residual block consists of two 3×3 convolutional layers, instance normalization, ReLU activation, and Dropout (probability = 0.5). Skip connections are used to alleviate gradient vanishing. The formula is as follows:

[0066] .

[0067] The design of residual blocks can capture long-range dependencies between multimodal features, improving the targeting of perturbation generation.

[0068] (4) Upsampling Decoding: The feature map is upsampled stepwise through two sets of 3×3 transposed convolutional layers (stride=2), halving the number of channels. Spatial resolution restored to The formula is:

[0069] x i+1 =

[0070] .

[0071] During upsampling, the transposed convolutional layer ( Combined with output padding ( Ensure accurate restoration of feature map dimensions.

[0072] (5) Output convolutional layer and perturbation generation: Finally, the feature map is mapped back to 3 channels (consistent with the number of channels in the input image) through a 7×7 convolutional layer and reflection padding, and then generated using the hyperbolic tangent function ( The output range is constrained to [−1, 1], as shown in the formula:

[0073] .

[0074] Disturbance strength constraints: Calculating counter-disturbances Find the pixel-wise maximum absolute norm (L∞ constraint) and limit it to a preset range. (In this embodiment) ), ensuring the disturbance is not visually visible: Where x is the original clean sample, x' is the adversarial sample, and clip() is the clipping function that restricts all element values ​​in the tensor to the interval [ ]. ]Inside, The disturbance intensity.

[0075] The encoder-decoder architecture effectively learns multi-scale features of the input image and ensures gradient flow through residual connections, facilitating the generation of complex, structured perturbations rather than simple noise. Instance normalization helps adapt to differences in distribution across different modalities. This architecture provides a robust network foundation for generating highly concealed and effective adversarial perturbations.

[0076] Step S2: Input the adversarial sample into the pre-trained target detection model, and obtain the output feature map of at least one intermediate feature layer of the target detection model and the final target detection result;

[0077] The adversarial sample generated in step S1 The input is fed into the target detection model to be attacked, such as using a YOLOv5 network. However, this method is also applicable to other deep learning-based target detection models. By registering forward hooks, the output feature map is extracted from the shallow layer and feature pyramid fusion layer of the YOLOv5 backbone network of the target detection model. The target detection results include the predicted target class, confidence score, and bounding box coordinates, which are used to quantify the actual impact of adversarial examples on the model's detection performance.

[0078] Specifically, the intermediate layers used to extract feature responses that resist perturbation primarily come from two sources. The first is the shallow convolutional layer in the backbone network (in this embodiment, the output after the first CSPDarknet block, denoted as layer1), corresponding to the downsampling resolution of the input image, containing rich details such as edges and textures (sensitive to the geometric structure of visible light / SAR). The second is the multi-scale fusion layer of the feature pyramid (such as the output of layers P3 / P4, denoted as layer2 and layer3), which fuses multi-level features from the backbone network, representing the target's local semantics and global contextual information (sensitive to the coupling features of infrared thermal targets and optical textures).

[0079] To achieve accurate extraction of intermediate layer outputs, this solution employs a PyTorch Hook mechanism to intercept the output of a specified layer and store it as an intermediate variable for subsequent analysis. This avoids modifying the original model structure and ensures the integrity and traceability of the feature data. First, a Hook function is defined to receive the input and output of the current layer and save the output feature map to a predefined global variable dictionary. Next, the Hook is registered to the target layer. After the YOLOv5 model is loaded, the intermediate layer to be monitored is located by traversing the model submodules, and the Hook function is registered. Finally, forward propagation and feature capture are performed to extract adversarial examples. Input a YOLOv5 model and perform forward propagation ( At this point, the Hook function will automatically intercept the output feature map of the target layer and store it in a dictionary.

[0080] By accurately intercepting key intermediate layer features of the target detection model, a direct and reliable data source is provided for subsequent feature difference calculation and adaptive weighting. Focusing on the feature layer that contributes the most to the detection task makes the training signal for adversarial attacks stronger and more direct, improving the efficiency of attack optimization.

[0081] Step S3: Based on the output feature map and the physical characteristics of different load modes, execute a modality-adaptive feature weighting strategy to generate feature weight matrices corresponding to each mode; based on the feature weight matrices, the output feature map, and the target detection results, construct a total loss function, and use the total loss function to update the parameters of the perturbation generator through backpropagation. The modality-adaptive feature weighting strategy employs different weight matrix generation methods for different load modes.

[0082] For the visible light modality, the core feature of a visible light image lies in its rich texture and edge information, which is obtained by calculating the feature map ( The spatial gradient magnitude is used to quantify the complexity of these regions and assign them higher weights in the adversarial loss, guiding the generator to produce targeted high-frequency perturbations. The core of this approach lies in constructing a first weight matrix based on the gradient magnitude. The first weight matrix Generate in the following way:

[0083]

[0084] in, and These represent the gradients of the feature map in the horizontal and vertical directions, respectively. This indicates a normalization operation;

[0085] For SAR modes, the physical characteristics of SAR images are stable strong scattering points. This can be achieved by extracting feature maps (( The highlighted pixels in the matrix are used to construct a second weight matrix that can highlight key scattering features. The second weight matrix Generate in the following way:

[0086]

[0087] in, This is the output feature map of the SAR mode. Threshold parameters for strong dispersion points, For indicator functions;

[0088] For infrared modalities, the core of infrared images is the thermal contrast between the target and the background. Feature maps are directly located through threshold segmentation. For the high-temperature target region, a weight matrix that highlights the thermal radiation characteristics is constructed. The third weight matrix Generate in the following way:

[0089] in, This is the output feature map of the infrared mode. for The mean, It is a linear rectification function.

[0090] Total loss function Loss due to feature differences Classification loss and bounding box regression loss The weighted summation consists of:

[0091] in, , , This is a hyperparameter.

[0092] Feature difference loss By weighted calculation of the difference between adversarial sample x′ and clean sample x in the intermediate layer feature map, we focus on the most critical characteristics for current modality detection:

[0093]

[0094] in, , , These represent the number of channels, height, and width of the feature map, respectively. For the feature weights of the corresponding modes, For the corresponding modal intermediate feature layer, Represents the square of the Euclidean distance.

[0095] Classification loss Measuring the difference between the predicted class probability and the true label of a target in adversarial examples:

[0096]

[0097] in Labels for the real category, Predict probabilities for the model.

[0098] Bounding box regression loss The intersection-over-union loss measure is used to measure the spatial deviation between the predicted bounding box and the true bounding box.

[0099]

[0100] in For the true bounding box, To predict the bounding box, This is for intersection, union, and ratio operations.

[0101] Transforming prior sensor knowledge into a computable weight matrix greatly enhances the intelligence and targeting of the attack. Meanwhile, feature difference loss ensures the accuracy and stealth of the attack (focusing on key features); classification loss and bounding box regression loss together ensure the thoroughness of the attack, rendering the model ineffective in both the classification and localization tasks simultaneously; the three complement each other.

[0102] Step S4: Iterate through steps S1 to S3 until the perturbation generator meets the training termination condition, and output the trained perturbation generator to generate adversarial examples for the multi-load remote sensing target detection model.

[0103] Monitor the performance of the perturbation generator on the validation set, and terminate training when the success rate of adversarial attacks (such as the decrease rate of target detection mAP) tends to stabilize or reaches a preset threshold.

[0104] The trained perturbation generator G is output. Subsequently, for any input multi-payload remote sensing image, the corresponding adversarial examples can be quickly generated simply by using this generator G, without needing to access or optimize the target detection model f again, thus achieving efficient black-box attack capabilities.

[0105] By introducing a modality-adaptive feature weighting strategy, the underlying physical characteristics (optical texture, SAR scattering, infrared thermal radiation) of different remote sensing payloads are deeply integrated with the high-level feature responses of the target detection model. This makes the perturbation generation process no longer a blind addition of noise, but rather targeted, significantly improving the accuracy and success rate of the attack. Simultaneously, the joint optimization of classification and regression losses ensures comprehensive disruption of the target detection task.

[0106] Effective adversarial examples were generated for multi-payload remote sensing images, and their effectiveness was systematically verified through Experiments 1 to 3. The three experiments used representative visible light remote sensing datasets: DIOR (Experiment 1), MSAR (Experiment 2), and INFARED (Experiment 3). During the experiments, two classic adversarial attack methods, FGSM and PGD, were selected for the adversarial example attack module, with iteration step sizes set accordingly. ) and disturbance intensity ( To generate strong adversarial examples; for our method (Ours), a perturbation generator is used for model training and inference, controlling the number of training rounds ( Learning rate ), disturbance intensity ( Hyperparameters such as mAP@50, mAP@50:95 (corresponding to the average precision within the IoU threshold range of 0.5 and 0.5:0.95, respectively), Recall, and Precision were selected as evaluation metrics commonly used in object detection tasks to quantify the model's performance on clean and adversarial samples from different dimensions.

[0107] Experiment 1 used the DIOR visible light remote sensing dataset. The detection performance of the YOLOv5l target detection model on clean samples and adversarial samples generated by different attack methods is shown in Table 1. Table 1 shows the adversarial attack results of the YOLOv5l model on the DIOR visible light remote sensing dataset. On clean samples without attacks, the target detection model achieved an mAP@50 of 0.888 and an mAP@50:95 of 0.675, demonstrating the model's original high performance level. However, after being subjected to classic adversarial attack methods FGSM and PGD, the model's performance significantly decreased: the FGSM attack reduced mAP@50 to 0.415, and the PGD attack further reduced it to 0.19. In contrast, the method proposed in this patent achieves a breakthrough improvement in attack effectiveness. Specifically, this method further reduces mAP@50 to 0.134, representing a relative reduction of 29.5% compared to the PGD attack. More importantly, our method achieved the best attack performance of 0.0992 on the mAP@50:95 metric, which reflects the overall detection performance of the model, significantly outperforming the comparison methods. Of particular note is our unique advantage in the recall metric, reducing the recall rate to an extremely low level of 0.0285. This result demonstrates that our method can effectively cause the model to miss most targets, making it more destructive and threatening in practical applications.

[0108]

[0109] Table 1

[0110] Experiment 2 used the Synthetic Aperture Radar (SAR) remote sensing dataset MSAR. The target detection performance of the YOLOv5l model on clean samples and adversarial samples generated by different attack methods is shown in Table 2. Table 2 illustrates the adversarial attack results of the YOLOv5l model on the MSAR SAR dataset, echoing its performance on the visible light dataset. This method also demonstrates excellent attack effectiveness in the SAR mode. On clean, unattacked samples, the target detection model exhibits robust detection capabilities, with mAP@50 reaching 0.846 and mAP@50:95 reaching 0.571. Recall and precision reached 0.808 and 0.873, respectively, showing the model's effective learning of SAR image features. In tests using traditional adversarial attack methods, the FGSM attack reduced mAP@50 to 0.524, while the PGD attack further suppressed this indicator to 0.498. This result indicates that although traditional methods can have some impact on SAR image detection models, there is still room for improvement in attack effectiveness. This patented method achieves groundbreaking attack results on SAR datasets, setting best attack records across all key metrics compared to traditional methods. Specifically, this method further reduces mAP@50 to 0.292, representing a 20.6% relative reduction compared to PGD attacks, and further reduces mAP@50:95 to 0.181, representing an 11.5% relative reduction compared to PGD attacks, significantly outperforming the comparative methods.

[0111]

[0112] Table 2

[0113] Experiment 3 used the INFRARED infrared remote sensing dataset. Table 3 shows the detection performance of the YOLOv5l target detection model on clean samples and adversarial samples generated by different attack methods. Table 3 illustrates the adversarial attack results of the YOLOv5l model on the INFRARED dataset. Compared to traditional adversarial attack techniques, this method exhibits a more destructive attack effect in the infrared modality, further validating the superior performance of this patented method in multimodal environments. On clean samples without attacks, all indicators of the detection model remain at a high level, with mAP@50 reaching 0.872 and the overall detection accuracy mAP@50:95 being 0.632, demonstrating the model's effective recognition ability of thermal radiation features. When subjected to traditional adversarial attacks, the FGSM method reduced the model's mAP@50 to 0.551, while the PGD attack further suppressed this indicator to 0.513. Although both methods caused performance degradation, their attack effects still have significant limitations. This patented method achieves a qualitative breakthrough on infrared datasets, drastically reducing mAP@50 to 0.292, representing a 43.1% improvement in attack efficiency compared to the best-performing comparison method, PGD. In terms of the overall performance metric mAP@50:95, this method also demonstrates a significant advantage, suppressing the metric to 0.18, a 34.1% improvement in attack effectiveness compared to PGD. Particularly noteworthy is that this method optimizes attack efficiency while maintaining attack strength. Regarding recall, this method suppresses the value to 0.285, a 65.1% decrease compared to clean samples, fully demonstrating its ability to effectively disrupt the model's ability to locate infrared thermal targets. This breakthrough achievement stems from the method's precise modeling of the thermal radiation characteristics of infrared images, successfully interfering with the model's learning of temperature distribution features through a targeted perturbation generation mechanism.

[0114]

[0115] Table 3

[0116] According to one aspect of the present invention, a generative adversarial attack device for a multi-payload remote sensing target detection model is provided, for implementing the method described in any of the above-described technical solutions. This device can be implemented as a software module, a hardware circuit, or a combination of software and hardware.

[0117] like Figure 4 As shown, the device mainly includes the following modules:

[0118] 1. Perturbation Generation Module: Corresponds to the perturbation generator G in Implementation Method 2. It internally encapsulates an encoder-decoder network and is responsible for receiving the original multimodal image. and outputs anti-disturbance output. This leads to the generation of adversarial examples. .

[0119] 2. Feature Extraction and Detection Module: This module integrates the target detection model and includes the hook management unit as described in Implementation Method 3. Its function is to receive adversarial examples. It performs forward propagation, intercepts the specified intermediate layer feature maps, and outputs the final detection results (category, bounding box, confidence score).

[0120] 3. Adaptive Weighting and Optimization Module: This is the core processing unit of this device. It includes:

[0121] The weight calculation submodule calculates the feature weight matrix for each modality based on the input intermediate feature map (separated by modality) and calls the algorithms described in Implementation Method 4 (gradient calculation, threshold judgment, mean segmentation). .

[0122] Loss calculation submodule: receives weight matrix The clean / adversarial feature map pairs and the detection results are used to calculate the feature difference loss according to the formula in Implementation Method 5. Classification loss and bounding box regression loss and according to preset weights , , Summing yields the joint loss .

[0123] Optimization submodule: based on the calculated The gradient is calculated using the backpropagation algorithm, and an optimizer (such as Adam) is called to update the parameters of the perturbation generation module.

[0124] 4. Control and Output Module: This module controls the iteration of the entire training process, including batch data loading, inter-module calls, training round control, and performance monitoring. Once training is complete, this module is responsible for saving and outputting the final perturbation generator model.

[0125] According to one aspect of the present invention, an electronic device is provided, comprising: one or more processors, one or more memories, and one or more computer programs; wherein the processor is connected to the memory, and the one or more computer programs are stored in the memory; when the electronic device is running, the processor executes the one or more computer programs stored in the memory to cause the electronic device to perform a generative adversarial attack method for a multi-load remote sensing target detection model as described in any of the above technical solutions.

[0126] The processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor can be a microprocessor or any conventional processor.

[0127] The memory can be an internal storage unit of the terminal device, such as a hard drive or RAM. Alternatively, it can be an external storage device, such as a plug-in hard drive, Smart Media Card (SMC), Secure Digital (SD) card, or Flash Card. Furthermore, the memory can include both internal and external storage units. The memory is used to store the computer program and other programs and data required by the terminal device. It can also be used to temporarily store data that has been output or will be output.

[0128] According to one aspect of the present invention, a computer-readable storage medium is provided for storing computer instructions, which, when executed by a processor, implement a generative adversarial attack method for a multi-payload remote sensing target detection model as described in any of the above technical solutions.

[0129] For example, computer-readable storage media can be read-only memory (ROM), random access memory (RAM), read-only optical disc (CD-ROM), magnetic tape, floppy disk, and optical data storage devices. They can be implemented using computer-executable program code, thus allowing them to be stored in a storage device for execution by a computing device, or they can be fabricated as separate integrated circuit modules, or multiple modules or steps can be fabricated as a single integrated circuit module. Therefore, this invention is not limited to any particular hardware and software combination.

[0130] In summary, this invention proposes a novel adaptive information balance enhancement target detection method. It introduces a modal adaptive feature weighting strategy that can identify and quantify features crucial for target detection in images with different loads (such as visible light texture edges, strong scattering points in SAR, and thermal contrast regions in infrared). By applying higher weights to these features, targeted attacks are achieved, significantly improving the success rate of adversarial examples in single-modal and multi-modal fusion scenarios in visible light, SAR, and infrared. Experiments show that, compared to methods such as FGSM and PGD, this invention's method can significantly reduce the mean accuracy (mAP) of the target detection model.

[0131] This invention constrains the perturbation intensity and utilizes the generative capabilities of generative adversarial networks to make the added perturbations visually imperceptible. More importantly, the weighted strategies for different modalities guide the generated perturbations to conform to the physical imaging laws of that modality (e.g., not introducing noise that violates the scattering mechanism into SAR images), enhancing the physical concealment of adversarial examples and making them more difficult to detect through preprocessing or manual verification.

[0132] The joint loss function of this invention includes feature difference loss, classification loss and bounding box regression loss. It not only causes the model to make incorrect judgments about the target category, but also effectively interferes with its bounding box positioning accuracy, resulting in missed target detection or serious positioning deviation, thus achieving a comprehensive and thorough attack on the target detection model.

[0133] The method framework of this invention does not depend on a specific target detection model structure and can be adapted to various mainstream detectors such as YOLO, SSD, and Faster R-CNN. Furthermore, its modality adaptation strategy facilitates the integration of new remote sensing payload modes, exhibiting good technical compatibility and scenario generalization capabilities.

[0134] Furthermore, it should be noted that the embodiments of the present invention may take the form of a computer program product implemented on one or more computer-usable storage media containing computer-usable program code.

[0135] Embodiments of the present invention are described with reference to flowchart illustrations and / or block diagrams of methods, terminal devices (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, embedded processor, or other programmable data processing terminal device to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing terminal device, generate instructions for implementing the flowchart illustrations. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0136] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing terminal device to operate in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The functions specified in one or more boxes. These computer program instructions may also be loaded onto a computer or other programmable data processing terminal equipment to cause a series of operational steps to be performed on the computer or other programmable terminal equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable terminal equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0137] It should also be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or terminal device that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or terminal device. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or terminal device that includes said element.

[0138] Finally, it should be noted that the above description represents a preferred embodiment of the present invention. It should be pointed out that although preferred embodiments have been described, those skilled in the art, once they understand the basic inventive concept of the present invention, can make various improvements and modifications without departing from the principles described herein. These improvements and modifications should also be considered within the scope of protection of the present invention. Therefore, the appended claims are intended to be interpreted as including both the preferred embodiments and all changes and modifications falling within the scope of the embodiments of the present invention.

Claims

1. A generative adversarial attack method for a multi-payload remote sensing target detection model, characterized in that, Includes the following steps: Step S1: Input the original remote sensing image containing at least two different load modes into the perturbation generator to generate adversarial perturbations, and superimpose the adversarial perturbations onto the original remote sensing image to obtain adversarial samples; Step S2: Input the adversarial sample into the pre-trained target detection model, and obtain the output feature map of at least one intermediate feature layer of the target detection model and the final target detection result; Step S3: Based on the output feature map and combined with the physical characteristics of different load modes, execute a modality-adaptive feature weighting strategy to generate a feature weight matrix corresponding to each mode; based on the feature weight matrix, the output feature map and the target detection result, construct a total loss function, and use the total loss function to update the parameters of the perturbation generator through backpropagation; Step S4: Iterate through steps S1 to S3 until the perturbation generator meets the training termination condition, and output the trained perturbation generator to generate adversarial examples for the multi-load remote sensing target detection model.

2. The method according to claim 1, characterized in that, The at least two different payload modes include at least two of the following: visible light mode, synthetic aperture radar mode, and infrared mode.

3. The method according to claim 1, characterized in that, In step S1, the perturbation generator is a generator in a generative adversarial network based on an encoder-decoder structure. The generator receives a normalized multimodal image and sequentially passes it through an initial convolutional block, a downsampling coding layer, a residual block modulation layer, an upsampling decoding layer, and an output convolutional layer to generate range-limited adversarial perturbations.

4. The method according to claim 3, characterized in that, The adversarial perturbations generated by the generator must satisfy preset perturbation strength constraints, specifically including: Computational counter-disturbance Find the pixel-wise maximum absolute norm and limit it to a preset range. To ensure its visual imperceptibility, the following applies: Where x is the original clean sample, x' is the adversarial sample, and clip() is the clipping function that restricts all element values ​​in the tensor to the interval [ ]. ]Inside, The disturbance intensity.

5. The method according to claim 2, characterized in that, In step S2, the output feature map is extracted from the shallow layer of the backbone network and the feature pyramid fusion layer of the target detection model by registering forward hooks. The target detection results include the predicted target category, confidence level, and bounding box coordinates.

6. The method according to claim 1, characterized in that, In step S3, the modal adaptive feature weighting strategy employs different weight matrix generation methods for different load modes: For the visible light mode, the first weight matrix is ​​generated based on the spatial gradient magnitude of its output feature map; For SAR modes, a second weighting matrix is ​​generated based on the distribution of strong scattering points in their output feature map; For the infrared mode, a third weight matrix is ​​generated based on the thermal target region segmentation of its output feature map.

7. The method according to claim 6, characterized in that, Step S3 includes: For the visible light mode, the first weight matrix Generate in the following way: in, and These represent the gradients of the feature map in the horizontal and vertical directions, respectively. This indicates a normalization operation; For SAR modes, the second weighting matrix Generate in the following way: in, This is the output feature map of the SAR mode. Threshold parameters for strong dispersion points, For indicator functions; For the infrared mode, the third weight matrix Generate in the following way: in, This is the output feature map of the infrared mode. Let F(IR) be the mean of F(IR). It is a linear rectification function.

8. The method according to claim 7, characterized in that, In step S3, the total loss function Loss due to feature differences Classification loss and bounding box regression loss The weighted summation consists of: in, , , This is a hyperparameter.

9. The method according to claim 8, characterized in that, The feature difference loss The calculation formula is: in, , , These represent the number of channels, height, and width of the feature map, respectively. For the feature weights of the corresponding modes, For the corresponding modal intermediate feature layer, Represents the square of the Euclidean distance.

10. The method according to claim 8 or 9, characterized in that, The classification loss The cross-entropy loss is expressed as: in Labels for the real category, Predict probabilities for the model; The bounding box regression loss The intersection-union ratio (IUU) between the predicted bounding box and the ground truth bounding box is calculated and expressed as follows: in, For the true bounding box, To predict the bounding box, This is for intersection, union, and ratio operations.

Images