Robustness testing method and system for aerial remote sensing image target detection model

Abstract

The application provides a kind of robustness test method and system of aerial remote sensing image target detection model, method includes the following steps: S1, constructs basic data set;S2, based on basic data set, using gradient and optimization-based white-box attack method and knowledge distillation-based black-box attack method obtains adversarial sample data set;S3, with the adversarial sample data set obtained in step S2 as input, using deep learning-based conditional generative adversarial network, generate robustness test set;S4, input robustness test set executes the robustness test of aerial remote sensing image target recognition model, calculates target detection task index, assesses whether test passes and outputs.The application proposes the robustness test of aerial remote sensing image target detection model, promotes the security deployment of aerial remote sensing image target detection model in practical application, and guarantees the quality and precision of aerial remote sensing image target detection model.

Description

Technical Field

[0001] This invention relates to the field of aviation intelligence, specifically to a robustness testing method and system for a target detection model in aerial remote sensing images. Background Technology

[0002] In recent years, big data and the Internet of Things have provided data support for model training, while intelligent chips and high-performance parallel computing have provided ample computing power. These factors have driven continuous technological breakthroughs. Image target detection technology, as one of the research hotspots and mainstream development directions in the field of artificial intelligence, has received widespread attention from industry and the military. With the large-scale application of deep learning in target detection, the accuracy and speed of target detection technology have been rapidly improved, and it has been widely applied in areas such as pedestrian detection, face detection, text detection, traffic sign and signal light detection, and remote sensing image detection. Aerial remote sensing image detection, in particular, is characterized by a large detection volume and high accuracy requirements, thus placing extremely high demands on the robustness of deep learning models.

[0003] Robustness of deep learning models is a key attribute posing new challenges in artificial intelligence systems. It defines and evaluates the ability of AI models and systems to avoid failures caused by anomalies and extreme conditions, describing the system's ability to maintain its functional correctness under any circumstances. Target detection models for aerial remote sensing images face not only environmental interference but also vulnerability to adversarial attacks. Therefore, testing the robustness of these models is a significant challenge. Existing test datasets primarily focus on ordinary scenarios, lacking comprehensive coverage of various robustness-influencing conditions, leading to inaccurate robustness assessments. Thus, a more comprehensive robustness testing method for aerial image target detection models is needed. Summary of the Invention

[0004] To address the shortcomings of the existing technology, the present invention aims to provide a robustness testing method and system for aerial remote sensing image target detection models. This method addresses the robustness testing requirements of mainstream artificial intelligence models such as image target detection, proposes robustness testing methods for models, promotes the safe deployment of aerial remote sensing image target detection models in practical applications, and thus ensures the quality and accuracy of aerial remote sensing image target detection models.

[0005] Specifically, the present invention provides a robustness testing method for a target detection model in aerial remote sensing images, which includes the following steps:

[0006] S1. Constructing the basic dataset: Based on the project test set, public dataset, and field-collected dataset of the target detection model in aerial remote sensing images, the basic dataset is constructed after data preprocessing.

[0007] S2. Based on the basic dataset, generate the first adversarial sample dataset and the second adversarial sample dataset using a gradient-based and optimization-based white-box attack method and a knowledge distillation-based black-box attack method, respectively. These are then combined to obtain the adversarial sample dataset. This process includes the following sub-steps:

[0008] S21. Generate the first adversarial dataset using gradient-based and optimization-based white-box attack methods. The specific attack methods include optimization-based target attacks and non-target attacks based on multiple iterations of gradients.

[0009] S22. Generate a second adversarial dataset using a black-box attack method based on knowledge distillation. The specific steps are as follows:

[0010] S221. Construct an alternative Student model based on knowledge distillation transfer attack. The specific steps are as follows:

[0011] S2211. The target detection model of the tested aerial remote sensing image is used as the Teacher model, and the alternative model is used as the Student model.

[0012] S2212. Using the basic dataset as input, simultaneously calculate the output of the Teacher model and the Student model, optimize the Student model parameters based on the model loss, and iterate multiple times to obtain an alternative Student model.

[0013] S222. Generate a second adversarial sample dataset by applying optimization-based targeted attacks or non-targeted attacks based on multiple iterations of gradients to an alternative Student model.

[0014] S23. The adversarial sample dataset is obtained by mixing the first adversarial sample dataset and the second adversarial sample dataset;

[0015] S3. Using the adversarial sample dataset obtained in step S2 as input, a robustness test set is generated using a deep learning-based conditional generative adversarial network. This specifically includes the following sub-steps:

[0016] S31. Construct a Conditional Generative Adversarial Network (CGAN) model. The CGAN model includes a generator and a discriminator. The generator receives a random noise vector z and a conditional input y as input and outputs a synthetic sample G(x) that matches the condition. The discriminator introduces a self-attention mechanism by adding an attention layer before the output layer and after the convolutional layer. The discriminator receives the real sample x, the conditional input y, and the generated sample G(x) as input and outputs the discrimination results D(x,y) and D(G(z,y),y).

[0017] S32. Prepare the dataset for training the CGAN model. The dataset includes real samples and corresponding label information.

[0018] S33. Training the CGAN model: Use real samples and generated samples as training data, and train the generator and discriminator alternately;

[0019] S34. Generate a robust test set: After the CGAN model is trained, take the adversarial sample dataset from step S2 as input, use the generator of the model to generate a robust test set and output it.

[0020] S4. Input the robustness test set to perform a robustness test on the aerial remote sensing image target recognition model, calculate the target detection task index, evaluate whether the test passes and output the result.

[0021] Preferably, the non-target attack based on multiple iterative gradients in step S21 specifically includes:

[0022] x′0=x

[0023]

[0024] Attack(x′ i )←{x′ i ;i%m=0,i≤U}

[0025] In the formula, x represents the original input image sample selected from the basic dataset, and x' n The adversarial perturbation image generated after the nth iteration is given. The perturbation amplitude ε is a small constant. Adjusting e sets the magnitude of the noise perturbation. In the formula, sign is the sign function, used to obtain the sign of the gradient in the model; J(x' n (x', y; θ) represents the attacked model with parameter θ, for x' n The input is y, and the loss function is y as the expected output. This represents the gradient of the loss function of the aerial remote sensing image target detection model obtained through backpropagation with respect to the input; the specific execution steps are as follows:

[0026] S211. Use the remote sensing image samples in the basic test set as the initial transformation sample x'0;

[0027] S212. For iterative sample transformations, apply a Clip pixel cropping operation to crop out transformed sample pixels that exceed a threshold.

[0028] S213. During the iteration process, select the attack step number m, which means that the sample is fed into the model for actual attack after each iteration of m sample calculations. The iteration continues until the attack is successful or the maximum number of iterations U is reached.

[0029] Preferably, in step S21, the target attack based on optimization increases the space of the optimal solution by utilizing the variables in the objective function, and adversarial examples are generated using the optimal objective function:

[0030] minimize||δ|| p +c·g(x+δ)

[0031] stx+δ∈[0,1] m

[0032] Where c is the equilibrium constant, g(x+δ) is the objective function, and ||δ|| p The distance of the perturbation is p, which takes values ​​of 0, 2, and ∞. The perturbation distance is calculated using the L2 norm, transforming the above formula into:

[0033]

[0034] g(x) = max[max{p i (x′)-p j (x′),i≠j},-v]

[0035] Where, p i (x') and p j (x') represents the probabilities of the model's logits output for x' after Softmax calculation for target categories i and j, where i is the target category labeled in the original image, j is the expected misclassification by the model, and v is a constant controlling the confidence level.

[0036] Preferably, the loss function of the generator in step S33 is the negative log-likelihood loss function, calculated as follows:

[0037] L G =-log(D(G(z,y),y))

[0038] Where y is the conditional input, i.e. the category of the target, z is the noise perturbation, and G(·) and D(·) are the outputs of the generator and discriminator, respectively.

[0039] The discriminator uses the binary cross-entropy loss function, calculated as follows:

[0040] L D =-log(D(x,y))-log(1-D(G(z,y),y)).

[0041] Where x is a real image sample, y is the conditional input, i.e. the category of the target, z is the noise perturbation, and G(·) and D(·) are the outputs of the generator and discriminator, respectively.

[0042] Preferably, step S2212 specifically includes the following steps:

[0043] S22121. Using the training data as input, simultaneously calculate the outputs of the Teacher model and the Student model, and perform forward propagation calculation;

[0044] The logits layer output of the Teacher model is first divided by a temperature parameter, and then the probability distribution of the sample for each class is calculated using Softmax.

[0045] P T =Softmax(Tx))

[0046] The student model takes the input sample (x) as input and generates the corresponding predicted probability distribution:

[0047] P S =Softmax(S(x))

[0048] S22122, Calculate the total loss of the model:

[0049] The loss of the student training model is calculated as follows:

[0050] L train =-∑(y*log(P) S '))

[0051] Where y represents the actual category label, P S This represents the class probability distribution predicted by the Student model.

[0052] The distillation loss function is used to transfer knowledge from the teacher's model to the student's model for training. The calculation method for distillation loss is as follows:

[0053] L distill =-∑(P T log(P S ))

[0054] The total loss of the model is:

[0055] L total =αL train +(1-α)L distill

[0056] Among them, L train This represents the original loss function of the student model, where α is the weight used to balance the two loss terms;

[0057] S22123. Perform iterative training and minimize the total loss function L using the Adam algorithm. total Optimize the parameters of the student model and train the student model.

[0058] Preferably, the deep neural network structure used by the generator in step S31 includes a conditional encoder, feature fusion, generator network, and output layer.

[0059] The discriminator uses a deep neural network structure that includes an input layer, convolutional layers, activation functions, pooling layers, batch normalization layers, attention layers, fully connected layers, and an output layer.

[0060] Preferably, in step S4, the F1 value and mAP index are selected as the target detection task indicators, and the threshold is set according to the task objective requirements.

[0061] Preferably, in step S4, before calculating the F1 score and mAP index, the Intersection over Union (IoU) is calculated. If IoU ≥ r, the target localization is considered successful, and the F1 score and mAP index are calculated. Otherwise, the model is directly deemed to have failed to locate the target, and a test failure result is output. The formula for calculating the Intersection over Union (IoU) is as follows:

[0062]

[0063] Where A is the bounding box range predicted by the model, and B is the bounding box range of the actual labeled target in the image.

[0064] Preferably, the specific steps for calculating the F1 score of the target detection model are as follows:

[0065] a. Based on the intersection-union matrix and the intersection-union threshold, calculate the precision and recall of successfully located categories. The precision P for category i is... i The recall R of class i is the proportion of samples correctly predicted by the model among all samples that are predicted as target class i. i It is the proportion of samples whose true target category is i that are correctly predicted.

[0066] b. The F1 score for category i is the harmonic mean of its prediction precision and recall. The formula for calculating Fi is as follows:

[0067]

[0068] c. Calculate the task's average F1 score:

[0069]

[0070] Preferably, the specific calculation steps for the mAP index of the target detection model are as follows:

[0071] a. Plotting the precision-recall curve: Plotting the precision-recall curve by calculating the model's precision and recall at different confidence thresholds;

[0072] b. Interpolation calculation and acquisition of the calculated precision-recall curve: Interpolation calculation is performed on the precision-recall curve for each category. The interpolation calculation method is to start from the recall rate of 0 and gradually increase the step size of the recall rate. For each recall rate level, the maximum precision value on the precision-recall curve that is greater than or equal to that recall rate is found, and a new precision-recall curve is constructed using the maximum precision value.

[0073] c. Calculate the area under the precision-recall curve for each class, which is the average precision (AP) for each class;

[0074] d. Calculate the average AP value (mAP) for all categories. The formula for calculating mAP is as follows:

[0075]

[0076] Preferably, the present invention also provides a robustness testing system for an aerial remote sensing image target detection model, which includes a basic dataset construction unit, an adversarial sample dataset generation unit, a robust test set generation unit, and a robust testing unit.

[0077] The basic dataset construction unit constructs a basic dataset based on the project test set, public dataset, and field-collected dataset of the aerial remote sensing image target detection model.

[0078] The adversarial sample dataset generation unit generates an adversarial sample dataset based on the basic test dataset, using gradient-based and optimized white-box attack methods and knowledge distillation-based black-box attack methods.

[0079] The robustness test set generation unit takes the adversarial sample dataset as input and uses a deep learning-based conditional generative adversarial network to generate a robustness test set.

[0080] The robustness testing unit performs robustness testing on the aerial remote sensing image target recognition model, calculates target recognition task indicators, and evaluates whether the test passes.

[0081] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0082] (1) The method of the present invention can propose a robustness test for the model in order to meet the robustness test requirements of mainstream artificial intelligence models such as image target detection, promote the safe deployment of aerial remote sensing image target detection models in practical applications, thereby ensuring the quality and accuracy of aerial remote sensing image target detection models. The robustness test has high accuracy and the test method is accurate, making it suitable for multiple application scenarios.

[0083] (2) Based on the robustness test results, the present invention can further optimize and improve the target detection model of aerial remote sensing images, such as iterative training, adjusting the parameters and hyperparameters of the model, increasing the number of training iterations, changing the perturbation generation method, etc., to enhance the robustness of the model. Attached Figure Description

[0084] Figure 1 This is a flowchart of the robustness testing method for the aerial remote sensing image target detection model of the present invention;

[0085] Figure 2 This is a block diagram of the robustness testing system for the aerial remote sensing image target detection model of the present invention;

[0086] Figure 3 This is the alternative model training method based on knowledge distillation selected in the embodiments of the present invention;

[0087] Figure 4 This is the conditional generative adversarial network training method based on the attention mechanism selected in the embodiments of the present invention. Detailed Implementation

[0088] Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings.

[0089] This invention provides a robustness testing method for a target detection model in aerial remote sensing images, such as... Figure 1 As shown, it includes the following steps:

[0090] S1. Constructing the basic dataset: Based on the project test set, public dataset, and field-collected dataset of the target detection model of aerial remote sensing images, the basic dataset is constructed after data preprocessing.

[0091] In practical applications, priority is given to using project test sets (requiring no prior participation in model training) and publicly available datasets from the target detection model development phase. These datasets are typically annotated and organized by professionals, and are characterized by large data volume, dynamic updates, and high quality. For specific tasks, dedicated data acquisition devices and equipment can also be used to collect aerial remote sensing image data in actual operational scenarios or similar scenarios.

[0092] S2. Based on the basic dataset, a first adversarial sample dataset and a second adversarial sample dataset are generated using a gradient-based and optimization-based white-box attack method and a knowledge distillation-based black-box attack method, respectively. These are then mixed to obtain the adversarial sample dataset. White-box attack methods refer to attacks where the attacker can obtain certain internal model information, while black-box attack methods refer to attacks where the attacker cannot obtain other model information besides publicly available and queried data. The method of this invention, by simulating both white-box and black-box attacks separately, obtains a more comprehensive adversarial sample dataset that can effectively simulate both attack methods, ensuring the accuracy of the final robustness test. This step specifically includes the following sub-steps:

[0093] S21. Generate the first adversarial dataset using gradient-based and optimization-based white-box attack methods. Specific attack methods include optimization-based target attacks and non-target attacks based on multiple iterations of gradients.

[0094] S22. A black-box attack method based on knowledge distillation is used to generate a second adversarial dataset. The specific steps are as follows:

[0095] S221. Construct a replacement Student model based on knowledge distillation-based transfer attack.

[0096] S222. Apply optimization-based targeted attack or multi-iteration gradient-based non-targeted attack to the alternative Student model to generate a second adversarial sample dataset. The optimization-based targeted attack or multi-iteration gradient-based non-targeted attack in this step are the methods in step S21, respectively. When attacking, the optimization-based targeted attack is preferred. If the targeted attack cannot obtain the second adversarial sample dataset, the multi-iteration gradient-based non-targeted attack is continued until the second adversarial sample dataset can be obtained.

[0097] The two attack methods are as follows: The non-target attack based on multiple iterative gradients is specifically as follows:

[0098] x′0=x

[0099]

[0100] Attack(x′ i )←{x′ i ;i%m=0,i≤U}

[0101] In the formula, x represents the original input image sample selected from the basic dataset, and x' nThe adversarial perturbation image generated after the nth iteration is given. The perturbation amplitude ε is a small constant. Adjusting e sets the magnitude of the noise perturbation. In the formula, sign is the sign function, used to obtain the sign of the gradient in the model; J(x' n (x', y; θ) represents the attacked model with parameter θ, for x' n The input is y, and the loss function is y as the expected output. This represents the gradient of the loss function of the aerial remote sensing image target detection model obtained through backpropagation with respect to the input; the specific execution steps are as follows:

[0102] S211. Use the remote sensing image samples in the basic test set as the initial transformation sample x'0.

[0103] S212. For iterative sample transformations, apply a Clip pixel cropping operation to crop transformed sample pixels that exceed a threshold.

[0104] S213. During the iteration process, select the attack step number m, which means that the sample is fed into the model for actual attack after each iteration of m sample calculations. The iteration continues until the attack is successful or the maximum number of iterations U is reached.

[0105] Among them, the optimization-based target attack increases the space of optimal solutions by utilizing variables in the objective function, and generates adversarial examples using the optimal objective function:

[0106] minimize||δ|| p +c·g(x+δ)

[0107] stx+δ∈[0,1] m

[0108] Where c is the equilibrium constant, g(x+δ) is the objective function, and ||δ|| p The distance of the perturbation is p, which takes values ​​of 0, 2, and ∞. The perturbation distance is calculated using the L2 norm, transforming the above formula into:

[0109]

[0110] g(x) = max[max{p i (x′)-p j (x′),i≠j},-v]

[0111] Where, p i (x') and p j (x') represents the probabilities of the model's logits output for x' after Softmax calculation for target categories i and j, where i is the target category labeled in the original image, j is the expected misclassification by the model, and v is a constant controlling the confidence level.

[0112] S23. The adversarial sample dataset is obtained by mixing the first adversarial sample dataset and the second adversarial sample dataset.

[0113] S3. Using the adversarial sample dataset obtained in step S2 as input, a robustness test set is generated using a deep learning-based conditional generative adversarial network. This specifically includes the following sub-steps:

[0114] S31. Construct a Conditional Generative Adversarial Network (CGAN) model, which includes a generator and a discriminator. The generator receives a random noise vector z and a conditional input y as input and outputs a synthetic sample G(x) that matches the condition. The discriminator adopts a CNN and Transformer Encoder architecture and introduces a self-attention mechanism. The discriminator receives a real sample x, a conditional input y, and a generated sample G(x) as input and outputs the discrimination results D(x,y) and D(G(z,y),y). The deep neural network structure used by the generator includes a conditional encoder, feature fusion, a generator network, and an output layer. The deep neural network structure used by the discriminator includes an input layer, a convolutional layer, an activation function, a pooling layer, a batch normalization layer, an attention layer, a fully connected layer, and an output layer.

[0115] S32. Prepare the dataset for training the CGAN model. The dataset includes real samples and their corresponding label information.

[0116] S33. Training the CGAN model: Use real samples and generated samples as training data to train the generator and discriminator alternately.

[0117] S34. Generate a robust test set: After the CGAN model is trained, take the adversarial sample dataset from step S2 as input, use the generator of the model to generate a robust test set and output it.

[0118] S4. Input the robustness test set to perform a robustness test on the aerial remote sensing image target recognition model, calculate the target detection task index, evaluate whether the test passes and output the result.

[0119] In a specific embodiment, in step S4, the target detection task metrics are selected as one or more of the following metrics: precision, recall, F1 score, and mAP. Thresholds are set according to the task objectives. Then, all test cases in the robustness test set are executed, the task metrics are calculated, and the pass rate of the robustness test is evaluated.

[0120] On the other hand, the present invention also provides a robustness testing system for a target detection model in aerial remote sensing images, such as... Figure 2As shown, it includes a basic dataset construction unit 1, an adversarial sample dataset generation unit 2, a robust test set generation unit 3, and a robust test unit 4.

[0121] Basic Dataset Construction Unit 1: Based on the project test set, public dataset, and field-collected dataset of the target detection model in aerial remote sensing images, a basic dataset is constructed.

[0122] The adversarial sample dataset generation unit 2 generates an adversarial sample dataset based on the basic test dataset, using gradient-based and optimized white-box attack methods and knowledge distillation-based black-box attack methods.

[0123] The robustness test set generation unit 3 takes the adversarial sample dataset as input and uses a deep learning-based conditional generative adversarial network to generate a robustness test set.

[0124] Robustness test unit 4 performs robustness tests on the aerial remote sensing image target recognition model, calculates target recognition task indicators, and evaluates whether the test is passed. Specific Implementation Example 1

[0126] The test object selected in this invention example is an aerial remote sensing image target detection model based on YOLOv8. The application scenario is remote sensing image recognition of transportation hubs, including important facilities such as airports, ports, and bridges, as well as major vehicles such as aircraft, ships, and vehicles. The robustness testing method of the aerial remote sensing image target detection model in this embodiment is as follows: Figure 1 As shown, the specific steps include:

[0127] S1. The basic dataset is constructed from the project test set, public dataset, and field-collected dataset based on the target detection model of aerial remote sensing images after data preprocessing.

[0128] Existing databases and open datasets are commonly used. These datasets are typically annotated and organized by professionals, and are characterized by large data volumes, dynamic updates, and high quality. They are widely used in the testing and development of various AI systems. For example, in the field of remote sensing imagery, there are publicly available datasets such as the TAS dataset, SZTAKI-INRIA dataset, NWPUVHR-10 dataset, VEDAI dataset, UCAS-AOD dataset, DIR3KVhice dataset, HRSC2016 dataset, RSOD dataset, DOTA dataset, DIOR dataset, and RSVG dataset. Open datasets are often used to generate positive test cases for normal classification. By analyzing the classification categories that the tested system expects to cover, samples containing different target types are selected from the target system to generate test data, which is particularly cost-effective and efficient.

[0129] This embodiment selects a portion of the aerial remote sensing images from the DOTA dataset as the basic test set. These images were acquired from Google Earth, GF-2, and JL-1 satellites provided by the China Resources Satellite Data Application Center. The aerial images provided by CycloMedia BV DOTA consist of RGB and grayscale images. The RGB images are from Google Earth and CycloMedia, while the grayscale images are from the panchromatic bands of the GF-2 and JL-1 satellite images.

[0130] After data preprocessing, each data point in the basic test dataset contains an aerial remote sensing image, a target bounding box, and a target category. All images are stored in "png" format, while the target bounding boxes and target categories are stored in "txt" format in YOLO format.

[0131] The basic dataset contains 316 images (each image may contain multiple targets), including 63 airport targets, 21 port targets, 85 bridge targets, 79 aircraft targets, 91 ship targets, and 132 vehicle targets.

[0132] S2. Based on the basic test dataset, the image samples are perturbed using gradient-based and optimized white-box attack methods, as well as knowledge distillation-based black-box attack methods, to generate the final adversarial sample dataset.

[0133] S21. Generate the first adversarial dataset using gradient-based and optimization-based white-box attack methods. Specific attack methods include optimization-based target attacks and non-target attacks based on multiple iterations of gradients.

[0134] In this embodiment, there are two basic requirements for the added perturbation: first, the added perturbation must not be perceptible to the naked eye, meaning the human eye cannot distinguish the difference between the adversarial sample and the original image; second, the adversarial sample image after adding the perturbation must be able to "fool" the model, causing it to be misclassified into a category other than the originally labeled target category. Depending on the attack result, it can be divided into non-targeted attacks and targeted attacks. Targeted attacks aim to misclassify the sample into the category specified by the attacker, while the purpose of non-targeted attacks is simply to make the model identify the target in the wrong category.

[0135] Non-target attacks based on multiple iterative gradients are performed as follows:

[0136] x′0=x

[0137]

[0138] Attack(x′ i )←{x′ i ;i%m=0,i≤U}

[0139] Where x represents the original input image sample selected from the basic dataset, x' n The adversarial perturbation image is generated after the nth iteration. The perturbation amplitude ε is a small constant. Adjusting e sets the magnitude of the noise perturbation. In the formula, sign is the sign function, used to obtain the sign of the gradient in the model. J(x' n (x', y; θ) represents the attacked model with parameter θ, for x' n The input is y, and the loss function is y as the expected output. This represents the gradient of the loss function obtained through backpropagation over the input.

[0140] The specific steps are as follows:

[0141] The remote sensing image samples in the basic test set are used as the initial transformation sample x'0. For iterative sample transformations, since multiple iterations may cause pixel values ​​to exceed the threshold of the model input, making them easily identifiable as artificially forged samples, a Clip pixel cropping operation is required to crop the pixels of the transformation sample that exceed the threshold. For example, the commonly used three-channel RGB pixel value range (-255, 255) is chosen as the pixel cropping threshold. During the iteration process, the attack step number m is selected, which means that every m iterations of sample calculation will feed the sample into the model for the actual attack. For example, if m=3, then the 0th, 3rd, 6th, etc. adversarial samples are added for the adversarial sample attack. The iteration continues until the attack is successful or the maximum number of iterations U is reached.

[0142] Optimization-based target attack: This embodiment uses an optimization-based target attack that does not explicitly calculate perturbation values ​​step by step. Instead, it treats the perturbation values ​​as an optimization problem to solve for the optimal path from the original data category i to the target attack category j. Variables in the objective function are used to increase the space of the optimal solution, and the optimal objective function is found experimentally to generate adversarial examples.

[0143] minimize||δ|| p +c·g(x+δ)

[0144] stx+δ∈[0,1] m

[0145] Where c is the equilibrium constant, and g(x+δ) is the objective function. p The distance of the perturbation is p, which can take values ​​of 0, 2, or ∞. Using the L2 norm to calculate the perturbation distance, the above formula becomes:

[0146]

[0147] g(x) = max[max{p i (x′)-pj (x′),i≠j},-v]

[0148] In the middle, p i (x') and p j (x') represent the probabilities of the model's logits output for x' after Softmax calculation for target class i and j, respectively, where i is the target class labeled in the original image, and j is the misclassification expected to be made by the model. v is a constant controlling the confidence level, which is set to 0 in this example.

[0149] The first adversarial sample dataset was obtained through the above targeted and non-targeted attacks.

[0150] S22. If the gradient information of a model is not easily accessible externally, it naturally possesses lower resistance to gradient-based attacks. The attacked model is like a black box to the attacker, who can construct query-based attacks, estimating gradient information through numerous queries or attacking the target model based on the given output. This invention will employ a knowledge distillation-based transfer attack method to simulate and test black-box attack methods with lower attack thresholds, more widespread attacks, broader applicability, and stronger transferability.

[0151] This invention utilizes deep learning knowledge distillation to train a replacement model for the tested black-box model. This replacement model can simulate the functionality of the unknown original model, reducing the query complexity of black-box attacks applied in this invention. It can also directly perform transfer attacks on the original model based on the replacement model. For example... Figure 3 As shown, the specific execution steps are as follows:

[0152] The target detection model of the tested aerial remote sensing image is used as the Teacher model, and the alternative model prepared in this invention (such as the commonly used deep learning target detection models YOLO and SSD) is used as the Student model.

[0153] Train the alternative Student model using the base dataset as input:

[0154] The training data is used as input to simultaneously compute the outputs of the Teacher model and the Student model, and then forward propagation is performed.

[0155] The output of the logits layer of the Teacher model is first divided by a temperature parameter, and then the probability distribution of the sample for each class (i.e., soft label) is calculated through Softmax. The distillation probability distribution after temperature adjustment is relatively smooth.

[0156] P T =Softmax(Tx))

[0157] Similarly, the student model takes the input sample (x) as input and generates the corresponding predicted probability distribution.

[0158] P S =Softmax(S(x))

[0159] The model loss is calculated by combining the knowledge distillation loss with the training loss function of the student model to comprehensively consider the model performance and the effect of knowledge distillation.

[0160] L train =-∑(y*log(P) S ))

[0161] Where y represents the actual category label, P S This represents the class probability distribution predicted by the Student model.

[0162] Distillation loss function is used to measure the difference between the student model and the teacher model, and to transfer the knowledge from the teacher model to the student model. A commonly used distillation loss function is the cross-entropy loss function. The calculation method for distillation loss is as follows:

[0163] L distill =-∑(P T log(P S ))

[0164] Total loss

[0165] L total =αL train +(1-α)L distill

[0166] Here, α is the weight used to balance the two loss terms.

[0167] Perform iterative training and minimize the total loss function L using the Adam algorithm. total The parameters of the student model are optimized and trained to achieve better performance under the guidance of the teacher model's knowledge. The student model is continuously refined to more closely resemble the teacher model, gradually narrowing the gap between them.

[0168] By applying targeted or non-targeted attack methods to the alternative Student model to generate successful attack samples, and using these samples to attack the original model, a high attack success rate can be achieved, thus achieving the purpose of transfer attack and obtaining a second adversarial sample dataset.

[0169] S23. The first adversarial sample dataset and the second adversarial sample dataset are mixed to obtain the adversarial sample dataset. The adversarial sample dataset contains 632 images (multiple targets may exist in one image), including 126 airport targets, 42 port targets, 170 bridge targets, 158 aircraft targets, 182 ship targets, and 262 vehicle targets.

[0170] S3. Using an adversarial sample dataset as input, apply a deep learning-based conditional generative adversarial network, such as... Figure 4 As shown, a robust test set is generated.

[0171] In practical applications, perturbation methods with high success rates are often easier to detect by the naked eye or image forgery screening systems. Generative adversarial networks (GANs), however, can apply perturbations to images in a way that more closely resembles the distribution of real data, thereby increasing the success rate of attacks. This invention provides a conditional generative adversarial network (CGAN) based on deep learning and attention mechanisms. It simulates the aforementioned attack methods, using the adversarial sample dataset generated by the white-box and black-box attack methods in step S2 as input to generate robustness test cases. The steps are as follows:

[0172] S31. Construct a conditional generative adversarial network model, which consists of a generator and a discriminator.

[0173] The generator takes a random noise vector z and a conditional input y (e.g., a class label) as input and outputs a synthetic sample G(x) that matches the condition. The generator aims to find statistical regularities within the observed data and produce realistic samples to deceive the discriminator.

[0174] The discriminator is responsible for distinguishing between real samples and generated samples, focusing on feature recognition. Therefore, this invention adopts a CNN and Transformer Encoder architecture and introduces a self-attention mechanism. The discriminator receives a real sample x, a conditional input y, and a generated sample G(x) as input, and outputs a discrimination result D(x,y) and D(G(z,y),y).

[0175] The deep neural network structure used in the CGAN generator of this invention includes the following components:

[0176] 1) Conditional Encoder: The CGAN generator receives two inputs: one is the generator's random noise input, usually a random vector following a certain distribution, used to produce diverse samples; the other is a conditional vector, used to guide the generator to generate samples of a specific class or feature. The conditional encoder is used to transform the conditional vector into an intermediate representation for use by the subsequent generator network.

[0177] 2) Feature Fusion: The random noise input of the generator and the output of the conditional encoder usually need to be fused to combine their information. This can be achieved through concatenation, element-wise addition, etc., to produce a comprehensive feature vector.

[0178] 3) Generator Network: A generator network takes a vector of fused features as input and generates synthetic samples through a series of neural network layers. Generator networks typically use deconvolutional layers (also called transposed convolutional layers) to gradually enlarge the input feature map into the generated image. Generator networks can be constructed using convolutional layers, batch normalization layers, activation functions (ReLU), etc.

[0179] 4) Output layer: The last layer of the generator network is usually an appropriate activation function, such as sigmoid or tanh, to ensure that the generated image has a suitable range.

[0180] The deep neural network structure used in the CGAN discriminator disclosed in this invention includes the following components:

[0181] 1) Input Layer: The discriminator receives generated or real samples as input. The input samples are typically images, which can be RGB or grayscale images. The size of the input layer matches the size of the input samples.

[0182] 2) Convolutional Layers: Convolutional layers are the core component of CNNs, used to extract features from images. The discriminator contains multiple convolutional layers, each using a set of learnable 3×3 or 5×5 kernels to perform convolution operations on the input image. Convolutional layers can extract features from the input image using a sliding window approach and generate a series of feature maps.

[0183] 3) Activation Functions: After each convolutional layer, a nonlinear activation function is applied to introduce a nonlinear transformation. This invention uses ReLU. The introduction of the activation function helps to increase the nonlinear expressive power of the discriminator.

[0184] 4) Pooling Layers: Pooling layers reduce the spatial size of feature maps while preserving key feature information. Common pooling operations include max pooling and average pooling; this invention employs max pooling. Pooling layers can reduce the number of parameters, improve the computational efficiency of the model, and exhibit a degree of invariance to small changes in the input.

[0185] 5) Batch Normalization Layers: Batch normalization layers normalize the output of each convolutional layer during training. This accelerates model convergence and improves model stability and generalization ability. Batch normalization layers normalize the mean and variance of each feature channel to a standard distribution, making the input to subsequent layers more stable.

[0186] 6) Attention Layer: This invention references the Transformer Encoder architecture and leverages the powerful semantic understanding capabilities of the self-attention mechanism. An attention layer is added before the classification and output layers and after the convolutional layers. This further extracts features from the sample images to fully understand their semantics.

[0187] 7) Fully Connected Layers: These are classification layers. After the convolutional layers, the discriminator maps the feature representations extracted by the attention layer to the output nodes through fully connected layers for the final discrimination task.

[0188] 8) Output Layer: The discriminator's output layer is a node used to output the discrimination result of the sample. The value of the output node can represent the probability that the sample belongs to the real sample and the fake sample. The Softmax function is applied to map the output value to the range [0,1] to represent the probability value.

[0189] S32. Data Preparation: Prepare the dataset for training the CGAN model, including real samples and their corresponding label information. For example, if using an image dataset, each image needs to be associated with its corresponding class label.

[0190] S33. Training the CGAN Model: The training process of CGAN involves an adversarial game between the generator and the discriminator. Using real and generated samples as training data, the generator's ability to generate realistic samples is improved by alternating training of the generator and discriminator. The generator's goal is to generate realistic samples to deceive the discriminator, while the discriminator's goal is to accurately distinguish between real and generated samples.

[0191] The generator's loss function uses the negative log-likelihood loss function, calculated as follows. The generator's parameters are updated by minimizing this loss function.

[0192] L G =-log(D(G(z,y),y))

[0193] Where y is the conditional input, i.e. the target category, z is the noise perturbation, and G(·) and D(•) are the outputs of the generator and discriminator, respectively.

[0194] The discriminator uses a binary cross-entropy loss function, calculated as follows. The discriminator updates its parameters by minimizing the classification loss between the real and generated samples.

[0195] L D =-log(D(x,y))-log(1-D(G(z,y),y))

[0196] Where x is a real image sample, y is the conditional input, i.e. the category of the target, z is the noise perturbation, and G(·) and D(·) are the outputs of the generator and discriminator, respectively.

[0197] S34. Generate a robust test set: After the CGAN model is trained, take the adversarial sample dataset from step three as input and use the generator of the model to generate a robust test set.

[0198] After removing low-quality samples such as blurry, deformed, and distorted images, the robustness test set consists of 564 images (each image may contain multiple targets), including 122 airport targets, 36 port targets, 163 bridge targets, 157 aircraft targets, 164 ship targets, and 239 vehicle targets.

[0199] S4: Perform robustness testing on the aerial remote sensing image target recognition model, calculate target detection task indicators, evaluate whether the test passes, and output the results.

[0200] In this embodiment, the F1 score and mAP metric are selected as the target detection task metrics, and thresholds are set according to the task objective requirements. After executing all test cases in the robustness test set, the task objective is calculated, and the test pass rate is evaluated.

[0201] Before calculating the F1 score and mAP, the Intersection over Union (IoU) needs to be calculated. If IoU ≥ r (r = 0.5 in this embodiment), the target localization is considered successful. Then, the F1 score and mAP are calculated to compare whether the target category is correct. Otherwise, the model is considered to have failed to locate the target, the image target recognition is inaccurate, and a detection failure result is output. The formula for calculating the Intersection over Union (IoU) is as follows:

[0202]

[0203] Where A is the bounding box range predicted by the model, and B is the bounding box range of the actual labeled target in the image.

[0204] F1 score and mAP are commonly used performance metrics for object detection models in two major object detection tasks. The F1 score focuses on evaluating the object detection performance of a single class, while also considering precision and recall. The mAP score, on the other hand, evaluates the overall detection accuracy of an object detection model across multiple classes, and is measured by calculating the area under the curve.

[0205] The steps for calculating the F1 score of an object detection model are as follows:

[0206] a. Calculate the precision and recall of successfully located categories based on the cross-union matrix and the cross-union threshold.

[0207] Precision P of category i i The recall R of class i is the proportion of samples correctly predicted by the model among all samples that are predicted as target class i. i It represents the proportion of samples whose true target category is i that are correctly predicted.

[0208] b. The F1 score for category i is the harmonic mean of its prediction precision and recall. The closer the F1 score is to 1, the closer the object detection model's results for each category are to the real situation.

[0209]

[0210] c. Calculate the task average F1 score.

[0211]

[0212] The steps for calculating the mAP metric of the object detection model are as follows:

[0213] a. Plot the Precision-Recall curve:

[0214] Before calculating mAP, the precision-recall curve for each class needs to be calculated. This curve is obtained by calculating the model's precision and recall at different confidence thresholds.

[0215] b. Interpolation calculation:

[0216] To calculate mAP, interpolation is needed for the precision-recall curves for each category to improve their smoothness. The purpose of interpolation is to find the maximum precision value at different recall levels. The interpolation method involves starting with a recall of 0 and gradually increasing the recall step size. For each recall level, the maximum precision value on the precision-recall curve that is greater than or equal to that recall level is found. These maximum precision values ​​constitute the new precision-recall curve.

[0217] c. Calculate Average Precision (AP): Calculate the average precision (AP) for each class, which is the area under the precision-recall curve.

[0218] d. Calculate mAP: Calculate the average AP (average AP) for all classes, i.e., mAP, to comprehensively evaluate the model's performance. mAP is the mean AP for each class, and the calculation formula is as follows:

[0219]

[0220] Table 1 shows the robustness test results of this embodiment, including the F1 score and AP index for each category of airport, port, bridge, aircraft, ship and vehicle, as well as the overall F1 score and mAP test index. Clean is the basic test dataset for normal scenarios constructed in step S1, while Adv is the robustness test set constructed in step S3.

[0221] Table 1. Robustness test results of the target detection model in airborne remote sensing images.

[0222]

[0223]

[0224] As shown in Table 1, under normal scenarios, the tested model can identify important transportation hubs and major vehicles relatively well. However, in the robustness test set constructed in this invention, the model struggles to accurately identify targets, significantly impacting its performance. The F1 score and mAP index decrease by 0.405 and 0.427 respectively due to interference. This demonstrates that the method of this invention can achieve a high attack success rate based on generating realistic perturbation data, thus comprehensively testing the robustness of the aerial image target detection model and ensuring the accuracy of the test results.

[0225] In other embodiments, based on the results of robustness tests, the aerial remote sensing image target detection model can be further optimized and improved, such as through iterative training, adjusting the model's parameters and hyperparameters, increasing the number of training iterations, and changing the perturbation generation method, thereby enhancing the model's robustness.

[0226] The embodiments described above are merely preferred embodiments of the present invention and are not intended to limit the scope of the present invention. Various modifications and improvements made by those skilled in the art to the technical solutions of the present invention without departing from the spirit of the present invention should fall within the protection scope defined by the claims of the present invention.

Claims

1. A robustness testing method for a target detection model in aerial remote sensing images, characterized in that: It includes the following steps: S1. Constructing the basic dataset: Based on the project test set, public dataset, and field-collected dataset of the target detection model in aerial remote sensing images, the basic dataset is constructed after data preprocessing. S2. Based on the basic dataset, generate the first adversarial sample dataset and the second adversarial sample dataset using a gradient-based and optimization-based white-box attack method and a knowledge distillation-based black-box attack method, respectively. These are then combined to obtain the adversarial sample dataset. This process includes the following sub-steps: S21. Generate the first adversarial dataset using gradient-based and optimization-based white-box attack methods. The specific attack methods include optimization-based target attacks and non-target attacks based on multiple iterations of gradients. S22. Generate a second adversarial dataset using a black-box attack method based on knowledge distillation. The specific steps are as follows: S221. Construct an alternative Student model based on knowledge distillation transfer attack. The specific steps are as follows: S2211. The target detection model of the tested aerial remote sensing image is used as the Teacher model, and the alternative model is used as the Student model. S2212. Using the basic dataset as input, simultaneously calculate the outputs of the Teacher model and the Student model, optimize the Student model parameters based on the model loss, and train iteratively multiple times to obtain an alternative Student model. S222. Generate a second adversarial sample dataset by applying optimization-based targeted attacks or non-targeted attacks based on multiple iterations of gradients to an alternative Student model. S23. The adversarial sample dataset is obtained by mixing the first adversarial sample dataset and the second adversarial sample dataset; S3. Using the adversarial sample dataset obtained in step S2 as input, a robustness test set is generated using a deep learning-based conditional generative adversarial network. This specifically includes the following sub-steps: S31. Construct a Conditional Generative Adversarial Network (CGAN) model. The CGAN model includes a generator and a discriminator; the generator receives a random noise vector. and a conditional input As input, it outputs a synthetic sample that matches the conditions. The discriminator introduces a self-attention mechanism, adding an attention layer before the output layer and after the convolutional layer. The discriminator receives real samples. Conditional input and generate samples As input, output the discrimination result. and ; S32. Prepare the dataset for training the CGAN model. The dataset includes real samples and their corresponding label information. S33. Training the CGAN model: Use real samples and generated samples as training data, and train the generator and discriminator alternately; S34. Generate a robust test set: After the CGAN model is trained, take the adversarial sample dataset from step S2 as input, use the generator of the model to generate a robust test set and output it. S4. Input the robustness test set to perform a robustness test on the aerial remote sensing image target recognition model, calculate the target detection task index, evaluate whether the test passes and output the result.

2. The robustness testing method for the target detection model in aerial remote sensing images according to claim 1, characterized in that: The non-target attack based on multiple iterations of gradient in step S21 is specifically as follows: In the formula, This represents the original input image samples selected from the base dataset. This is the adversarial interference image generated after the nth iteration. The disturbance amplitude is a small constant, and the adjustment The amplitude of the noise disturbance can be set, as shown in the formula. This is the sign function, used to obtain the sign of the gradient in the model; Representative parameters are The attacked model, for The input is y, and the loss function is y as the expected output. This represents the gradient of the loss function of the aerial remote sensing image target detection model obtained through backpropagation with respect to the input; the specific execution steps are as follows: S211. Use the remote sensing image samples in the basic test set as the initial transformation samples. ; S212. For iterative sample transformations, apply... The pixel cropping operation crops out the transformed sample pixels that exceed the threshold. S213. During the iteration process, select the attack step number m, which means that the sample is fed into the model for actual attack after each iteration of m sample calculations. The iteration continues until the attack is successful or the maximum number of iterations U is reached.

3. The robustness testing method for the target detection model in aerial remote sensing images according to claim 1, characterized in that: In step S21, the optimization-based target attack increases the space of optimal solutions by utilizing variables in the objective function, and generates adversarial examples using the optimal objective function: Where c is the equilibrium constant. It is the objective function. The distance of the perturbation is p, which takes values ​​of 0, 2, and ∞. The perturbation distance is calculated using the L2 norm, transforming the above formula into: in, and For the model respectively The logits output, after being processed by Softmax, represents the probability of target class i and j, where j is the expected misclassification made by the model. This is a constant that controls the confidence level.

4. The robustness testing method for the target detection model in aerial remote sensing images according to claim 1, characterized in that: In step S33, the generator's loss function is the negative log-likelihood loss function, calculated as follows: in, It is a conditional input, namely the category of the target. It's noise disturbance. and These are the outputs of the generator and the discriminator, respectively. The discriminator uses the binary cross-entropy loss function, calculated as follows: ； Where x is a real image sample, It is a conditional input, namely the category of the target. It's noise disturbance. and These are the outputs of the generator and the discriminator, respectively.

5. The robustness testing method for the target detection model in aerial remote sensing images according to claim 1, characterized in that: Step S2212 specifically includes the following steps: S22121, Use training data as input Simultaneously calculate the outputs of the Teacher model and the Student model, and perform forward propagation calculations. The logits layer output of the Teacher model is first divided by a temperature parameter, and then the probability distribution of the sample for each class is calculated using Softmax. The logits layer output of the Student model is first divided by a temperature parameter, and then the result is calculated using Softmax to obtain the probability distribution of the sample for each class. S22122, Calculate the total loss of the model: The loss of the student training model is calculated as follows: Where y represents the actual category label, This represents the class probability distribution predicted by the Student model; The distillation loss function is used to transfer knowledge from the teacher's model to the student's model for training. The calculation method for distillation loss is as follows: The total loss of the model is: in, This represents the original loss function of the student model. These are the weights used to balance the two loss terms; S22123. Perform iterative training and minimize the total loss function using the Adam algorithm. Optimize the parameters of the student model and train the student model.

6. The robustness testing method for the target detection model in aerial remote sensing images according to claim 1, characterized in that: The deep neural network structure used by the generator in step S31 includes a conditional encoder, feature fusion, generator network, and output layer. The discriminator uses a deep neural network structure that includes an input layer, convolutional layers, activation functions, pooling layers, batch normalization layers, attention layers, fully connected layers, and an output layer.

7. The robustness testing method for the target detection model in aerial remote sensing images according to claim 1, characterized in that: In step S4, the F1 value and mAP index are selected as the target detection task indicators, and the threshold is set according to the task objective requirements.

8. The robustness testing method for the target detection model in aerial remote sensing images according to claim 7, characterized in that: In step S4, before calculating the F1 score and mAP, the Intersection over Union (IoU) is calculated. If IoU ≥ r, where r is the IoU threshold, the target localization is considered successful, and the F1 score and mAP are calculated. Otherwise, the model is deemed to have failed to locate the target, and a test failure result is output. The formula for calculating the IoU is as follows: Where A is the bounding box range predicted by the model, and B is the bounding box range of the actual labeled target in the image.

9. The robustness testing method for the target detection model in aerial remote sensing images according to claim 8, characterized in that: The specific steps for calculating the F1 score of the object detection model are as follows: a. Calculate the precision and recall of successfully located categories based on the intersection-union matrix and the intersection-union threshold. Precision of category i The recall rate for class i is the proportion of samples correctly predicted by the model among all samples that are predicted as target class i. It is the proportion of samples whose true target category is i that are correctly predicted. b. The F1 score for category i is the harmonic mean of its prediction precision and recall. The formula for calculating Fi is as follows: c. Calculate the task's average F1 score: The specific calculation steps for the mAP index of the object detection model are as follows: a. Plotting precision-recall curves: Plot precision-recall curves by calculating the model's precision and recall at different confidence thresholds; b. Interpolation calculation and acquisition of the calculated precision-recall curve: Interpolation calculation is performed on the precision-recall curve for each category. The interpolation calculation method is to start from the recall rate of 0 and gradually increase the step size of the recall rate. For each recall rate level, the maximum precision value on the precision-recall curve that is greater than or equal to that recall rate is found, and a new precision-recall curve is constructed using the maximum precision value. c. Calculate the area under the precision-recall curve for each class, which is the average precision (AP) for each class; d. Calculate the average AP value (mAP) for all categories. The formula for calculating the mAP value is as follows: 。 10. A robustness testing system for the robustness testing method of the aerial remote sensing image target detection model according to claim 1, characterized in that: It includes a basic dataset construction unit, an adversarial sample dataset generation unit, a robust test set generation unit, and a robust test unit; The basic dataset construction unit constructs a basic dataset based on the project test set, public dataset, and field-collected dataset of the aerial remote sensing image target detection model. The adversarial sample dataset generation unit generates an adversarial sample dataset based on the basic test dataset, using gradient-based and optimized white-box attack methods and knowledge distillation-based black-box attack methods. The robustness test set generation unit takes the adversarial sample dataset as input and uses a deep learning-based conditional generative adversarial network to generate a robustness test set. The robustness testing unit performs robustness testing on the aerial remote sensing image target recognition model, calculates target recognition task indicators, and evaluates whether the test passes.

Images