Remote sensing image depth recognition method based on feature correction and multi-level adversarial defense

Abstract

The present application relates to a method for deep recognition of remote sensing images based on feature correction and multi-level adversarial defense. The method comprises: performing an adversarial attack on a remote sensing dataset according to the PGD algorithm and adding perturbations to produce adversarial samples; adding a feature correction module to the middle layer of each convolutional neural network in multiple convolutional neural network sub-networks to construct an integrated model; conducting adversarial training on the integrated model according to the adversarial samples and a pre-set adversarial loss function to obtain a deep integrated network model; annotating the remote sensing dataset and adversarial samples, training multiple adversarial detectors on each sub-network of the deep integrated network model based on the annotated samples, integrating the output of each adversarial detector through a logistic regression classifier to construct a passive defense module; sequentially connecting the passive defense module and the deep integrated network model to construct a remote sensing image deep recognition model, and performing image recognition on the remote sensing image to be tested according to the remote sensing image deep recognition model. This method can improve the adversarial robustness of the UAV visual recognition system.

Description

Technical Field

[0001] The present application relates to the technical field of remote sensing image recognition, and in particular to a remote sensing image depth recognition method based on feature correction and multi-level adversarial defense. Background Art

[0002] In recent years, with the popularization of deep neural networks, the performance of automatic interpretation of high-resolution remote sensing images has been greatly improved. At the same time, more and more modern unmanned aerial vehicles are equipped with visual navigation and recognition systems based on deep neural networks, which can support on-board reasoning of real-time images and quickly provide useful image interpretation and analysis for use in civilian fields.

[0003] Despite their numerous successes, deep neural network models exhibit significant vulnerability to adversarial examples, offering a new avenue for countering drones. After obtaining the original input, attackers can add carefully crafted, subtle perturbations to the raw data, maliciously degrading the performance of the onboard deep recognition model and causing significant harm. For example, when an intelligent drone is performing target recognition and tracking tasks, adversaries may attempt to evade the recognition of a high-value target, forcing the drone to track another target, or disrupt the drone's visual navigation to induce an emergency landing. This places higher demands on the environmental perception of the visual navigation system based on deep neural network models. Attackers can illegally access the image transmission channel between the drone and the controller and manipulate the real-time remote sensing imagery from the sensor during target recognition tasks, resulting in incorrect predictions. Furthermore, previous research has demonstrated that adversarial examples generated against a proxy model can mislead the target model with a high probability due to similar feature representations. This adversarial transferability significantly reduces the difficulty of launching an attack, as attackers do not need to understand the target model's internal structure and parameters. This black-box attack further enhances the security threat posed by adversarial examples to AI applications. Therefore, defenders need to design highly robust deep recognition systems combined with adversarial defense techniques to resist adversarial attacks. Currently, researchers have proposed numerous adversarial defense methods for natural images, which can be categorized into two main categories: active and passive. Active defense aims to improve the target network's robust recognition rate for adversarial examples while maintaining high recognition accuracy for original examples. However, generating sufficiently robust models is impractical, and sometimes inconsistent decisions between robust and non-robust classifiers are not necessarily caused by adversarial perturbations. Therefore, passive defense (i.e., adversarial detection) is considered an alternative, addressing the binary classification problem of whether an input is attacked. For both types of defense, most current strategies focus on a single model. However, even after achieving enhanced adversarial robustness, a single model may still become a target for stronger and unknown attacks. Therefore, ensembling multiple deep neural network models is considered to achieve even higher levels of adversarial robustness. Ensembles are more robust than individual models when the errors of the sub-models are uncorrelated and their predictions are diverse. Each sub-model in a deep model must be trained completely independently, and the diversity among its members depends on the randomness of the initialization and learning process. However, simply assembling multiple independently trained sub-models and fusing their predictions at the output layer is not very effective. It is also necessary to fully utilize the complex feature representations of remote sensing images using deep neural networks. Some researchers have proposed deep model ensemble defense strategies, such as ensemble adversarial detectors for passive defense, ensemble adversarial training for active defense, and various training loss functions that can limit the adversarial transferability between sub-models.However, these methods are relatively independent and do not combine the complementary characteristics of active and passive defense. The robustness of the adversarial defense framework currently used by drones for high-security tasks such as scene or target recognition is still very low. Summary of the Invention

[0004] Based on this, it is necessary to address the above technical problems and provide a remote sensing image depth recognition method based on feature correction and multi-level adversarial defense that can improve the adversarial robustness of the UAV visual recognition system.

[0005] A remote sensing image depth recognition method based on feature correction and multi-level adversarial defense, the method comprising:

[0006] Obtain a remote sensing dataset; perform an adversarial attack on the remote sensing dataset using the PGD algorithm and add perturbations to create adversarial samples; the adversarial samples include remote sensing images and images after the adversarial attack;

[0007] In multiple convolutional neural network sub-networks, a feature correction module is added to the middle layer of each convolutional neural network sub-network to build an integrated model. The integrated model is trained adversarially based on adversarial examples and a pre-set adversarial loss function to obtain a deep integrated network model.

[0008] The remote sensing dataset and adversarial samples are labeled. Multiple adversarial detectors are trained based on the labeled samples on each sub-network of the deep ensemble network model. The output of each adversarial detector is integrated through a logistic regression classifier to construct a passive defense module.

[0009] The passive defense module and the deep integration network model are connected in sequence to construct a remote sensing image deep recognition model, and image recognition is performed on the remote sensing image to be tested based on the remote sensing image deep recognition model.

[0010] In one embodiment, performing image recognition on a remote sensing image to be tested based on a remote sensing image depth recognition model includes:

[0011] Passing the remote sensing image to be tested through the passive defense module to obtain multiple confidence levels of the remote sensing image to be tested;

[0012] The average of multiple confidence levels of the remote sensing image to be tested is taken to obtain the overall confidence level;

[0013] A confidence threshold is set in advance. If the overall confidence is greater than the confidence threshold, the remote sensing image to be tested is determined to be an adversarial sample and rejected to proceed to the next step. If the overall confidence is less than the confidence threshold, the remote sensing image to be tested is determined to be a normal sample and passed.

[0014] The remote sensing image to be tested that passes through the passive defense module is passed to the deep integration network model for robust recognition to obtain the recognition category of the remote sensing image to be tested.

[0015] In one embodiment, the process of acquiring the remote sensing dataset includes:

[0016] Acquire multiple remote sensing images, perform cropping and normalization on the remote sensing images to obtain a normalized image set; randomly divide the normalized image set into a training set and a test set, preprocess the training set and the test set, and linearly transform the image data in the training set and the test set into a data set with a mean of 0 and a variance of 1 through the normalize operation to obtain a remote sensing data set.

[0017] In one embodiment, an adversarial attack is performed on a remote sensing dataset according to a PGD algorithm and perturbations are added to create adversarial samples, including:

[0018] According to the PGD algorithm, the remote sensing dataset is attacked and perturbations are added to create adversarial samples.

[0019]

[0020]

[0021] in, is the adversarial sample obtained by iterating the gradient of the remote sensing image x n times in the remote sensing dataset, S is the random ball projection, ε is the perturbation limit, α is the attack step size, represents the C&W attack, Z is the output of the softmax layer, t is the predicted category, i is any category except t, and -κ is the noise limit.

[0022] In one embodiment, the process of adding a feature correction module includes:

[0023] The process of adding a feature correction module includes a separation phase and a recalibration phase; the feature correction module includes a separation network and a fully connected network; the fully connected network is an auxiliary layer of the integrated model;

[0024] In the separation phase, a separation network is defined to learn the robustness of feature units and output a robustness map. A soft mask is set according to the robustness map, and the soft mask and the pre-set feature threshold are used to decompose the feature map to obtain robust features and non-robust features.

[0025] Robust features and non-robust features are added to the fully connected network. Based on the pre-set first loss function, the separation network is guided to assign higher robustness scores to feature units that help the auxiliary layer make correct decisions, helping the auxiliary layer to make correct predictions. Correct predictions include correct robust features and correct non-robust features.

[0026] Calibrate the correct non-robust features according to the calibration network to obtain calibrated non-robust features;

[0027] In the recalibration stage, the calibrated non-robust features are recalibrated according to the auxiliary layer to obtain the recalibrated non-robust features;

[0028] Output feature maps based on robust features and recalibrated non-robust features.

[0029] In one embodiment, the first loss function is

[0030] Among them, y c is the true label, is the corresponding confidence score; y c ′ is the wrong label with the highest confidence score, is the confidence score.

[0031] In one embodiment, the recalibrated calibration loss function is

[0032]

[0033] in, is the output prediction score of the auxiliary layer, y c is the true label.

[0034] In one embodiment, the adversarial loss function is

[0035]

[0036] Among them, λ sep and λ rec is a hyperparameter, f i (·) represents a single sub-model in a deep ensemble network model, and Indicates different weights, is the loss function for any variant of adversarial training.

[0037] In one embodiment, the objective function of adversarial training is

[0038]

[0039] Among them, θ represents the network parameters of the deep integration network model, x represents the remote sensing image in the remote sensing dataset, and y represents the true label of the remote sensing image. represents the underlying data distribution of remote sensing images, δ represents the generated adversarial perturbation, represents the general constraint against perturbations, represents the loss function for training a deep neural network model f with parameters θ.

[0040] In one embodiment, the adversarial detector includes local intrinsic dimension, kernel density estimation, and Mahalanobis distance; the remote sensing image to be tested is passed through a passive defense module to obtain multiple confidence levels of the remote sensing image to be tested, including:

[0041] The remote sensing image to be tested is passed through the passive defense module, and the adversarial detector and logistic regression classifier are used to calculate the remote sensing image to be tested, and the confidence of the remote sensing image to be tested is obtained as

[0042] p(adv|V(x0))=(1+exp(β0+β T ·V(x0))) -1

[0043] V(x0)={D1(x0), D2(x0), D3(x0)}

[0044] D i (x0)={d i (a1), d i (a2), ..., d i (a L )}, i=1,2,3

[0045] Among them, d1 represents the local intrinsic dimension, d2 represents the kernel density estimate, d3 represents the Mahalanobis distance, and a L Denotes the feature map extracted by the sub-model layer l in the deep ensemble network model, D i (x0) represents the score counted by the adversarial detector, x0 represents the real-time remote sensing image to be tested, β T is the weight vector, and β0 represents the weight bias.

[0046] The above-mentioned remote sensing image deep recognition method based on feature correction and multi-level adversarial defense fully utilizes the rich and complex features in remote sensing images, introduces a feature correction module, activates more non-robust features in adversarial training, and improves the robustness of the deep ensemble network model. The passive defense module is connected with the actively reinforced deep ensemble recognition model to build a multi-level adversarial defense framework, achieving the highest level of adversarial robustness enhancement and reducing the threat of adversarial vulnerability posed by the deep recognition model on intelligent drones. In the passive defense, multiple adversarial detection statistical features are integrated and the two modules are connected front and back to jointly defend against adversarial attack images that the drone may encounter when performing recognition tasks, thus achieving defense against remote sensing adversarial samples. The passive defense module and the actively reinforced deep ensemble neural network are deployed on a modern intelligent drone vision system. When the enemy obtains and maliciously attacks the remote sensing images captured by the drone camera and sensor through the channel, the adversarial robustness of the drone vision recognition system will be greatly improved, reducing the probability of system recognition errors and thus mission failure. In addition, the present application can also use a relatively lightweight neural network as a sub-network architecture to reduce inference time and is suitable for edge environments. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Figure 1 A flowchart of a remote sensing image depth recognition method based on feature correction and multi-level adversarial defense in one embodiment;

[0048] Figure 2 A schematic diagram of the position of a feature correction module in a deep integration model and an optimization process of an active enhancement model in one embodiment;

[0049] Figure 3 A schematic diagram of the structure of a passive defense module in one embodiment;

[0050] Figure 4 FIG1 is a simplified flowchart of performing image recognition on a remote sensing image to be tested based on a remote sensing image depth recognition model in another embodiment. DETAILED DESCRIPTION

[0051] In order to make the purpose, technical solutions and advantages of this application more clear, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0052] In one embodiment, Figure 1 As shown, a remote sensing image depth recognition method based on feature correction and multi-level adversarial defense is provided, comprising the following steps:

[0053] Step 102: Obtain a remote sensing dataset; perform an adversarial attack on the remote sensing dataset according to the PGD algorithm and add perturbations to create adversarial samples; the adversarial samples include remote sensing images and images after the adversarial attack.

[0054] When preparing the dataset, standardize the size of the remote sensing images entering the model and divide the data into training and test sets. The first step in using deep neural networks to perform scene or object recognition tasks in remote sensing images is to standardize the input image size and normalize all data. Due to limitations such as remote sensing image resolution, unifying image size through operations such as pooling often damages image characteristics. Therefore, all images must be normalized to a uniform size through a cropping operation. The images are then numerically normalized, compressing all pixel values ​​to between 0 and 1. The images are randomly divided into training and test sets. Image preprocessing involves linearly transforming the image data to a dataset with a mean of 0 and a variance of 1 through the normalize operation.

[0055] Through adversarial attacks, such as using the PGD algorithm to generate adversarial samples, greater robustness can be achieved in the subsequent training of the integrated model.

[0056] Step 104: Add a feature correction module to the middle layer of each convolutional neural base subnetwork in multiple convolutional neural base subnetworks to construct an integrated model; perform adversarial training on the integrated model based on the adversarial sample and the pre-set adversarial loss function to obtain a deep integrated network model.

[0057] This application introduces a new module feature correction module FCM in the middle layer of each convolutional neural network sub-network to enable the deep neural network after adversarial training to reactivate some information in the non-robust features, thereby improving the adversarial robustness of the model. Then, the generated adversarial samples are added to the training set to minimize the training loss and minimize the parameters in the neural network to learn to identify adversarial samples with real labels. This will obtain a deep integrated network model with strong adversarial robustness and high generalization ability after active reinforcement. Here, we use three (N=3) ResNet-18 network structures with random initialization parameters and a completely independent adversarial training process as an example. ResNet-18 is a relatively lightweight neural network architecture commonly used in resource-constrained edge environments, such as modern drones.

[0058] Step 106: annotate the remote sensing dataset and adversarial samples, train multiple adversarial detectors based on the annotated samples on each sub-network of the deep ensemble network model, and integrate the output of each adversarial detector through a logistic regression classifier to construct a passive defense module.

[0059] Deeply integrated networks It will be further used for training the passive defense module. When the output of the baseline model is inconsistent with the output of the robust model, passive defense will show its value. We want to know whether this is because the input has been attacked by an adversarial attack. In the passive defense part, this application adopts a strategy with integration ideas in two places. Given that most remote sensing images have complex information and rich features, there are statistical feature differences between remote sensing datasets and adversarial samples in the representation of the middle layer of deep neural networks. The remote sensing dataset is marked as 0 and the adversarial sample is marked as 1. In a sub-network f i (i=1,2,...,N) to train n adversarial detectors d1(·), d2(·),..., d n (·). The output of each detector is integrated through the logistic regression classifier, and the confidence level p of whether the prediction is an adversarial sample is returned. i (i=1,2,...,N), and repeat the above steps in the N sub-networks in the deep ensemble network model to obtain the passive defense module.

[0060] The possible outputs of the passive detection module can be categorized into four categories: if the input is an adversarial example and the passive detection module correctly judges it, it is recorded as TP; if it is an incorrect judgment, it is recorded as F; if the input is a normal example and the passive detection module correctly judges it, it is recorded as TN; if it is an incorrect judgment, it is recorded as FP. The passive detection module is connected back-to-back with the actively reinforced deep integration model, where TP and FP are rejected first, while FN and TN are identified by the deep integration model.

[0061] Step 108 : Connect the passive defense module and the deep integration network model in sequence to construct a remote sensing image depth recognition model, and perform image recognition on the remote sensing image to be tested according to the remote sensing image depth recognition model.

[0062] In view of the complex information and rich features of remote sensing images, this paper combines the independent adversarial training of multiple deep models in active defense, the integration of multiple adversarial detection statistical features in passive defense, and the front-to-back connection of the two modules to jointly defend against adversarial attack images that drones may encounter when performing recognition tasks. This realizes the defense against remote sensing adversarial samples, improves the adversarial robustness of the drone visual recognition system, and thus improves the accuracy of deep recognition of remote sensing images.

[0063] In the above-mentioned remote sensing image deep recognition method based on feature correction and multi-level adversarial defense, this application fully utilizes the rich and complex features in remote sensing images, introduces a feature correction module, activates more non-robust features in adversarial training, and improves the robustness of the deep integration network model. The passive defense module is connected with the deep integration recognition model after active reinforcement to build a multi-level adversarial defense framework, achieving the highest level of adversarial robustness enhancement and reducing the threat of adversarial vulnerability posed by the deep recognition model on the intelligent drone. In the passive defense, multiple adversarial detection statistical features are integrated and the two modules are connected front and back to jointly defend against adversarial attack images that the drone may encounter when performing recognition tasks, thus achieving defense against remote sensing adversarial samples. The passive defense module and the actively reinforced deep integration neural network are deployed on a modern intelligent drone vision system. When the enemy obtains and maliciously attacks the remote sensing images captured by the drone camera and sensor through the channel, the adversarial robustness of the drone vision recognition system will be greatly improved, reducing the probability of system recognition errors and thus mission failure. In addition, this application can also use a relatively lightweight neural network as a sub-network architecture to reduce inference time and is suitable for edge environments.

[0064] In one embodiment, performing image recognition on a remote sensing image to be tested based on a remote sensing image depth recognition model includes:

[0065] Passing the remote sensing image to be tested through the passive defense module to obtain multiple confidence levels of the remote sensing image to be tested;

[0066] The average of multiple confidence levels of the remote sensing image to be tested is taken to obtain the overall confidence level;

[0067] A confidence threshold is set in advance. If the overall confidence is greater than the confidence threshold, the remote sensing image to be tested is determined to be an adversarial sample and rejected to proceed to the next step. If the overall confidence is less than the confidence threshold, the remote sensing image to be tested is determined to be a normal sample and passed.

[0068] The remote sensing image to be tested that passes through the passive defense module is passed to the deep integration network model for robust recognition to obtain the recognition category of the remote sensing image to be tested.

[0069] In a specific embodiment, Figure 3 and Figure 4 As shown, if there is a real-time remote sensing image x0, its sub-model f in the deep integration model i (i=1,2,...,N) The feature extracted from the lth layer is a l (x0), the hidden layer feature a can be calculated using the trained adversarial detector l (x0) multiple statistical index scores d1(a l (x0)), d2(al (x0)) and d3(a l (x0)). Complete the above steps on a total of L layers, extract features at each layer, and input them into three adversarial detectors to calculate the index score. For convenience, the score calculated by each adversarial detector is denoted as D i (x0)={d i (a1),d i (a2),...,d i (a L )}, i=1,2,3, and we get three sets of vectors. Let V(x0)={D1(x0),D2(x0),D3(x0)}.

[0070] Then, a simple logistic regression classifier is trained to calculate the posterior probability, that is, the confidence score of judging that the input real-time remote sensing image is an adversarial sample, that is, p(adv|V(x0))=(1+exp(β0+β T ·V(x0))) -1 β T is a weight vector used to fit the training data and adjust the importance of the three adversarial detectors. Finally, the confidence level p of whether the prediction is an adversarial sample is obtained.

[0071] The second integration is performed at the decision layer, that is, the above steps are performed simultaneously on N sub-networks to obtain a set of confidence vectors P = {p1, p2, p3}. The confidence scores p1, p2, p3 output by them are averaged, that is, The final overall confidence level p is obtained. A threshold θ is set. If p is greater than θ, the test image is considered an adversarial sample; if p is less than θ, the test image is considered a normal sample.

[0072] In one embodiment, the process of acquiring the remote sensing dataset includes:

[0073] Acquire multiple remote sensing images, perform cropping and normalization on the remote sensing images to obtain a normalized image set; randomly divide the normalized image set into a training set and a test set, preprocess the training set and the test set, and linearly transform the image data in the training set and the test set into a data set with a mean of 0 and a variance of 1 through the normalize operation to obtain a remote sensing data set.

[0074] In one embodiment, an adversarial attack is performed on a remote sensing dataset according to a PGD algorithm and perturbations are added to create adversarial samples, including:

[0075] According to the PGD algorithm, the remote sensing dataset is attacked and perturbations are added to create adversarial samples.

[0076]

[0077]

[0078] in, is the adversarial sample obtained by iterating the gradient of the remote sensing image x n times in the remote sensing dataset, S is the random ball projection, ε is the perturbation limit, α is the attack step size, represents the C&W attack, Z is the output of the softmax layer, t is the predicted category, i is any category except t, and -κ is the noise limit.

[0079] In one embodiment, the process of adding a feature correction module includes:

[0080] The process of adding a feature correction module includes a separation phase and a recalibration phase; the feature correction module includes a separation network and a fully connected network; the fully connected network is an auxiliary layer of the integrated model;

[0081] In the separation phase, a separation network is defined to learn the robustness of feature units and output a robustness map. A soft mask is set according to the robustness map, and the soft mask and the pre-set feature threshold are used to decompose the feature map to obtain robust features and non-robust features.

[0082] Robust features and non-robust features are added to the fully connected network. Based on the pre-set first loss function, the separation network is guided to assign higher robustness scores to feature units that help the auxiliary layer make correct decisions, helping the auxiliary layer to make correct predictions. Correct predictions include correct robust features and correct non-robust features.

[0083] Calibrate the correct non-robust features according to the calibration network to obtain calibrated non-robust features;

[0084] In the recalibration stage, the calibrated non-robust features are recalibrated according to the auxiliary layer to obtain the recalibrated non-robust features;

[0085] Output feature maps based on robust features and recalibrated non-robust features.

[0086] In a specific embodiment, during the general adversarial training process, the algorithm encourages the model to capture robust features and ignores some non-robust features. Non-robust features refer to general features that cannot constitute the correct prediction of the model, and they may become the target of the attacker. However, the non-robust activation features still contain a large number of discriminant clues, which are not used to enhance the adversarial robustness of the model. For remote sensing images with rich features and complex ground information, the above-mentioned phenomenon of discarding non-robust features and resulting in insufficient enhancement of adversarial robustness is very obvious. Taking into account the lightweight and real-time nature of inference of intelligent drone edge devices, in order to enable the deep neural network after adversarial training to reactivate some information in the non-robust features, such as Figure 2As shown, this application introduces a new module, the Feature Correction Module (FCM), in each sub-model of the deep ensemble. This module acts like a plug-in and does not affect the end-to-end adversarial training, with almost no additional computational overhead. The Feature Correction Module (FCM) is attached to the intermediate layer, including the separation and recalibration stages.

[0087] In the separation phase, a separation network is defined that learns the robustness of each feature unit and outputs a robustness map The robustness map contains the score of each feature unit. The higher the score, the stronger the robustness of the feature activation. To decompose the feature map, a soft mask based on r is defined using Gumbel softmax.

[0088]

[0089] Where σ(·) is the sigmoid function, g1 and g2 represent two random values ​​sampled from the Gumbel distribution. A threshold is set, and the score greater than the threshold corresponds to a robust feature, denoted by γ + , the rest are non-robust features γ - .

[0090] In order to learn the score in the robustness graph based on the impact of each feature unit on the correctness of the prediction, a fully connected network (MLP) is added as an auxiliary layer in the separation stage. Two feature units γ are randomly selected. + and γ - Input into the network and use the first loss function To guide the separation network to assign higher robustness scores to feature units that help the auxiliary layer make correct decisions, helping the auxiliary layer to make correct predictions.

[0091]

[0092] Among them, y c is the true label, is the corresponding confidence score; y c ′ is the wrong label with the highest confidence score, Give it a confidence score.

[0093] Get γ + and γ - After that, the calibration network R(·) is introduced to transform the non-robust feature γ - Adjust to The goal of the recalibration phase is to enable the non-stationary activation to recapture clues and help the model make correct decisions. In order to guide the calibration network to achieve this goal, this application uses the recalibration feature Afterwards the auxiliary layers are attached again and the recalibration loss is calculated. The calculation formula is: in, is the output prediction score of the auxiliary layer. By training this auxiliary layer to make correct decisions based on the recalibrated features, we can guide the calibration network to adjust the non-robust activations so that it provides clues related to the ground truth category.

[0094] After the recalibration phase, the robust features γ + and recalibrated non-robust features Thus, the output feature map is obtained And pass it to the subsequent layers of the model. Through the recalibration stage, more useful clues can be captured from non-robust activations, which are suppressed in traditional adversarial training. The FCM module is added to each sub-model undergoing adversarial training, and the overall loss function can be summarized as Among them, λ sep and λ rec is a hyperparameter used to control each sub-model f i (·) and The weight of . It can be any loss function of any adversarial training variant (such as TRADES, FAT, etc.).

[0095] In one embodiment, the first loss function is

[0096] Among them, y c is the true label, is the corresponding confidence score; y c ′ is the wrong label with the highest confidence score, is the confidence score.

[0097] In one embodiment, the recalibrated calibration loss function is

[0098]

[0099] in, is the output prediction score of the auxiliary layer, y c is the true label.

[0100] In one embodiment, the adversarial loss function is

[0101]

[0102] Among them, λ sep and λ rec is a hyperparameter, fi (·) represents a single sub-model in a deep ensemble network model, and Indicates different weights, is the loss function for any variant of adversarial training.

[0103] In one embodiment, the objective function of adversarial training is

[0104]

[0105] Among them, θ represents the network parameters of the deep integration network model, x represents the remote sensing image in the remote sensing dataset, and y represents the true label of the remote sensing image. represents the underlying data distribution of remote sensing images, δ represents the generated adversarial perturbation, represents the general constraint against perturbations, represents the loss function for training a deep neural network model f with parameters θ.

[0106] In one embodiment, the adversarial detector includes local intrinsic dimension, kernel density estimation, and Mahalanobis distance; the remote sensing image to be tested is passed through a passive defense module to obtain multiple confidence levels of the remote sensing image to be tested, including:

[0107] The remote sensing image to be tested is passed through the passive defense module, and the adversarial detector and logistic regression classifier are used to calculate the remote sensing image to be tested, and the confidence of the remote sensing image to be tested is obtained as

[0108] p(adv|V(x0))=(1+exp(β0+β T ·V(x0))) -1

[0109] V(x0)={D1(x0),D2(x0),D3(x0)}

[0110] D i (x0)={d i (a1),d i (a2),...,d i (a L )},i=1,2,3

[0111] Among them, d1 represents the local intrinsic dimension, d2 represents the kernel density estimate, d3 represents the Mahalanobis distance, and a L Denotes the feature map extracted by the sub-model layer l in the deep ensemble network model, D i (x0) represents the score counted by the adversarial detector, x0 represents the real-time remote sensing image to be tested, β T is the weight vector, and β0 represents the weight bias.

[0112] It should be understood that although Figure 1 The steps in the flowchart are shown in sequence as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. In addition, Figure 1 At least part of the steps may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least part of the sub-steps or stages of other steps.

[0113] Those skilled in the art will appreciate that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0114] The technical features of the above embodiments can be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0115] The above-described embodiments merely represent several implementation methods of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present invention. It should be noted that a person skilled in the art could make various modifications and improvements without departing from the spirit of the present application, all of which fall within the scope of protection of the present application. Therefore, the scope of protection of the present patent application shall be determined by the appended claims.

Claims

1. A remote sensing image depth recognition method based on feature correction and multi-level adversarial defense, characterized in that: The method comprises: Acquire a remote sensing dataset; perform an adversarial attack on the remote sensing dataset according to the PGD algorithm and add perturbations to create adversarial samples; the adversarial samples include remote sensing images and images after the adversarial attack; Adding a feature correction module to the middle layer of each convolutional neural network in multiple convolutional neural network sub-networks to construct an integrated model; performing adversarial training on the integrated model according to the adversarial sample and a preset adversarial loss function to obtain a deep integrated network model; Annotating the remote sensing dataset and the adversarial samples, training multiple adversarial detectors based on the annotated samples on each sub-network of the deep ensemble network model, and integrating the output of each adversarial detector through a logistic regression classifier to construct a passive defense module; The passive defense module and the deep integration network model are sequentially connected to construct a remote sensing image depth recognition model, and image recognition is performed on the remote sensing image to be tested according to the remote sensing image depth recognition model; The process of adding the feature correction module includes: The process of adding a feature correction module includes a separation phase and a recalibration phase; the feature correction module includes a separation network and a fully connected network; the fully connected network is an auxiliary layer of the integrated model; In the separation phase, a separation network is defined to learn the robustness of feature units and output a robustness map. A soft mask is set according to the robustness map, and the soft mask and a pre-set feature threshold are used to decompose the feature map to obtain robust features and non-robust features. Robust features and non-robust features are added to the fully connected network, and a pre-set first loss function is used to guide the separation network to assign a higher robustness score to feature units that help the auxiliary layer make correct decisions, thereby helping the auxiliary layer make correct predictions; the correct predictions include correct robust features and correct non-robust features; Calibrate the correct non-robust features according to the calibration network to obtain calibrated non-robust features; In the recalibration stage, the calibrated non-robust features are recalibrated according to the auxiliary layer to obtain the recalibrated non-robust features; Output a feature map according to the robust features and the recalibrated non-robust features; The first loss function is Among them, y c is the true label, is the corresponding confidence score; y′ c is the wrong label with the highest confidence score, is the confidence score; The calibration loss function of the recalibration is in, is the output prediction score of the auxiliary layer, y c is the true label.

2. The method according to claim 1, characterized in that Performing image recognition on the remote sensing image to be tested according to the remote sensing image depth recognition model includes: Passing the remote sensing image to be tested through the passive defense module to obtain multiple confidence levels of the remote sensing image to be tested; The average of multiple confidence levels of the remote sensing image to be tested is taken to obtain the overall confidence level; A confidence threshold is set in advance. If the overall confidence is greater than the confidence threshold, the remote sensing image to be tested is determined to be an adversarial sample and rejected to proceed to the next step. If the overall confidence is less than the confidence threshold, the remote sensing image to be tested is determined to be a normal sample and passed. The remote sensing image to be tested that passes through the passive defense module is passed to the deep integration network model for robust recognition to obtain the recognition category of the remote sensing image to be tested.

3. The method according to claim 1, characterized in that The process of acquiring the remote sensing dataset includes: Acquire multiple remote sensing images, perform cropping and normalization on the remote sensing images to obtain a normalized image set; randomly divide the normalized image set into a training set and a test set, perform preprocessing on the training set and the test set, and linearly convert the image data in the training set and the test set into a data set with a mean of 0 and a variance of 1 through a normalize operation to obtain a remote sensing data set.

4. The method according to any one of claims 1 to 3, characterized in that Conduct an adversarial attack on the remote sensing dataset using the PGD algorithm and add perturbations to create adversarial samples, including: According to the PGD algorithm, the remote sensing dataset is attacked and perturbations are added to create adversarial samples. in, is the adversarial sample obtained by iterating the gradient of the remote sensing image x n times in the remote sensing dataset, S is the random ball projection, ε is the perturbation limit, α is the attack step size, represents the C&W attack, Z is the output of the softmax layer, t is the predicted category, i is any category except t, and -κ is the noise limit.

5. The method according to claim 1, wherein The adversarial loss function is Among them, λ sep and λ rec is a hyperparameter, f i (·) represents a single sub-model in a deep ensemble network model, and Indicates different weights, is the loss function for any variant of adversarial training.

6. The method according to claim 1, characterized in that The objective function of the adversarial training is Among them, θ represents the network parameters of the deep integration network model, x represents the remote sensing image in the remote sensing dataset, and y represents the true label of the remote sensing image. represents the underlying data distribution of remote sensing images, δ represents the generated adversarial perturbation, represents the general constraint against perturbations, represents the loss function for training a deep neural network model f with parameters θ.

7. The method according to claim 2, characterized in that The adversarial detector includes local intrinsic dimension, kernel density estimation and Mahalanobis distance; the remote sensing image to be tested is passed through the passive defense module to obtain multiple confidence levels of the remote sensing image to be tested, including: The remote sensing image to be tested is passed through the passive defense module, and the adversarial detector and logistic regression classifier are used to calculate the remote sensing image to be tested, and the confidence of the remote sensing image to be tested is obtained as p(adv|V(x0))=(1+exp(β0+β T ·V(x0))) -1 V(x0)={D1(x0),D2(x0),D3(x0)} <h2 style=";text-align:left;direction:ltr">D<h2 style=";text-align:left;direction:ltr"> i <h2 style=";text-align:left;direction:ltr"> (x0) = {d<h2 style=";text-align:left;direction:ltr"> i <h2 style=";text-align:left;direction:ltr"> (a1),d<h2 style=";text-align:left;direction:ltr"> i <h2 style=";text-align:left;direction:ltr"> (a2),...,d<h2 style=";text-align:left;direction:ltr"> i <h2 style=";text-align:left;direction:ltr"> (a<h2 style=";text-align:left;direction:ltr"> L <h2 style=";text-align:left;direction:ltr"> )},i = 1,2,3 Among them, d1 represents the local intrinsic dimension, d2 represents the kernel density estimate, d3 represents the Mahalanobis distance, and a L Denotes the feature map extracted by the sub-model layer l in the deep ensemble network model, D i (x0) represents the score counted by the adversarial detector, x0 represents the real-time remote sensing image to be tested, β T is the weight vector, and β0 represents the weight bias.

Images