Counter-defense method based on multi-view image super-resolution

Abstract

The invention discloses a confrontation and defense method based on multi-view image super-resolution, and the method comprises the steps: obtaining an attacked image, and enabling the attacked image and a clean image to form a clean image and attacked image pair; inputting the attacked image and the auxiliary view angle image into a network for coding to obtain implicit representation; obtaining a homography matrix from an auxiliary view angle image to an attacked image, and performing spatial transformation on the auxiliary view angle image on a feature level to realize spatial alignment; implicit representation is fused in a feature space through a decoding function, a clean pixel value is obtained by polling each pixel position, and defensive output after the network purifies the attacked image is obtained. The method does not depend on the limitation of a single view angle image, reduces the dependence on the consistency of observation angles, effectively achieves the purification of adversarial disturbance, improves the visual quality of the image, enhances the defense capability of the network for potential adversarial attacks, and can be flexibly applied to an image processing robustness task in a complex real environment.

Description

Technical Field

[0001] The present invention belongs to the field of artificial intelligence and computer vision, and relates to a counterattack and defense method, specifically a counterattack defense method based on multi-view image super-resolution. Background Art

[0002] Adversarial defense is a technique designed to protect deep learning models from adversarial attacks. Adversarial attacks trick deep learning models into making incorrect judgments by making subtle but intentional modifications to input data, which is particularly common in the field of image classification. Therefore, the goal of adversarial defense is to enhance the robustness of the model so that it can resist or at least identify tampered input data. Existing defense methods can be divided into defense based on modifying the model and defense based on modifying the input. The defense method based on modifying the input is to eliminate or reduce the adversarial perturbations in the adversarial sample by performing some preprocessing or transformation on the input image without changing the model structure. The most advanced methods currently use data enhancement, feature compression and other technologies to suppress adversarial noise in the image. However, it should be noted that with the continuous advancement of adversarial attack technology, it is still challenging to develop more effective defense strategies. For data enhancement defense methods, when facing a variety of subtle attacks such as noise injection, pattern distortion or semantic tampering, current technologies do not introduce external information, and it is difficult to guarantee complete effectiveness or show good generalization ability. Further studies have pointed out that deep learning models are extremely sensitive to different perspectives of objects, so that categories that can be correctly identified from certain, familiar perspectives are easily misclassified from most other perspectives, making the vulnerability of deep learning models in dealing with perspective diversity itself easy to become a means of adversarial attacks. At the same time, single-perspective images may have interference factors such as posture and occlusion, and the visual information conveyed is easily lost after being attacked. Multi-perspective images capture the same scene from different perspectives. Through their differences and diversity, they can make up for the shortcomings of single-perspective images when attacked, and have certain natural advantages in adversarial defense tasks. Summary of the invention

[0003] In view of the above-mentioned prior art, the present invention provides an adversarial defense method based on multi-view image super-resolution, which can effectively restore image details by aligning and fusing multi-view information and reconstructing the image using local implicit functions. The present invention does not rely on the limitations of single-view images, reduces the dependence on the consistency of observation angles, effectively realizes the purification of adversarial disturbances, not only improves the visual quality of the image, but also enhances the defense capability against potential adversarial attacks, and can be flexibly applied to image processing robustness tasks in complex real environments.

[0004] In order to solve the above technical problems, the present invention proposes an adversarial defense method based on multi-view image super-resolution, comprising the following steps:

[0005] Step 1: The original unmodified clean image C obtained from the multi-view image dataset is attacked using the PGD (Projected Gradient Descent) attack algorithm to generate an attacked image A, and the clean image C is combined with the attacked image A to form a clean image-attacked image pair (A, C). This process is repeated until all N clean images in the dataset are processed to form an image pair set D.

[0006] Step 2: Take an auxiliary view image V of the clean image C from the multi-view image dataset, take the attacked image A of the clean image C from the image pair set D, input the attacked image) and the auxiliary view image V into the EDSR (Enhanced Deep Super-Resolution) encoder (the network structure is a mature method in the industry and does not require an explanation of the architecture) for encoding, and obtain the implicit representation (f A , f V ), and use the features to expand the content of the rich representation, and get

[0007] Step 3: Use the homography pre-alignment module of the network of this method to extract deep features for alignment and learn a content-aware mask for rejecting abnormal areas. After weighting the two, the homography estimator provided to the module estimates the homography and obtains a homography matrix H from the auxiliary view image V to the attacked image A. H is used to implicitly represent the auxiliary view image V. Perform spatial transformation at the feature level to achieve perspective alignment and obtain the aligned auxiliary perspective implicit representation

[0008] Step 4: In the feature space, a decoding function with a ReLU (Rectified Linear Unit) activation function and a 5-layer MLP (Multiple Layer Perception) structure with 256 hidden layers is used to decode the implicit representation obtained in step 2 for each pixel position of the attacked image A. The features of the position are obtained from the aligned auxiliary perspective. The feature corresponding to the position is also found in , and the cosine similarity between the two features is calculated to evaluate their similarity. If the similarity is greater than or equal to the preset threshold, and Perform weighted fusion, otherwise these two features will not be considered, and obtain the fused feature F fusion . F fusion It is further decoded by the decoding function into the purified RGB value p i , all M pixel positions are polled by the decoding function to obtain the clean RGB values The network is formed to purify the attacked image A and obtain the defended image O.

[0009] Furthermore, the adversarial defense method based on multi-view image super-resolution described in the present invention comprises:

[0010] Step 1: The specific method for obtaining the set D of clean image-attacked image pairs is as follows:

[0011] (1-1) Select classifier CN.

[0012] (1-2) For each original, unmodified clean image C obtained from the multi-view image dataset C i , attack the classifier CN through the PGD attack method to generate the attacked image A i .

[0013] (1-3) The obtained clean image C i and the attacked image A i Compose a clean image-attacked image pair D i =(A i , C i ).

[0014] (1-4) Repeat (1-2) and (1-3) until there are N clean images C in the dataset. i are processed to form an image pair set D,

[0015] In step 1, the specific method of generating the attacked image is: using PGD attack as the attack method, the maximum amplitude of the infinite norm perturbation ε ∞ The kernel size is 8 / 255, the step size is 2 / 255, and the number of steps is 100, respectively attacking the ResNet18 and WideResNet28 classifiers trained on the original dataset.

[0016] In step 2, the attacked image A and the auxiliary view image V are input into the EDSR encoder for encoding to obtain the implicit representation (f A , f v) is as follows: The EDSR encoder is a sequence of 15 residual blocks, each of which consists of a convolutional layer, a ReLu layer, and a second convolutional layer to extract pixel-by-pixel features of the input image. The encoder represents the input image as a two-dimensional feature map with a channel number C of 64, a length H and a width W equal to the dimensions of the input image. The feature at position [i, j] is denoted as f ij The H×W feature vectors obtained by the encoder are evenly distributed in the two-dimensional space of the continuous image domain. The two received inputs are mapped to the same low-dimensional latent space through the same encoder and become implicit representations (f A , f V ).

[0017] In step 2, the specific method of using features to expand the content of the representation when obtaining the implicit representation is: for f∈(f A , f V ) Apply feature expansion to obtain The feature vector is the concatenation of the 3×3 neighborhood feature vectors in f. Formally, the feature expansion for positions j and k is defined as: The Concat operation refers to the concatenation of a set of vectors and filling the original feature vector boundaries with zero vectors.

[0018] In step three, the specific method of using the homography pre-alignment module of the network of this method to obtain a homography matrix H from the auxiliary view image V to the attacked image A is as follows: the homography pre-alignment module receives two input images, the auxiliary view image V and the attacked image A, and after converting them into grayscale images, generates a homography matrix H from V to A as output. The entire module can be divided into three submodules: feature extractor w(·), mask predictor m(·), and homography estimator h(·). This module is a plug-in of the overall architecture and is independently trained using a ternary loss to constrain the image A′ deformed by the homography matrix to be farther from A and closer to V. w(·) and m(·) are fully convolutional networks that accept inputs of any size. The specific steps are:

[0019] (3-1)w(·) learns the deep features for alignment, the size is H×W×C, where H is the length, W is the width, and C is the number of channels, denoted as W x =w(x), x∈{A,V}.

[0020] (3-2) m(·) learns a content-aware mask for rejecting anomalous regions. Specifically, submodule m(·) learns to generate a probability map or mask that highlights the content in the feature map that contributes greatly to the homography estimation, while outliers are perceived and discarded and are not used for homography estimation. The size of the mask is the same as the size of the feature map extracted by w(·), denoted as M x=m(x), x∈{A,V}.

[0021] (3-3) Using the mask, the features are further weighted and then provided to the homography estimator to obtain two weighted feature maps G a and G v : G x =W x M x , x∈{A,V}.

[0022] (3-4)h(·) has the same structure as w(·), generating 4 2D offset vectors, with a total of 8 values ​​as the elements of the homography matrix. Given a weighted feature map G a and G v , connect them to construct a feature map [G a , G v ], with a size of H×W×2C, is input into the homography estimator, and after obtaining the offset vector, a linear system is solved to obtain the homography matrix H with 8 degrees of freedom av :H av =h([G a , G v ]).

[0023] (3-5) After estimating the homography matrix H av Based on the features of the auxiliary view image V Perform homography transformation to adjust the spatial layout while maintaining the features to achieve spatial alignment. The aligned features are:

[0024] In step 4, the decoding function fuses the implicit representations of different perspectives to obtain the fused feature F fusion The specific method is as follows: the decoding function is a 5-layer MLP with ReLU activation function and 256 hidden layer dimensions. Given a pixel position coordinate q, the decoding function first finds the pixel position coordinate v in the input image that is closest to q * =[i * , j * ] and the corresponding features And find the same position coordinate v′=v in the corresponding auxiliary view image * The characteristic f i′j′ Then, the decoding function considers v * and v′ neighborhood features to predict the clean RGB value p i＝q The specific steps are:

[0025] (4-1) Feature similarity calculation: For the neighborhood feature map, calculate the cosine similarity between feature vectors at the same spatial position

[0026] (4-2) Similarity threshold judgment: For the preset similarity threshold τ, when S(i, j) < τ, it is considered that the features of the two perspectives at this position have no reference significance, and the features of the auxiliary perspective are not used; when S(i, j) ≥ τ, it is considered that the features of the two perspectives can be fused.

[0027] (4-3) Weighted feature fusion: For features with similarity higher than a threshold value τ, these features are weighted according to the similarity between the features, and features with high similarity are given greater weight.

[0028]

[0029] In step 4, F fusion It is further decoded by the decoding function into the purified RGB value p i , all M pixel positions are polled by the decoding function to obtain the clean RGB values The specific method of forming the defended image O obtained after the network purifies the attacked image A is as follows: all images share a decoding function S=s with θ as a parameter θ (f, p), this function maps coordinates to RGB values, where f is a vector, p is a two-dimensional coordinate in the continuous image domain, and s is the predicted signal RGB value. i and its implicit representation f i , define the RGB value at coordinate q as: I i (q) = s θ (z * ,qv * ), where z * Yes i The eigenvector with the closest Euclidean distance to q, v * is feature z * The coordinates in the image domain. After taking the weights into account, the above formula is expanded to: Where z t * (t∈{00, 01, 10, 11}) is the closest feature in the upper left, upper right, lower left, and lower right subspaces, v t * Yes t * The coordinates of R t It is q and v t′ * The area of ​​the rectangle between , where t′ is the diagonal position of t. The feature map f iOutside the boundary, the filling is mirrored, so the above formula also applies near the boundary. The decoding function then votes to combine these four signals using the area as the confidence, and the normalized confidence is proportional to the area of ​​the rectangle between the query point and its nearest potential feature diagonal corresponding point.

[0030] Compared with the prior art, the present invention has the following beneficial effects:

[0031] (1) In the process of completing the adversarial defense, the present invention utilizes the information complementarity of multiple perspectives and the richness of details and textures between subtle differences in perspectives, which can effectively reconstruct image details and help the super-resolution algorithm better understand the structure of the object. At the same time, since the image implicit representation describes the entire image in a parameterized manner, the natural ability to efficiently compress image information enables it to achieve better super-resolution effects under limited network complexity, thereby achieving better defense effects.

[0032] (2) The present invention separates the training of defense from the training of classifiers. When the actual settings (such as architecture, hyperparameters or number of classes) change, there is no need to retrain the classifier, which reduces the complexity of defense. In addition, since the present method is based on a local statistical model, the parameter efficiency and computational efficiency are high. Therefore, the present invention can basically ensure a low processing time and quickly adapt to near-real-time application requirements, such as dynamic image countermeasures defense, real-time security protection and the like. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 The present invention is a flowchart of an adversarial defense method based on multi-view image super-resolution. DETAILED DESCRIPTION

[0034] The design concept of an adversarial defense method based on multi-view image super-resolution proposed in the present invention is to purify the image disturbance caused by adversarial attacks by utilizing the rich complementary information of multi-view images and super-resolution technology to achieve the purpose of adversarial defense. Multi-view images contain scene and object information captured from different angles, which can provide additional context and depth perception for the model, and help the model to better restore the original, unattacked image details when facing adversarial disturbances that damage image quality. Super-resolution technology provides an effective means to restore the details and clarity of the image by reconstructing high-resolution images to compensate for the information loss caused by the attack. Therefore, by utilizing super-resolution technology, this method attempts to not only reduce or eliminate the pixel-level disturbances introduced by adversarial attacks, but also improve the resolution and quality of the image on this basis. Combining the application of multi-view images and super-resolution technology, the present invention aims to effectively purify adversarial disturbances, achieve adversarial defense, significantly improve the robustness of the network in the face of adversarial attacks, and ensure the security and reliability of image data.

[0035] The technical solution of the present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments. The described specific embodiments are only used to explain the present invention and are not intended to limit the present invention.

[0036] 1. Obtain a set of clean image-attacked image pairs

[0037] Before network training and adversarial defense inference, obtain the attacked image (Adversarialimage), and clean image (Clean image) forms a clean image and an attacked image pair The specific steps are as follows:

[0038] (1) Select classifier CN.

[0039] (2) For each original, unmodified clean image C obtained from the multi-view image dataset C i , attack the classifier CN through the PGD attack method to generate the attacked image A i .

[0040] (3) The obtained clean image C i and the attacked image A i Compose a clean image-attacked image pair D i =(A i , C i ).

[0041] (4) Repeat (2) and (3) until there are N clean images C in the dataset. i are processed to form an image pair set D,

[0042] The specific method of creating the attacked image is: using PGD attack as the attack method, the maximum amplitude of the infinite norm perturbation ε ∞ The kernel size is 8 / 255, the step size is 2 / 255, and the number of steps is 100, respectively attacking the ResNet18 and WideResNet28 classifiers trained on the original dataset.

[0043] 2. Encode the input

[0044] The attacked image A and the auxiliary view image V are input into the EDSR encoder for encoding to obtain the implicit representation (f A , f V ).

[0045] Encoder structure. The encoder extracts pixel-by-pixel features of the input image. It uses an EDSR design and is a sequence of 15 residual blocks. Each residual block consists of a convolutional layer, a ReLu layer, and a second convolutional layer. The encoder represents a continuous image as a two-dimensional feature map with a channel number C of 64, a length H, and a width W. The feature at position [i, j] is denoted as f ij The H×W feature vectors obtained by the encoder are evenly distributed in the two-dimensional space of the continuous image domain. The two received inputs are mapped to the same low-dimensional latent space through the same encoder and become implicit representations (f A , f V ).

[0046] Feature expansion strategy. In order to enrich the information contained in each potential feature in f, this method performs A , f V ) Apply the feature expansion to get The feature vector in is the concatenation of the 3×3 neighborhood feature vectors in f. Formally, the feature expansion is defined as:

[0047]

[0048] The Concat operation refers to the concatenation of a set of vectors, and f is padded with zero vectors at its boundaries. This strategy is applied to all calculations. For simplicity, only the f symbol will be used in the following sections to represent implicit representations.

[0049] 3. Homography Pre-Alignment of Multi-View Images

[0050] The homography pre-alignment module of the network of this method is used to obtain a homography matrix H from the auxiliary view image V to the attacked image A, and H is used to perform spatial transformation on the auxiliary view image V at the feature level, and its spatial layout is adjusted while maintaining the features to achieve view alignment, and obtain the implicit representation of the aligned auxiliary view

[0051] Homography pre-alignment strategy. The calculation of the homography matrix can be regarded as a mapping problem from the input image pair to the homography transformation parameters: the homography pre-alignment module receives two input images, the auxiliary view image V and the attacked image A, and after converting them to grayscale images, generates the homography matrix H from V to A as output. The entire module can be divided into two submodules: feature extractor w(·), mask predictor m(·), and homography estimator h(·). This module is a plug-in of the overall architecture and is independently trained using a ternary loss to constrain the image A′ deformed by the homography matrix to be farther from A and closer to V. w(·) and m(·) are fully convolutional networks that accept inputs of any size, where w(·) learns deep features for alignment, with a size of H×W×C, where H is the length, W is the width, and C is the number of channels.

[0052] W x =w(x),x∈{A,V}

[0053] m(·) learns a content-aware mask for rejecting anomalous regions. Specifically, submodule m(·) learns to generate a probability map or mask that highlights the content in the feature map that contributes greatly to the homography estimate, while outliers are sensed and discarded and are not used for homography estimation. The size of the mask is the same as the size of the feature map extracted by w(·).

[0054] M x =m(x),x∈{A,V}

[0055] Using the mask, the features are further weighted and then fed to the homography estimator to obtain two weighted feature maps G a and G b :

[0056] G x =W x M x , x∈{A,V}

[0057] The structure of h(·) is the same as w(·), generating 4 2D offset vectors, with a total of 8 values ​​as the elements of the homography matrix. Given a weighted feature map G a and G v , connect them to construct a feature map [G a , G v ], with a size of H×W×2C, is input into the homography estimator. After obtaining the offset vector, a linear system is solved to obtain the homography matrix H with 8 degrees of freedom. av :

[0058] H av =h([G a , G v ]).

[0059] After estimating the homography matrix H av Based on the features of the auxiliary view image V Perform homography transformation to adjust the spatial layout while maintaining the features to achieve spatial alignment. The aligned features are:

[0060] 4. Decoding the features

[0061] In the feature space, the implicit representation obtained in step 2 is used for each pixel position of the attacked image A using a decoding function with a ReLU (Rectified Linear Unit) activation function and a 5-layer MLP (Multiple Layer Perception) structure with 256 hidden layer dimensions. The features of the position are obtained from the aligned auxiliary perspective. The feature corresponding to the position is also found in , and the cosine similarity between the two features is calculated to evaluate their similarity. If the similarity is greater than or equal to the preset threshold, and Perform weighted fusion, otherwise these two features will not be considered, and obtain the fused feature F fusio n. F fusion It is further decoded by the decoding function into the purified RGB value p i , all M pixel positions are polled by the decoding function to obtain the clean RGB values The network is formed to purify the attacked image A and obtain the defended image O.

[0062] Decoder structure. The decoding function is a 5-layer MLP with ReLU activation function and 256 hidden layer dimensions. All images share a decoding function S=s with θ as parameter θ (f, p), this function maps coordinates to RGB values, where f is a vector, p is a two-dimensional coordinate in the continuous image domain, and s is the predicted signal RGB value. i and its implicit representation f i , define the RGB value at coordinate q as:

[0063] I i (q) = s θ (z * ,qv * )

[0064] where z * Yes i The eigenvector with the closest Euclidean distance to q, v * is feature z *coordinates in the image domain. Since the decoder is modeled locally, qv * A relative coordinate is formed, and then input into the decoder together with the feature z* at v* to query the value of the pixel. At this time, the decoder's prediction of the pixel value is still discontinuous. Specifically, since the signal prediction at q is by querying the nearest potential code z in f * When q moves in the 2D domain, multiple v that meet the conditions can appear simultaneously in the image coordinate domain. * , so z * The choice can suddenly switch from one to another, showing mutation. Therefore, the above formula is expanded to:

[0065]

[0066] Where z t * (t∈{00, 01, 10, 11}) is the closest feature in the upper left, upper right, lower left, and lower right subspaces, v t * Yes t * The coordinates of R t It is q and v t′ * The area of ​​the rectangle between , where t′ is the diagonal position of t. The feature map f i Outside the boundary, it is mirrored and filled, so the above formula also applies near the boundary. This formula refines the resolution grid and overlaps the local block with its neighboring blocks. At each coordinate, there are four features of independent prediction signals. The decoding function then uses the area as the confidence and votes to merge these four signals. The normalized confidence is proportional to the area of ​​the rectangle between the query point and its nearest potential feature diagonal corresponding point. Therefore, the confidence will be higher when the query coordinate is closer, making the decoding function in z * Since this module achieves a continuous representation of the image and the decoder decoding is based only on adjacent pixels, the original grid size is not important, making the decoding result independent of the size or resolution of the original image.

[0067] Multi-view local fusion strategy. Given a pixel position coordinate q, the decoding function first finds the pixel position coordinate v closest to q in the input image. * =[i * , j * ] and the corresponding features And find the same position coordinate v′=v in the corresponding auxiliary view image * The characteristic f i′j′ Then, the decoding function considers v * and v′ neighborhood features to predict the clean RGB value pi＝q The specific steps are as follows:

[0068] (1) Feature similarity calculation: For the neighborhood feature map, the cosine similarity between feature vectors is calculated at the same spatial position.

[0069] (2) Similarity threshold judgment: For the preset similarity threshold τ, when S(i, j) < τ, it is considered that the features of the two perspectives at this position have no reference significance, and the features of the auxiliary perspective are not used; when S(i, j) ≥ τ, it is considered that the features of the two perspectives can be fused.

[0070] (3) Weighted feature fusion: For features with similarity higher than a threshold value τ, these features are weighted according to the similarity between them, and features with high similarity are given greater weight.

[0071]

[0072] Predict the purified defense image. Ideally, the predicted clean RGB value is consistent with the clean image at that position. The pixel values ​​of the predicted clean RGB values ​​are the same, that is, the noise added by the adversarial attack can be completely removed. In practice, the predicted clean RGB value is close to the pixel value of the clean image at that position, and the closer it is to the clean image, the better. The similarity measurement methods of SSIM (Structural Similarity Index Measure) and MSE (Mean Squared Error) are used to measure the quality of the defended image by comparing the pixel values, structure and brightness differences between the clean image and the defended image.

[0073] Efficiency of the network structure. The defense process of the present invention is not computationally intensive, because the encoder feature map is only calculated once and used to predict the RGB values ​​of all query pixel positions. In addition, the classifier and attack method used when generating training image pairs can be different from the classifier and attack method used during inference, so the defense effect is more flexible and robust.

[0074] In the training phase, the L1 loss between the clean image C and the defensive output O is calculated and minimized based on their RGB values. In the inference phase, an image is input and the output size is specified to obtain the defensive image O.

[0075] Training of the network. Use random patch pairs cropped at the same position of images A, V, and C for training, for example, random patches of size 48×48 are sampled from the training pairs in the dataset as the true input of the network. The network is trained to minimize the L1 loss between the RGB values ​​of the clean image C and the defensive output O.

[0076] Inference of the network. During the inference phase, the network outputs the predicted fortified image. The fortified image O can be synthesized by predicting the RGB values ​​of each pixel position in a grid of size H0×W0×3. The size of the fortified image O is generally the same as the size of the input image. This method does not require the size of O to be the same as the size of the input image, and can arbitrarily specify the size for output to achieve the effect of simultaneous super-resolution.

[0077] Example

[0078] The feasibility of the method of the present invention is verified by combining specific examples, as described below:

[0079] In this example, 40 epochs were trained on Cifar10 and Cifar100, the learning rate was set to 0.0001, and the Adam optimizer was used. In order to create training pairs, the PGD attack was used as the attack method (ε ∞ The attack and defense evaluation parts use the Autoattack and RobustBench baselines. All experiments are performed on a single Nvidia 3060ti GPU using Pytorch, and each result is averaged over 3 experiments.

[0080] The present invention defines the following two types of defense capabilities:

[0081] (1) Standard Accuracy (SA): refers to the classification accuracy of the model on a clean dataset (data that has not been subjected to adversarial attacks). It measures the model’s ability to correctly classify input data under normal conditions.

[0082] (2) Robust Accuracy (RA): refers to the classification accuracy of the model when facing adversarial samples. It measures the performance of the model in resisting these intentionally created perturbations and reflects the robustness of the model.

[0083] Table 1 Comparison of experimental results between the method of the present invention and the existing optimal method

[0084] method SA RA average Dataset No Defense 92.25 0 46.13 Cifar10 Method (1) 85.46 60.33 72.90 Cifar10 Method of the present invention 86.12 71.58 78.85 Cifar10 No Defense 91.38 0 45.69 Cifar100 Method (1) 86.61 78.8 85.09 Cifar100 Method of the present invention 87.27 83.74 85.5 Cifar100

[0085] The results of the present method and the best method among the existing adversarial defense methods based on modified input (the method of directly using generated data to enhance the model adversarial defense (1)) shown in Table 1 show that: without defense, the attack deceives the classifier on almost all examples, while the image examples after defense can be reclassified correctly by the classifier to a certain extent, and the image examples after defense by the method of the present invention can be reclassified correctly by the classifier with a high probability. The defense effect of the method of the present invention is better than the best method among the existing adversarial defense methods based on modified input. Therefore, the feasibility and superiority of the invented method can be demonstrated.

[0086] Although the present invention has been described above in conjunction with the accompanying drawings, the present invention is not limited to the above-mentioned specific embodiments, which are merely illustrative rather than restrictive. Under the guidance of the present invention, ordinary technicians in this field can make many improvements and changes without departing from the purpose of the present invention, which are all within the protection of the present invention.

Claims

1. An adversarial defense method based on multi-view image super-resolution, It is characterized in that The method comprises the following steps: Step 1: The original unmodified clean image C obtained from the multi-view image dataset is attacked using the PGD attack algorithm to generate an attacked image A, and the clean image C is combined with the attacked image A to form a clean image-attacked image pair (A, C); this process is repeated until all N clean images in the dataset are processed to form an image pair set D. Step 2: Take an auxiliary view image V of the clean image C from the multi-view image dataset, take the attacked image A of the clean image C from the image pair set D, input the attacked image A and the auxiliary view image V into the EDSR encoder for encoding, and obtain the implicit representation of the attacked image A and the auxiliary view image V (f A , f V ), and use the features to expand the content of the rich representation, and get Step 3: Use the homography pre-alignment module of the network of this method to extract deep features for alignment and learn a content-aware mask for rejecting abnormal areas. After weighting the two, the homography estimator provided to the module estimates the homography and obtains a homography matrix H from the auxiliary view image V to the attacked image A. H is used to implicitly represent the auxiliary view image V. Perform spatial transformation at the feature level to achieve perspective alignment and obtain the aligned auxiliary perspective implicit representation Step 4: In the feature space, the implicit representation obtained in step 2 is obtained for each pixel position of the attacked image A using the decoding function of a 5-layer MLP structure with ReLU activation function and 256 hidden layer dimensions. The features of the position are obtained from the aligned auxiliary perspective. The feature corresponding to the position is also found in , and the cosine similarity between the two features is calculated to evaluate their similarity. If the similarity is greater than or equal to the preset threshold, and Perform weighted fusion, otherwise these two features will not be considered, and obtain the fused feature F fusion ; F fusion It is further decoded by the decoding function into the purified RGB value p i , all M pixel positions are polled by the decoding function to obtain the clean RGB values The network is formed to purify the attacked image A and obtain the defended image O.

2. The adversarial defense method based on multi-view image super-resolution according to claim 1, It is characterized in that In step 1, the specific method for obtaining the set D of clean image-attacked image pairs is: 1-1) Select classifier CN; 1-2) For each original unmodified clean image C obtained from the multi-view image dataset C i , attack the classifier CN through the PGD attack method to generate the attacked image A i ; 1-3) The clean image C i and the attacked image A i Compose a clean image-attacked image pair D i =(A i , C i ); 1-4) Repeat 1-2) and 1-3) until there are N clean images C in the dataset i are processed to form an image pair set D, 3. The adversarial defense method based on multi-view image super-resolution according to claim 1, It is characterized in that In step 1, the specific method of generating the attacked image A is: using PGD attack as the attack method, the maximum amplitude of the infinite norm perturbation ε ∞ The kernel size is 8 / 255, the step size is 2 / 255, and the number of steps is 100, respectively attacking the ResNet18 and WideResNet28 classifiers trained on the original dataset.

4. The adversarial defense method based on multi-view image super-resolution according to claim 1, It is characterized in that In the step 2, the attacked image A and the auxiliary view image V are input into the EDSR encoder for encoding to obtain an implicit representation (f A , f V ) is as follows: the EDSR encoder is a sequence of 15 residual blocks, each of which consists of a convolutional layer, a ReLu layer, and a second convolutional layer to extract pixel-by-pixel features of the input image; the encoder represents the input image as a two-dimensional feature map with a channel number C of 64, a length H, and a width W equal to the dimension of the input image. The feature at position [i, j] is denoted as f ij ; The H×W feature vectors obtained by the encoder are evenly distributed in the two-dimensional space of the continuous image domain; the two received inputs are mapped to the same low-dimensional latent space through the same encoder and become implicit representations (f A , f V ).

5. The adversarial defense method based on multi-view image super-resolution according to claim 1, It is characterized in that In the step 2, when obtaining the implicit representation of the attacked image A and the auxiliary view image V, the specific method of using the feature to expand the content of the representation is: for f∈(f A , f V ) obtained by applying feature expansion The feature vector is the concatenation of the 3×3 neighborhood feature vectors in f; formally, the feature expansion for positions j and k is defined as: The Concat operation refers to the concatenation of a set of vectors and filling the original feature vector boundaries with zero vectors.

6. The adversarial defense method based on multi-view image super-resolution according to claim 1, It is characterized in that In the step 3, the specific method of using the homography pre-alignment module of the network of the method to obtain a homography matrix H from the auxiliary view image V to the attacked image A is as follows: the homography pre-alignment module receives two input images, the auxiliary view image V and the attacked image A, and after converting them into grayscale images, generates a homography matrix H from V to A as an output; the entire module can be divided into three submodules: a feature extractor w(·), a mask predictor m(·) and a homography estimator h(·); the module is a plug-in of the overall architecture, and is independently trained using a ternary loss to constrain the image A′ deformed by the homography matrix to be farther from A and closer to V; w(·) and m(·) are fully convolutional networks that accept inputs of any size; the specific steps are: 3-1) w(·) learns deep features for alignment, with a size of H×W×C, where H is the length, W is the width, and C is the number of channels, denoted as W x =w(x), x∈{A,V}; 3-2) m(·) learns a content-aware mask for rejecting anomalous regions; specifically, submodule m(·) learns to generate a probability map or mask that highlights the content in the feature map that contributes greatly to the homography estimation, while outliers are perceived and discarded and are not used for homography estimation; the size of the mask is the same as the size of the feature map extracted by w(·), denoted as M x = m(x), x∈(A, V}; 3-3) Using the mask, the features are further weighted and then provided to the homography estimator to obtain two weighted feature maps G a and G v : G x =W x M x , x∈{A,V}; 3-4) h(·) has the same structure as w(·), generating 4 2D offset vectors, with a total of 8 values ​​as the elements of the homography matrix; given the weighted feature map G a and G v , connect them to construct a feature map [G a , G v ], with a size of H×W×2C, is input into the homography estimator, and after obtaining the offset vector, a linear system is solved to obtain the homography matrix H with 8 degrees of freedom av :H av =h([G a , G v ]); 3-5) After estimating the homography matrix H av Based on the features of the auxiliary view image V Perform homography transformation to adjust the spatial layout while maintaining the features to achieve spatial alignment. The aligned features are:

7. The adversarial defense method based on multi-view image super-resolution according to claim 1, It is characterized in that In step 4, the decoding function fuses the implicit representations of different perspectives to obtain the fused feature F fusion The specific method is as follows: the decoding function is a 5-layer MLP with ReLU activation function and 256 hidden layer dimensions; given a pixel position coordinate q, the decoding function first finds the pixel position coordinate v closest to q in the input image * =[i * , j * ] and the corresponding features And find the same position coordinate v′=v in the corresponding auxiliary view image * The characteristic f i′j′ ; Then, the decoding function considers v * and v′ neighborhood features to predict the clean RGB value p i＝q ; The specific steps are: 4-1) Feature similarity calculation: For the neighborhood feature map, calculate the cosine similarity between feature vectors at the same spatial position 4-2) Similarity threshold judgment: For the preset similarity threshold τ, when S(i, j) < τ, it is considered that the features of the two perspectives at this position have no reference significance, and the features of the auxiliary perspective are not used; when S(i, j) ≥ τ, it is considered that the features of the two perspectives can be fused; 4-3) Weighted feature fusion: For features with similarity higher than the threshold τ, these features are weighted according to the similarity between the features, and features with high similarity are given greater weights; 8. The adversarial defense method based on multi-view image super-resolution according to claim 1, It is characterized in that In the step 4, F fusion It is further decoded by the decoding function into the purified RGB value p i , all M pixel positions are polled by the decoding function to obtain the clean RGB values The specific method of forming the defended image O obtained after the network purifies the attacked image A is as follows: All images share a decoding function S=s with θ as parameter θ (f, p), this function maps coordinates to RGB values, where f is a vector, p is a two-dimensional coordinate in the continuous image domain, and s is the predicted signal RGB value; for a continuous image I i and its implicit representation f i , define the RGB value at coordinate q as: I i (q) = s θ (z * ,qv * ), where z * Yes i The eigenvector with the closest Euclidean distance to q, v * is feature z * Coordinates in the image domain; the above formula is expanded to: Where z t * (t∈(00, 01, 10, 11}) is the closest feature in the upper left, upper right, lower left, and lower right subspaces, v t * Yes t * The coordinates of R t It is q and v t′ * The rectangular area between t′ and t′ is the diagonal position of t; the feature map f i The filling is mirrored outside the boundary, so the above formula also applies near the boundary; the decoding function then uses the area as the confidence to vote and merge the four signals, and the normalized confidence is proportional to the area of ​​the rectangle between the query point and its nearest potential feature diagonal corresponding point.

Images