An adversarial defense method based on multi-view image super-resolution

Abstract

The application discloses an adversarial defense method based on multi-view image super-resolution, comprising: obtaining an attacked image, forming a clean image and an attacked image pair with a clean image; inputting the attacked image and an auxiliary view image into a network for encoding to obtain implicit representation; obtaining a homography matrix from the auxiliary view image to the attacked image, performing spatial transformation on the auxiliary view image at the feature level to realize spatial alignment; through a decoding function, fusing the implicit representation in the feature space, and through polling each pixel position, obtaining a clean pixel value to obtain a defensive output after the network purifies the attacked image. The application does not depend on the limitation of a single view image, reduces the dependence of observation angle consistency, effectively realizes the purification of the adversarial disturbance, not only improves the visual quality of the image, but also enhances the defense ability of the network to potential adversarial attacks, and can be flexibly applied to image processing robustness tasks in complex real environments.

Description

Technical Field

[0001] This invention belongs to the fields of artificial intelligence and computer vision, and relates to adversarial attack and defense methods, specifically an adversarial defense method based on multi-view image super-resolution. Background Technology

[0002] Adversarial defense is a technique designed to protect deep learning models from adversarial attacks. Adversarial attacks deceive deep learning models by subtly but intentionally modifying the input data, causing them to make incorrect judgments; this is particularly common in image classification. Therefore, the goal of adversarial defense is to enhance the robustness of the model, enabling it to resist or at least identify tampered input data. Existing defense methods can be divided into model-modification-based defenses and input-modification-based defenses. Input-modification-based defenses, without altering the model structure, eliminate or reduce adversarial perturbations in adversarial examples by preprocessing or transforming the input image. Current state-of-the-art methods utilize data augmentation, feature compression, and other techniques to suppress adversarial noise in images. However, it is important to note that with the continuous advancement of adversarial attack techniques, developing more effective defense strategies remains challenging. For data augmentation-based defense methods, when facing various subtle attacks such as noise injection, pattern distortion, or semantic manipulation, current techniques do not introduce external information, making it difficult to guarantee complete effectiveness or demonstrate good generalization ability. Further research indicates that deep learning models are extremely sensitive to different viewpoints of objects. Categories that can be correctly identified from certain familiar viewpoints are easily misclassified from most other viewpoints. This vulnerability of deep learning models in handling viewpoint diversity makes them susceptible to adversarial attacks. Furthermore, single-view images may contain interference factors such as pose and occlusion, easily resulting in the loss of visual information after an attack. In contrast, multi-view images, capturing the same scene from different perspectives, can compensate for the shortcomings of single-view images when attacked through their differences and diversity, giving them a natural advantage in adversarial defense tasks. Summary of the Invention

[0003] To address the aforementioned limitations of existing technologies, this invention provides an adversarial defense method based on multi-view image super-resolution. By aligning and fusing multi-view information and using local implicit functions to reconstruct the image, image details can be effectively restored. This invention does not rely on the limitations of a single-view image, reduces the dependence on the consistency of observation angles, and effectively eliminates adversarial perturbations. It not only improves the visual quality of the image but also enhances the defense capability against potential adversarial attacks, and can be flexibly applied to robust image processing tasks in complex real-world environments.

[0004] To address the aforementioned technical problems, this invention proposes an adversarial defense method based on multi-view image super-resolution, comprising the following steps:

[0005] Step 1: The original, unmodified clean image C obtained from the multi-view image dataset is attacked using the PGD (Projected Gradient Descent) attack algorithm to generate the attacked image A. The clean image C and the attacked image A are then combined to form a clean image-attacked image pair (A, C). This process is repeated until all N clean images in the dataset have been processed, forming an image pair set D.

[0006] Step 2: Extract an auxiliary viewpoint image V from the clean image C in the multi-view image dataset. Extract the attacked image A from the clean image pair set D. Input the attacked image and the auxiliary viewpoint image V together into the EDSR (Enhanced Deep Super-Resolution) encoder (this network structure is a mature, industry-standard, and interpretation-free architecture) to obtain the implicit representations (f) of the attacked image A and the auxiliary viewpoint image V. A f V ), and use features to expand the content of the rich representation, to obtain

[0007] Step 3: The homography pre-alignment module of this method's network extracts deep features for alignment and learns a content-aware mask for rejecting anomalous regions. After weighting the two, the homography estimator in the module estimates the homography, obtaining a homography matrix H from the auxiliary viewpoint image V to the attacked image A. The implicit representation of the auxiliary viewpoint image V is then used in H. Spatial transformation is performed at the feature level to achieve viewpoint alignment, resulting in an aligned implicit representation of the auxiliary viewpoint.

[0008] Step four: In the feature space, using a decoding function with a ReLU (Rectified Linear Unit) activation function and a 5-layer MLP (Multiple Layer Perception) structure with 256 hidden layers, the implicit representation obtained in step two is used for each pixel location of the attacked image A. The features at that location are obtained, and implicit representations are also obtained from the aligned auxiliary viewpoint. The system also finds the features corresponding to that position, calculates the cosine similarity between the two features to assess their similarity, and if the similarity is greater than or equal to a preset threshold, then... and Weighted fusion is performed; otherwise, these two features will not be considered, resulting in the fused feature F. fusion F fusion It is further decoded by the decoding function into the purified RGB value p i The clean RGB values ​​obtained by polling all M pixel positions using the decoding function. The defended image O is obtained after the network is cleaned up and the attacked image A is generated.

[0009] Furthermore, the adversarial defense method based on multi-view image super-resolution described in this invention, wherein:

[0010] The first step, obtaining the set D of clean image-attacked image pairs, is as follows:

[0011] (1-1) Select classifier CN.

[0012] (1-2) For each clean, unmodified original image C obtained from the multi-view image dataset C i The classifier CN is attacked using the PGD attack method to generate the attacked image A. i .

[0013] (1-3) The resulting clean image C i and the attacked image A i Composition of clean image - attacked image pair D i =(A i C i ).

[0014] (1-4) Repeat (1-2) and (1-3) until there are N clean images C in the dataset. i All images are processed to form a set D of image pairs.

[0015] In step one, the specific method for generating the attacked image is as follows: using PGD attack as the attack method, the maximum amplitude ε of the infinite norm perturbation. ∞ The step size is 2 / 255, the number of steps is 100, and the ResNet18 and WideResNet28 classifiers trained on the original dataset are attacked respectively.

[0016] In step two, the attacked image A and the auxiliary viewpoint image V are input into the EDSR encoder for encoding to obtain the implicit representation (f A f vThe specific method is as follows: The EDSR encoder is a sequence containing 15 residual blocks. Each residual block consists of a convolutional layer, a ReLU layer, and a second convolutional layer, extracting pixel-wise features from the input image. The encoder represents the input image as a two-dimensional feature map with 64 channels C, length H, and width W equal to the dimensions of the input image. The feature at position [i, j] is denoted as f. ij The H×W feature vectors obtained by the encoder are uniformly distributed in a two-dimensional space within the continuous image domain. The two received inputs are mapped to the same low-dimensional latent space by the same encoder, becoming implicit representations (f...). A f V ).

[0017] In step two, the specific method for enriching the content of the implicit representation using feature expansion is as follows: for f∈(f A f V ) obtained by applying feature expansion The eigenvector is a concatenation of the eigenvectors from the 3×3 neighborhood of f. Formally, the feature expansion at positions j and k is defined as: The Concat operation refers to concatenating a set of vectors and filling the boundaries of the original feature vectors outward with zero vectors.

[0018] In step three, the specific method for obtaining a homography matrix H from the auxiliary viewpoint image V to the attacked image A using the homography pre-alignment module of this method network is as follows: The homography pre-alignment module receives two input images, the auxiliary viewpoint image V and the attacked image A. After converting them to grayscale images, it generates the homography matrix H from V to A as the output. The entire module can be divided into three sub-modules: feature extractor w(·), mask predictor m(·), and homography estimator h(·). This module is a plug-in of the overall architecture and is trained independently using a ternary loss, constraining the image A′ after homography matrix deformation to be farther from A and closer to V. w(·) and m(·) are fully convolutional networks that accept inputs of arbitrary size. The specific steps are as follows:

[0019] (3-1) w(·) learns the deep features used for alignment, with a size of H×W×C, where H is the length, W is the width, and C is the number of channels, denoted as W x = w(x), x∈{A,V}.

[0020] (3-2) m(·) learns a content-aware mask to reject anomalous regions. Specifically, the submodule m(·) learns to generate a probabilistic map or mask that highlights content in the feature map that contributes significantly to homography estimation, while outliers are detected and discarded, and not used for homography estimation. The size of the mask is the same as the size of the feature map extracted by w(·), denoted as M. x= m(x), x∈{A,V}.

[0021] (3-3) Using a mask, the features are further weighted and then fed to the homography estimator to obtain two weighted feature maps G. a and G v :G x =W x M x , x∈{A,V}.

[0022] (3-4) The h(·) structure is the same as w(·), generating four 2-dimensional offset vectors, with a total of eight values ​​serving as elements of the homography matrix. Given a weighted eigenmap G... a and G v Connect them to construct a feature map [G] a G v A homography matrix of size H×W×2C is input into a homography estimator. After obtaining the offset vector, a linear system is solved to obtain a homography matrix H with 8 degrees of freedom. av H av =h([G a G v ]).

[0023] (3-5) After estimating the homography matrix H av Based on this, the features of the auxiliary viewpoint image V Perform homography transformation to adjust the spatial layout of features while preserving them, achieving spatial alignment. The resulting aligned features are:

[0024] In step four, the decoding function fuses implicit representations from different perspectives to obtain the fused feature F. fusion The specific method is as follows: the decoding function is a 5-layer MLP with ReLU activation and 256 hidden layers. Given pixel location coordinates q, the decoding function first finds the pixel location coordinates v that are closest to q in the input image. * =[i * j * ] and corresponding features And the same position coordinates v′=v were also found in the corresponding auxiliary viewpoint image. * Features f i′j′ Then, the decoding function considers v * Features from the v′ neighborhood are used to predict clean RGB values ​​p. i＝q The specific steps are as follows:

[0025] (4-1) Feature similarity calculation: For neighborhood feature maps, calculate the cosine similarity between feature vectors at the same spatial location.

[0026] (4-2) Similarity threshold judgment: For the preset similarity threshold τ, when S(i,j)<τ, it is considered that the features of the two perspectives at this position have no reference significance and the features of the auxiliary perspective are not adopted; when S(i,j)≥τ, it is considered that the features of the two perspectives can be fused.

[0027] (4-3) Feature weighted fusion: For features with similarity higher than the threshold τ, these features are weighted according to the similarity between features, and features with high similarity are given greater weight.

[0028]

[0029] In step four, F fusion It is further decoded by the decoding function into the purified RGB value p i The clean RGB values ​​obtained by polling all M pixel positions using the decoding function. The specific method for generating the defended image O after cleaning the attacked image A is as follows: all images share a decoding function S = s with θ as the parameter. θ (f, p), this function maps coordinates to RGB values, where f is a vector, p is the two-dimensional coordinate in the continuous image domain, and s is the RGB value of the predicted signal. For a continuous image I i and its implicit representation f i Define the RGB value at coordinate q as: I i (q)=s θ (z * ,qv * ), where z * It is f i The eigenvector v that is closest to q in Euclidean distance * It is feature z * The coordinates in the image domain. The above formula, after considering the weights, can be extended to: In the formula z t * (t∈{00, 01, 10, 11}) is the closest feature in the top-left, top-right, bottom-left, and bottom-right subspaces, v t * It is z t * The coordinates, R t It's q and v t′ * The area of ​​the rectangle between t and t' is the diagonal position of t. Feature mapping f iOutside the boundary, the fill is mirrored, so the formula above also applies near the boundary. The decoding function then uses the area as the confidence level, voting to merge the four signals. The normalized confidence level is proportional to the area of ​​the rectangle between the query point and its nearest corresponding point on the diagonal of the latent feature.

[0030] Compared with the prior art, the beneficial effects of the present invention are:

[0031] (1) In the process of completing the adversarial defense, the present invention utilizes the information complementarity of multiple perspectives and the richness of details and textures between subtle differences in perspectives, which can effectively reconstruct image details and help the super-resolution algorithm better understand the structure of the object. At the same time, since the implicit image representation describes the entire image in a parameterized way, its natural ability to efficiently compress image information enables it to achieve better super-resolution effects with limited network complexity, thereby achieving better defense effects.

[0032] (2) This invention separates the training of the defense from the training of the classifier. When the actual settings (such as architecture, hyperparameters, or number of classes) change, there is no need to retrain the classifier, thus reducing the complexity of the defense. In addition, since this method is based on a local statistical model, it has high parameter efficiency and computational efficiency. Therefore, this invention can basically ensure low processing time and quickly adapt to near real-time application requirements, such as dynamic image adversarial defense and real-time security protection. Attached Figure Description

[0033] Figure 1 This is a flowchart of an adversarial defense method based on multi-view image super-resolution according to the present invention. Detailed Implementation

[0034] This invention proposes an adversarial defense method based on multi-view image super-resolution. The design concept utilizes the rich complementary information of multi-view images and super-resolution technology to mitigate image perturbations caused by adversarial attacks, achieving the goal of adversarial defense. Multi-view images, containing scene and object information captured from different angles, provide the model with additional context and depth perception, helping it to better recover the original, unattacked image details when faced with adversarial perturbations that impair image quality. Super-resolution technology compensates for information loss caused by attacks by reconstructing high-resolution images, providing an effective means to restore image details and clarity. Therefore, by utilizing super-resolution technology, this method aims not only to mitigate or eliminate pixel-level perturbations introduced by adversarial attacks but also to improve image resolution and quality. By combining multi-view images and super-resolution technology, this invention aims to effectively mitigate adversarial perturbations, achieve adversarial defense, significantly improve the robustness of networks against adversarial attacks, and ensure the security and reliability of image data.

[0035] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. The specific embodiments described are only for explanation and illustration of the present invention and are not intended to limit the present invention.

[0036] I. Obtain a set of clean image-attacked image pairs

[0037] Before network training and adversarial defense inference, obtain the attacked image. (Adversarialimage), with clean images (Clean image) Creates a clean image, compared to an attacked image. The specific steps are as follows:

[0038] (1) Select classifier CN.

[0039] (2) For each clean, unmodified original image C obtained from the multi-view image dataset C. i The classifier CN is attacked using the PGD attack method to generate the attacked image A. i .

[0040] (3) The resulting clean image C i and the attacked image A i Composition of clean image - attacked image pair D i =(A i C i ).

[0041] (4) Repeat (2) and (3) until N clean images C are found in the dataset. i All images are processed to form a set D of image pairs.

[0042] The specific method for creating the attacked image is as follows: using PGD attack as the attack method, the maximum amplitude ε of the infinite norm perturbation. ∞ The step size is 2 / 255, the number of steps is 100, and the ResNet18 and WideResNet28 classifiers trained on the original dataset are attacked respectively.

[0043] II. Encoding the Input

[0044] The attacked image A and the auxiliary viewpoint image V are input into the EDSR encoder for encoding to obtain the implicit representation (f A f V ).

[0045] Encoder Structure. The encoder extracts pixel-wise features from the input image using an EDSR design, which is a sequence of 15 residual blocks. Each residual block consists of a convolutional layer, a ReLU layer, and a second convolutional layer. The encoder represents the continuous image as a two-dimensional feature map with 64 channels (C), length H, and width W. The feature at position [i, j] is denoted as f. ij The H×W feature vectors obtained by the encoder are uniformly distributed in a two-dimensional space within the continuous image domain. The two received inputs are mapped to the same low-dimensional latent space by the same encoder, becoming implicit representations (f...). A f V ).

[0046] Feature expansion strategy. To enrich the information contained in each latent feature in f, this method expands f ∈ (f... A f V ) Application feature expansion to obtain The eigenvectors in f are concatenated eigenvectors from the 3×3 neighborhood of f. Formally, feature expansion is defined as:

[0047]

[0048] The Concat operation refers to the concatenation of a set of vectors, with f filling the boundaries outwards with zero vectors. This strategy applies to all computations, and for simplicity, the implicit representation will be represented by the symbol f in the following sections.

[0049] III. Homography Prealigned Multi-View Images

[0050] The homography pre-alignment module of this method network is used to obtain a homography matrix H from the auxiliary viewpoint image V to the attacked image A. Then, H is used to perform spatial transformation on the auxiliary viewpoint image V at the feature level, adjusting its spatial layout while preserving features to achieve viewpoint alignment and obtain the aligned implicit representation of the auxiliary viewpoint.

[0051] Homography pre-alignment strategy. Obtaining the homography matrix can be viewed as a mapping problem from input image pairs to homography transformation parameters: the homography pre-alignment module receives two input images, an auxiliary viewpoint image V and the image to be attacked A. After converting to grayscale, it generates a homography matrix H from V to A as the output. The entire module can be divided into two sub-modules: a feature extractor w(·), a mask predictor m(·), and a homography estimator h(·). This module is a plug-in to the overall architecture, trained independently using a ternary loss, constraining the image A′ after homography matrix deformation to be farther from A and closer to V. w(·) and m(·) are fully convolutional networks that accept inputs of arbitrary size. w(·) learns deep features for alignment, with a size of H×W×C, where H is the length, W is the width, and C is the number of channels.

[0052] W x = w(x), x∈{A,V}

[0053] The `m(·)` module learns a content-aware mask to reject anomalous regions. Specifically, the `m(·)` submodule learns to generate a probabilistic map or mask that highlights content in the feature map that contributes significantly to homography estimation, while outliers are detected and discarded and not used for homography estimation. The size of the mask is the same as the size of the feature map extracted by `w(·)`.

[0054] M x = m(x), x∈{A,V}

[0055] Using a mask, the features are further weighted and then fed to the homography estimator to obtain two weighted feature maps G. a and G b :

[0056] G x =W x M x x∈{A,V}

[0057] The h(·) structure is the same as w(·), producing four 2-dimensional offset vectors, with a total of eight values ​​serving as elements of the homography matrix. Given a weighted eigenmap G... a and G v Connect them to construct a feature map [G] a G v A homography matrix of size H×W×2C is input into a homography estimator. After obtaining the offset vector, a linear system is solved to obtain a homography matrix H with 8 degrees of freedom. av :

[0058] H av =h([G a G v ]).

[0059] After estimating the homography matrix H av Based on this, the features of the auxiliary viewpoint image V Perform homography transformation to adjust the spatial layout of features while preserving them, achieving spatial alignment. The resulting aligned features are:

[0060] IV. Decoding the features

[0061] In the feature space, using a decoding function with a ReLU (Rectified Linear Unit) activation function and a 5-layer MLP (Multiple Layer Perception) structure with 256 hidden layer dimensions, the implicit representation obtained in step two is used for each pixel location of the attacked image A. The features at that location are obtained, and implicit representations are also obtained from the aligned auxiliary viewpoint. The system also finds the features corresponding to that position, calculates the cosine similarity between the two features to assess their similarity, and if the similarity is greater than or equal to a preset threshold, then... and Weighted fusion is performed; otherwise, these two features will not be considered, resulting in the fused feature F. fusio n. F fusion It is further decoded by the decoding function into the purified RGB value p i The clean RGB values ​​obtained by polling all M pixel positions using the decoding function. The defended image O is obtained after the network is cleaned up and the attacked image A is generated.

[0062] Decoder structure. The decoding function is a 5-layer MLP with ReLU activation and 256 hidden layers. All images share a single decoding function S = s with parameter θ. θ (f, p), this function maps coordinates to RGB values, where f is a vector, p is the two-dimensional coordinate in the continuous image domain, and s is the RGB value of the predicted signal. For a continuous image I i and its implicit representation f i Define the RGB value at coordinate q as:

[0063] I i (q)=s θ (z * ,qv * )

[0064] Where z * It is f i The eigenvector v that is closest to q in Euclidean distance * It is feature z *Coordinates in the image domain. Because the decoder models locally, qv * A relative coordinate is formed, and then input into the decoder along with the feature z* at v* to query the pixel value. At this point, the decoder's prediction of the pixel value is still discontinuous. Specifically, since the signal prediction at q is done by querying the nearest latent code z in f... * This is accomplished by the fact that when q moves in the 2D domain, multiple v values ​​that satisfy the conditions can appear simultaneously in the image coordinate domain. * Therefore z * The choice can abruptly switch from one to another, exhibiting abruptness. Therefore, the above formula expands to:

[0065]

[0066] In the formula z t * (t∈{00, 01, 10, 11}) is the closest feature in the top-left, top-right, bottom-left, and bottom-right subspaces, v t * It is z t * The coordinates, R t It's q and v t′ * The area of ​​the rectangle between t and t' is the diagonal position of t. Feature mapping f i Outside the boundary, the grid is mirror-filled, so the formula above also applies near the boundary. This formula refines the resolution grid, overlapping local blocks with their neighboring blocks, resulting in four independent predicted signal features at each coordinate. The decoding function then uses the area as confidence, voting to merge these four signals. The normalized confidence is proportional to the area of ​​the rectangle between the query point and its nearest latent feature diagonal point; therefore, the confidence is higher when the query coordinates are closer, making the decoding function more efficient at the z-axis. * Continuity is achieved at the coordinates of the transformation. Because this module provides a continuous representation of the image, and the decoder decodes only based on adjacent pixels, the original grid size is not important, making the decoding result independent of the size or resolution of the original image.

[0067] Multi-view local fusion strategy. Given pixel location coordinates q, the decoding function first finds the pixel location coordinates v that are closest to q in the input image. * =[i * j * ] and corresponding features And the same position coordinates v′=v were also found in the corresponding auxiliary viewpoint image. * Features f i′j′ Then, the decoding function considers v * Features from the v′ neighborhood are used to predict clean RGB values ​​p.i＝q The specific steps are as follows:

[0068] (1) Feature similarity calculation: For neighborhood feature maps, calculate the cosine similarity between feature vectors at the same spatial location.

[0069] (2) Similarity threshold judgment: For the preset similarity threshold τ, when S(i,j)<τ, it is considered that the features of the two perspectives at this position have no reference significance and the features of the auxiliary perspective are not adopted; when S(i,j)≥τ, it is considered that the features of the two perspectives can be fused.

[0070] (3) Feature weighted fusion: For features with similarity higher than the threshold τ, these features are weighted according to the similarity between features, and features with high similarity are given greater weight.

[0071]

[0072] Predict the cleaned defensive image. Ideally, the predicted clean RGB values ​​should match the clean image at that location. The pixel values ​​are identical, meaning that noise added by adversarial attacks can be completely removed. In practice, the predicted clean RGB value should be close to the pixel value of the clean image at that location, and the closer it is to the clean image, the better. Using SSIM (Structural Similarity Index Measure) and MSE (Mean Squared Error) similarity metrics, the quality of the defended image is measured by comparing the differences in pixel values, structure, and brightness between the clean image and the defended image.

[0073] The network structure is highly efficient. The defense process of this invention is not computationally intensive because the encoder feature map is calculated only once and used to predict the RGB values ​​of all query pixel locations. Furthermore, the classifier and attack method used to generate training image pairs can be different from the classifier and attack method used during inference, thus making the defense more flexible and robust.

[0074] During the training phase, the L1 loss between the clean image C and the defensive output O is calculated and minimized based on their RGB values; during the inference phase, an image is input and the output size is specified to obtain the defended image O.

[0075] Network training. The network is trained using random patches cropped at the same locations in images A, V, and C. For example, random patches of size 48×48 are sampled from the training pairs in the dataset as the network's real input. The network is trained to minimize the L1 loss between the clean image C and the RGB values ​​of the defensive output O.

[0076] Network inference. During the inference phase, the network outputs the predicted post-defense image. The post-defense image O can be synthesized by predicting the RGB values ​​of each pixel position in a grid of size H0×W0×3. The size of the post-defense image O is generally the same as the size of the input image. This method does not require the size of O to be the same as the size of the input image; the size can be arbitrarily specified for output, achieving the effect of simultaneous super-resolution.

[0077] Example

[0078] The feasibility of the method of the present invention is verified below with specific examples, as detailed in the following description:

[0079] In this embodiment, training was performed for 40 epochs each on Cifar10 and Cifar100, with a learning rate set to 0.0001, using the Adam optimizer. To create training pairs, PGD attack was used as the attack method (ε). ∞ The attack and defense methods of this invention were tested by attacking the ResNet18 classifier on the original dataset with a step size of 8 / 255, a step size of 2 / 255, and a step count of 100. The attack and defense evaluations used the Autoattack and RobustBench baselines. All experiments were performed on a single Nvidia 3060ti GPU using PyTorch, and each result was averaged across three experiments.

[0080] In this invention, defensive capabilities are defined as follows:

[0081] (1) Standard Accuracy (SA): refers to the classification accuracy of the model on a clean dataset (data that has not been subjected to adversarial attacks). It measures the model's ability to correctly classify input data under normal conditions.

[0082] (2) Robust Accuracy (RA): refers to the classification accuracy of the model when facing adversarial samples. It measures the model’s performance in resisting these intentionally created perturbations and reflects the robustness of the model.

[0083] Table 1 Comparison of experimental results between the method of this invention and the existing best method

[0084] method SA RA average Dataset No defense 92.25 0 46.13 Cifar10 Method (1) 85.46 60.33 72.90 Cifar10 Method of the present invention 86.12 71.58 78.85 Cifar10 No defense 91.38 0 45.69 Cifar100 Method (1) 86.61 78.8 85.09 Cifar100 Method of the present invention 87.27 83.74 85.5 Cifar100

[0085] Based on the results shown in Table 1 of the present method and the best existing adversarial defense method based on modified input (the method of directly using generated data to enhance the model's adversarial defense (1)), it is found that without defense, the attack deceives the classifier in almost all examples, while the image examples after defense can be correctly reclassified by the classifier to a certain extent. The image examples after defense by the method of the present invention can be correctly reclassified by the classifier with a high probability. The defense effect of the method of the present invention is better than the best existing adversarial defense method based on modified input. Therefore, the feasibility and superiority of the present invention can be demonstrated.

[0086] Although the present invention has been described above in conjunction with the accompanying drawings, the present invention is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many improvements and changes under the guidance of the present invention without departing from the spirit of the present invention, and these improvements and changes are all within the protection scope of the present invention.

Claims

1. An adversarial defense method based on multi-view image super-resolution, characterized in that, The method includes the following steps: Step 1: The original, unmodified clean image C obtained from the multi-view image dataset is attacked using the PGD attack algorithm to generate the attacked image A. The clean image C and the attacked image A are then combined to form a clean image-attacked image pair (A, C). This process is repeated until all N clean images in the dataset have been processed, forming an image pair set D. Step 2: Extract an auxiliary viewpoint image V from the clean image C in the multi-view image dataset. Extract the attacked image A from the clean image pair set D. Input the attacked image A and the auxiliary viewpoint image V together into the EDSR encoder for encoding to obtain the implicit representations (f) of the attacked image A and the auxiliary viewpoint image V. A f V ), and use features to expand the content of the rich representation, to obtain Step 3: The homography pre-alignment module of this method's network extracts deep features for alignment and learns a content-aware mask for rejecting anomalous regions. After weighting the two, the homography estimator in the module estimates the homography, obtaining a homography matrix H from the auxiliary viewpoint image V to the attacked image A. The implicit representation of the auxiliary viewpoint image V is then used in H. Spatial transformation is performed at the feature level to achieve viewpoint alignment, resulting in an aligned implicit representation of the auxiliary viewpoint. Step four: In the feature space, using a decoding function with a 5-layer MLP structure having a ReLU activation function and 256 hidden layer dimensions, for each pixel location of the attacked image A, the implicit representation obtained in step two is used. The features at that location are obtained, and implicit representations are also obtained from the aligned auxiliary viewpoint. The system also finds the features corresponding to that position, calculates the cosine similarity between the two features to assess their similarity, and if the similarity is greater than or equal to a preset threshold, then... and Weighted fusion is performed; otherwise, these two features will not be considered, resulting in the fused feature F. fusion ;F fusion It is further decoded by the decoding function into the purified RGB value p i The clean RGB values ​​obtained by polling all M pixel positions using the decoding function. The defended image O is obtained after the network is cleaned up and the attacked image A is generated.

2. The adversarial defense method based on multi-view image super-resolution according to claim 1, characterized in that, In step one, the specific method for obtaining the clean image-attacked image pair set D is as follows: 1-1) Select classifier CN; 1-2) For each clean, unmodified original image C obtained from the multi-view image dataset C i The classifier CN is attacked using the PGD attack method to generate the attacked image A. i ; 1-3) The resulting clean image C i and the attacked image A i Composition of clean image - attacked image pair D i =(A i C i ); 1-4) Repeat steps 1-2) and 1-3) until N clean images C are found in the dataset. i All images are processed to form a set D of image pairs.

3. The adversarial defense method based on multi-view image super-resolution according to claim 1, characterized in that, In step one, the specific method for generating the attacked image A is as follows: using PGD attack as the attack method, the maximum amplitude ε of the infinite norm perturbation. ∞ The step size is 2 / 255, the number of steps is 100, and the ResNet18 and WideResNet28 classifiers trained on the original dataset are attacked respectively.

4. The adversarial defense method based on multi-view image super-resolution according to claim 1, characterized in that, In step two, the attacked image A and the auxiliary viewpoint image V are input into the EDSR encoder for encoding to obtain the implicit representation (f A f V The specific method is as follows: The EDSR encoder is a sequence containing 15 residual blocks. Each residual block consists of a convolutional layer, a ReLU layer, and a second convolutional layer, which extracts pixel-wise features from the input image. The encoder represents the input image as a two-dimensional feature map with 64 channels C, length H, and width W equal to the dimensions of the input image. The feature at position [i, j] is denoted as f. ij The H×W feature vectors obtained by the encoder are uniformly distributed in a two-dimensional space in the continuous image domain; the two received inputs are mapped to the same low-dimensional latent space by the same encoder, becoming implicit representations (f A f V ).

5. The adversarial defense method based on multi-view image super-resolution according to claim 1, characterized in that, In step two, when obtaining the implicit representations of the attacked image A and the auxiliary viewpoint image V, the specific method for enriching the representation content using feature expansion is as follows: for f∈(f A f V ) obtained by applying feature expansion The eigenvector is a concatenation of the eigenvectors of the 3×3 neighborhood in f; formally, the feature expansion at positions j and k is defined as: The Concat operation refers to concatenating a set of vectors and filling the boundaries of the original feature vectors outward with zero vectors.

6. The adversarial defense method based on multi-view image super-resolution according to claim 1, characterized in that, In step three, the specific method for obtaining a homography matrix H from the auxiliary viewpoint image V to the attacked image A using the homography pre-alignment module of this method network is as follows: The homography pre-alignment module receives two input images, the auxiliary viewpoint image V and the attacked image A. After converting them to grayscale images, it generates a homography matrix H from V to A as the output. The entire module can be divided into three sub-modules: feature extractor w(·), mask predictor m(·), and homography estimator h(·). This module is a plug-in of the overall architecture and is trained independently using a ternary loss, constraining the image A′ after homography matrix deformation to be farther from A and closer to V. w(·) and m(·) are fully convolutional networks that accept inputs of arbitrary size. The specific steps are as follows: 3-1) w(·) learns the deep features used for alignment, with a size of H×W×C, where H is the length, W is the width, and C is the number of channels, denoted as W x = w(x), x∈{A,V}; 3-2) m(·) learns a content-aware mask to reject anomalous regions; specifically, the submodule m(·) learns to generate a probability map or mask that highlights content in the feature map that contributes significantly to homography estimation, while outliers are detected and discarded and not used for homography estimation; the size of the mask is the same as the size of the feature map extracted by w(·), denoted as M. x = m(x), x∈(A,V}; 3-3) Using a mask, the features are further weighted and then fed to the homography estimator to obtain two weighted feature maps G. a and G v :G x =W x M x x∈{A,V}; 3-4) The h(·) structure is the same as w(·), generating four 2-dimensional offset vectors, with a total of eight values ​​serving as elements of the homography matrix; given the weighted eigenmap G a and G v Connect them to construct a feature map [G] a G v A homography matrix of size H×W×2C is input into a homography estimator. After obtaining the offset vector, a linear system is solved to obtain a homography matrix H with 8 degrees of freedom. av H av =h([G a G v ]); 3-5) After estimating the homography matrix H av Based on this, the features of the auxiliary viewpoint image V Perform homography transformation to adjust the spatial layout of features while preserving them, achieving spatial alignment. The resulting aligned features are:

7. The adversarial defense method based on multi-view image super-resolution according to claim 1, characterized in that, In step four, the decoding function fuses implicit representations from different perspectives to obtain the fused feature F. fusion The specific method is as follows: the decoding function is a 5-layer MLP with ReLU activation and 256 hidden layers; given pixel location coordinates q, the decoding function first finds the pixel location coordinates v that are closest to q in the input image. * =[i * j * ] and corresponding features And the same position coordinates v′=v were also found in the corresponding auxiliary viewpoint image. * Features f i′j′ Then, the decoding function considers v * Features from the v′ neighborhood are used to predict clean RGB values ​​p. i＝q The specific steps are as follows: 4-1) Feature Similarity Calculation: For neighborhood feature maps, calculate the cosine similarity between feature vectors at the same spatial location. 4-2) Similarity threshold judgment: For the preset similarity threshold τ, when S(i,j)<τ, it is considered that the features of the two perspectives at this position have no reference significance and the features of the auxiliary perspective are not adopted; when S(i,j)≥τ, it is considered that the features of the two perspectives can be fused. 4-3) Feature weighted fusion: For features with similarity higher than the threshold τ, these features are weighted according to the similarity between features, and features with high similarity are given greater weight; 8. The adversarial defense method based on multi-view image super-resolution according to claim 1, characterized in that, In step four, F fusion It is further decoded by the decoding function into the purified RGB value p i The clean RGB values ​​obtained by polling all M pixel positions using the decoding function. The specific method for generating the defended image O after cleaning the attacked image A is as follows: All images share a single decoding function S = s with parameter θ. θ (f, p), this function maps coordinates to RGB values, where f is a vector, p is the two-dimensional coordinate in the continuous image domain, and s is the RGB value of the predicted signal; for a continuous image I i and its implicit representation f i Define the RGB value at coordinate q as: I i (q)=s θ (z * ,qv * ), where z * It is f i The eigenvector v that is closest to q in Euclidean distance * It is feature z * The coordinates in the image domain; the above formula, after considering the weights, is extended to: In the formula z t * (t∈(00, 01, 10, 11}) is the closest feature in the top-left, top-right, bottom-left, and bottom-right subspaces, v t * It is z t * The coordinates, R t It's q and v t′ * The area of ​​the rectangle between t and t', where t′ is the diagonal position of t; feature map f i Outside the boundary, the area is mirror-filled, so the formula above also applies to the vicinity of the boundary; the decoding function then uses the area as the confidence level, votes to merge the four signals, and the normalized confidence level is proportional to the area of ​​the rectangle between the query point and the corresponding point on the diagonal of its nearest latent feature.

Images