Adaptive adversarial patch image inpainting and defense method based on intention perception

Abstract

This invention belongs to the field of computer vision and image processing technology, specifically an intention-aware adaptive adversarial patching image restoration and defense method. The method includes: acquiring an input image containing occlusions, performing feature extraction based on a two-stream perceptual network, determining intent attributes, and identifying the missing image based on the determination results; after generating the missing image, concatenating the missing image with the original image and feeding it into a residual translation-diffusion network, extracting the identity feature vectors of the unoccluded regions from the image, injecting these vectors into the residual translation-diffusion network through a cross-attention mechanism, predicting the residual map in Markov chain iteration, and determining the final reconstructed image based on the residual map prediction results; this invention reconstructs a high-fidelity, identity-consistent complete face image by predicting and superimposing residuals in an ultra-fast Markov chain iteration.

Description

Technical Field

[0001] This invention belongs to the field of computer vision and image processing technology, specifically an intention-aware adaptive adversarial patching image repair and defense method. Background Technology

[0002] Existing adversarial patch defense methods mainly include image preprocessing-based methods (such as JPEG compression and spatial smoothing), detection and repair-based methods (such as SAC and Jedi), and diffusion model-based purification methods (such as DiffPure and RePaint). These methods attempt to destroy adversarial perturbations by removing high-frequency noise through global filtering, interpolating and repairing after locating the patch region through entropy analysis, or adding and removing noise through diffusion models, which improves the robustness of the model to physical attacks to a certain extent.

[0003] However, existing technologies are mainly limited by the contradiction between the lack of intent perception capability and the efficiency and high quality of the repair process when achieving effective defense against adversarial patches and high-fidelity repair. This makes it difficult to balance adversarial robustness (defense success rate) and image fidelity (repair quality and identity consistency) in practical applications, thus affecting overall performance. Summary of the Invention

[0004] To address the aforementioned technical problems, this invention provides an intention-aware adaptive adversarial patching image repair and defense method, which resolves the contradiction between the lack of intention-aware capabilities and the high efficiency and quality of the repair process in existing technologies.

[0005] An intent-aware adaptive adversarial patching image inpainting and defense method includes the following steps: intent classification based on spatial-frequency domain dual-stream awareness, specifically:

[0006] The system acquires an input image containing occlusions, extracts features based on a dual-stream sensing network, concatenates the features extracted by the dual-stream sensing network, determines the intent attribute based on the concatenated features, and outputs a classification result of benign or malicious occlusion.

[0007] The mask generation strategy is dynamically switched based on the judgment result, and an image cutout operation is performed to determine the defective image, specifically as follows:

[0008] The mask generation strategy is dynamically switched based on the judgment result. If it is determined to be benign occlusion, the high-precision segmentation model is called to generate the mask; if it is determined to be malicious occlusion, the adversarial anomaly perception model is called to generate the mask. The image is then cut out based on the generated mask to determine the missing image.

[0009] And, residual diffusion repair based on cross-attention identity injection, specifically:

[0010] After generating the defective image, the defective image and the original image are concatenated by channels and fed into the residual translation-diffusion network. The identity feature vectors of the unoccluded areas are extracted from the defective image and injected into the residual translation-diffusion network through a cross-attention mechanism. The residual map is predicted in the Markov chain iteration, and the final reconstructed image is determined based on the prediction results of the residual map.

[0011] Preferably, the determination of intent attributes based on the concatenated features is as follows:

[0012] The spatial feature map output from the spatial stream and the high-frequency energy heat map output from the frequency domain stream are stitched together along the channel dimension to obtain a fused feature map.

[0013] The fused feature map is input into the intent classifier, which consists of a global average pooling layer, a fully connected layer, and a softmax activation function, and outputs the classification probability distribution of the occlusion attribute. It is used to determine intent attributes, specifically:

[0014] If the output of the intent classifier satisfies the formula If the occluded area in the fused feature map shows semantic coherence with the global facial structure and the high-frequency energy distribution is normal, it is judged as benign occlusion.

[0015] If the output of the intent classifier satisfies the formula If the occlusion region in the fused feature map has significant semantic breaks and abnormal high-frequency energy aggregation, it is determined to be malicious occlusion.

[0016] Preferably, the dynamic switching of the mask generation strategy based on the determination result is as follows:

[0017] When the output of the intent classifier satisfies the formula At that time, the high-precision segmentation model SAM-2 is called to generate a mask;

[0018] When the output of the intent classifier satisfies the formula At that time, a lightweight adversarial anomaly-aware two-node diffusion model is invoked to generate a mask.

[0019] Preferably, the step of calling the high-precision segmentation model SAM-2 to generate the mask is as follows:

[0020] The original image is input into the SAM-2 model, using the center point of the occluded region as a cue. The model outputs a high-precision instance segmentation mask through an encoder-decoder structure. The output instance segmentation mask can closely fit the geometric contours of natural occlusions, ensuring that benign occlusion areas are marked completely and accurately.

[0021] Preferably, the step of using a lightweight adversarial anomaly detection two-node diffusion model to generate the mask is as follows:

[0022] Construct a two-branch diffusion model where the two branches share the same network structure but use different noise intensity parameters;

[0023] First-level noise and second-level noise are injected into the input image respectively, and then denoising and reconstruction are performed separately to obtain two reconstructed images. The variance of the two reconstructed images at each pixel position is calculated.

[0024] Set a variance threshold, mark pixel regions with variance values ​​greater than the variance threshold in the variance map as abnormal regions, and remove isolated noise points through morphological closing operations to obtain the malicious patch mask.

[0025] Preferably, the step of performing the image cutout operation to determine the defective image is as follows:

[0026] Performing pixel-by-pixel operations on the original image and the mask, we get:

[0027] ;

[0028] in, Indicates pixel-by-pixel multiplication. This represents the output image with defects. This represents the original input image. The generated mask includes: instance segmentation mask and malicious patch mask.

[0029] Preferably, the step of injecting the identity feature vector of the unoccluded region into the residual translation-diffusion network through the cross-attention mechanism is as follows:

[0030] Let the feature map output by the current layer be... ,in, d represents the spatial size of the feature map, and d represents the number of channels.

[0031] Identity feature vector The feature map is mapped to an identity key matrix and an identity value matrix through a linear transformation layer, and simultaneously mapped to a query matrix through another linear transformation layer.

[0032] And by calculating cross-attention, we have:

[0033] ;

[0034] in, This represents the computation of cross attention. Represents the identity key matrix, Represents the identity value matrix. Represents the query matrix. Dimensions representing attention mechanisms;

[0035] The output attention feature map is added to the original feature map by residual addition, and then passed through layer normalization and feedforward network to obtain the feature map after injecting identity information.

[0036] Preferably, the prediction of the residual map in the Markov chain iteration is as follows:

[0037] Let the number of iterations in the Markov chain of the diffusion process be T, and define the target clean face image as... The defective image is The residual between the two is ;

[0038] Build a from Towards The transition distribution of a forward Markov chain with transitions is defined as follows:

[0039] ;

[0040] in, , For the first The residual transfer rate of the step satisfies the formula , It is a hyperparameter for controlling noise intensity. It is the identity matrix;

[0041] In the reverse process, a neural network is trained. Predict clean images And define the mean of the inverse transition distribution as:

[0042] ;

[0043] in, For the front The cumulative transfer rate of the step;

[0044] In each iteration In the middle, the residual translation and diffusion network is based on the current state and conditional input Predict the residual map; the prediction result is the same as that of the clean image. The estimate is .

[0045] Preferably, the step of determining the final reconstructed image based on the residual map prediction results is as follows:

[0046] from Starting with distribution Initial state of mid-sampling ;

[0047] for According to the reverse transition distribution Stepwise sampling;

[0048] go through After one iteration, the obtained For reconstructed facial images .

[0049] Compared with the prior art, the present invention has the following beneficial effects:

[0050] 1. This invention constructs a dual-stream feature perception mechanism that combines spatial and frequency-domain flows. Compared with existing methods that rely solely on spatial features for adversarial patch detection (such as methods based on convolutional features or visual Transformer single-stream structures), this invention achieves joint modeling of semantic structural information and high-frequency anomaly information, which significantly improves the accuracy of adversarial patch detection. In particular, the false negative rate is significantly reduced in complex backgrounds and low-contrast occlusion scenes, thus solving the problem of insufficient robustness in existing technologies.

[0051] 2. By introducing an intent classification mechanism based on fusion features, this invention achieves semantic-level differentiation between benevolent occlusion and malicious adversarial patches compared to existing image inpainting methods that do not distinguish between occlusion types and process them uniformly (such as diffusion model methods like DiffPure and RePaint). This significantly reduces the false processing rate and effectively avoids over-repairing of naturally occluded areas, thereby improving the accuracy and rationality of the overall inpainting strategy.

[0052] 3. This invention adaptively selects a differentiated mask generation strategy based on the intent determination result. Compared with the traditional technology that uses a single mask generation method (such as unified segmentation or unified detection), it achieves the optimal mask generation path for different occlusion types, which significantly improves the mask positioning accuracy and regional integrity, thereby solving the problems of poor mask generalization ability and insufficient adaptability in the existing technology.

[0053] 4. This invention generates benevolent occlusion masks by using a high-precision instance segmentation model. Compared with traditional mask generation methods based on threshold segmentation or coarse-grained detection, it achieves fine characterization of the boundaries of natural occlusions, significantly reducing boundary errors and thus significantly improving the edge continuity and visual realism of the subsequent image restoration process.

[0054] 5. This invention constructs an anomaly detection mechanism based on the difference in reconstruction of dual noise diffusion. Compared with adversarial patch detection methods based on single perturbation response or fixed feature detection, it realizes the detection of differences in sensitivity to different noise intensities by adversarial samples, which significantly enhances the ability to identify unknown attack types, thereby improving the system's generalization robustness and defense against new attacks.

[0055] 6. This invention constructs a residual translation diffusion network and introduces a cross-attention mechanism to inject identity features. Compared with traditional diffusion models (such as DiffPure) that rely solely on unconditional or weakly conditional generation, this invention achieves strong constraints on identity information during image reconstruction, which significantly improves the realism of details and identity consistency of the reconstructed image, thereby solving the problem of insufficient semantic consistency in existing diffusion models.

[0056] 7. This invention adopts an iterative residual prediction and progressive reconstruction mechanism based on Markov chains, which achieves stable reconstruction with fewer iterations compared to one-time generation or high-step diffusion reconstruction methods. This significantly improves inference efficiency (e.g., from hundreds of steps to several steps) while reducing artifact generation, thus balancing reconstruction quality and computational efficiency. Attached Figure Description

[0057] Figure 1 This is a schematic diagram of the overall method steps of the intention-aware adaptive adversarial patching image repair and defense method of the present invention. Detailed Implementation

[0058] The embodiments of the present invention will be described in further detail below with reference to the accompanying drawings and examples. The following examples are for illustrative purposes only and should not be construed as limiting the scope of the invention.

[0059] Example 1

[0060] Reference Figure 1 As an embodiment of the present invention, an intention-aware adaptive adversarial patching image repair and defense method is provided, comprising the following steps:

[0061] S1: Intent classification based on spatial-frequency domain dual-stream perception.

[0062] Specifically, the intent classification based on spatial-frequency domain dual-stream perception involves determining the intent attribute (good or bad) of the occluded region after acquiring the input image containing occlusions using a dual-stream perception network. The specific implementation is as follows:

[0063] For the input raw image, two feature extraction branches, spatial flow and frequency flow, are fed in parallel;

[0064] Spatial flow employs a Swing Transformer network structure, using a multi-layer, multi-head self-attention mechanism to perform global semantic modeling of the image, extracting semantic conflict information between local texture features of occluded areas and the overall facial topology. Specifically:

[0065] The input image is divided into several non-overlapping image blocks, which are then processed sequentially through a linear embedding layer, a SwinTransformer block sequence, and layer normalization to output a spatial feature map, where each feature point corresponds to the semantic representation of a local region in the original image.

[0066] The frequency domain stream first performs a fast Fourier transform (FFT) on the input image to transform the image from the spatial domain to the frequency domain, obtaining the frequency domain amplitude spectrum and phase spectrum. Then, it performs a high-pass filter on the amplitude spectrum to extract the high-frequency energy components, and maps the high-frequency energy distribution back to the spatial domain through an inverse Fourier transform to generate a high-frequency energy heat map, which is used to characterize the high-frequency abnormal clustering features unique to the adversarial patch in the occluded area.

[0067] The spatial feature map output from the spatial stream and the high-frequency energy heatmap output from the frequency domain stream are concatenated along the channel dimension to obtain a fused feature map. The fused feature map contains both global semantic information and high-frequency energy distribution information, which is used for subsequent intent classification.

[0068] Furthermore, the fused feature maps are input into the intent classifier, which consists of a global average pooling layer, a fully connected layer, and a softmax activation function, and outputs the classification probability distribution of the occlusion attributes. or This is used to determine whether the obstruction is intentional, specifically:

[0069] If the output of the intent classifier satisfies the formula If the occluded area in the fused feature map shows semantic coherence with the global facial structure and the high-frequency energy distribution is normal, it is judged as benign occlusion.

[0070] If the output of the intent classifier satisfies the formula If the occlusion region in the fused feature map has significant semantic breaks and abnormal high-frequency energy aggregation, it is determined to be malicious occlusion.

[0071] S2: Dynamic mask generation and image cutout based on intent classification results.

[0072] Dynamic mask generation and image cutout based on intent classification results involves dynamically switching the mask generation strategy according to the judgment result and performing image cutout operations to generate a defective image. The specific implementation is as follows:

[0073] Based on the output of the intent classifier, the mask generation strategy is dynamically switched as follows:

[0074] When the output of the intent classifier satisfies the formula At that time, the high-precision segmentation model SAM-2 (Segment Anything Model 2) is called to generate the mask, specifically:

[0075] The original image is input into the SAM-2 model, using the center point of the occluded region as a cue. The model outputs a high-precision instance segmentation mask through an encoder-decoder structure. The output instance segmentation mask can closely fit the geometric contours of natural occlusions (such as the edge of a mask or the frame of glasses), ensuring that benign occlusion areas are marked completely and accurately.

[0076] When the output of the intent classifier satisfies the formula At that time, the lightweight Adversarial Anomaly Perception (AAP) two-node diffusion model is invoked to generate a mask, specifically as follows:

[0077] Construct a two-branch diffusion model where the two branches share the same network structure but use different noise intensity parameters;

[0078] Inject first-level intensity noise into the input image (noise level) Noise of the second level intensity (noise level) Then, denoising and reconstruction are performed separately to obtain two reconstructed images;

[0079] Calculate the variance of the two reconstructed images at each pixel location and generate a variance map. ;

[0080] Since the anti-patch region is significantly more sensitive to different noise levels than the normal region, and its variance exhibits an abnormal peak, then:

[0081] Set variance threshold The variance values ​​in the variance plot are greater than The pixel regions are marked as abnormal regions, and isolated noise points are removed by morphological closing operations (dilation followed by erosion) to obtain the malicious patch mask. .

[0082] It should be noted that the dual-branch diffusion model adopts a lightweight U-Net structure, with fewer parameters than the standard diffusion model, resulting in faster inference speed and meeting the requirements for real-time detection.

[0083] The image cutout operation is performed based on the generated mask, specifically as follows:

[0084] Original image With mask ( or Performing pixel-by-pixel calculations, we have:

[0085] ;

[0086] in, Indicates pixel-by-pixel multiplication. This represents the output image with defects. This represents the original input image. The generated mask includes: instance segmentation mask and malicious patch mask.

[0087] Furthermore, the image cutout operation is performed based on the pixel positions corresponding to the mask values, specifically as follows:

[0088] For the pixel position with a value of 1 in the mask (i.e. the occluded area), set the pixel value of the corresponding position in the original image to 0;

[0089] For pixel positions in the mask that have a value of 0, keep the original pixel value unchanged.

[0090] It should be noted that in the damaged image, the occluded area has been completely hollowed out (pixel value of 0), while the unoccluded area retains the original information completely, including key biometric features such as facial contours, eyes, nose, and mouth, providing clean and accurate contextual boundaries for subsequent identity feature extraction and repair.

[0091] S2: Residual diffusion repair based on cross-attention identity injection.

[0092] Specifically, residual diffusion inpainting based on cross-attention identity injection involves concatenating the channels of the missing image with the original image after generating the missing image, and then feeding it into a Residual ShiftDiffusion Network (ResShiftNet). Identity feature vectors of the unoccluded regions are extracted from the missing image, and these vectors are injected into the ResShiftNet using a cross-attention mechanism. The residual map is predicted during Markov chain iterations, ultimately outputting a thoroughly cleaned and high-fidelity face image. The specific implementation is as follows:

[0093] After generating the damaged image, the damaged image and the original image are concatenated channel by channel. This allows the repair network to simultaneously reference the global context information of the original image and the information of the unoccluded areas preserved in the damaged image. Specifically:

[0094] For the acquired defective images and the corresponding original input image If we perform channel splicing, then we have:

[0095] First, we concatenate the data along the channel dimension to obtain the concatenated feature map, which gives us:

[0096] ;

[0097] in, For the image height and width, and These are the number of channels for the original image and the damaged image, respectively (usually 3, i.e., RGB three channels). This represents the spliced ​​feature map.

[0098] Using a pre-trained ArcFace face recognition model as an identity feature extractor, the identity feature vector of the unoccluded area is extracted, specifically as follows:

[0099] Original image Inputting the data into the ArcFace model, the model extracts a 512-dimensional facial feature vector using a deep convolutional neural network. ;

[0100] Furthermore, since the original image contains both unoccluded and occluded areas, the ArcFace model has already acquired robust representation capabilities of the overall structure of the face during training. Its output feature vector can effectively encode the biometric information of the unoccluded areas (such as the relative positions and textures of the eyes, nose, and mouth), while the contamination of the feature vector by the occluded areas is suppressed to a low level.

[0101] It should be noted that, to ensure the purity of identity features, a masking weighting strategy can also be used to extract the identity feature vector of the unmasked area, specifically as follows:

[0102] Defective image When input into the ArcFace model, since the occluded areas in the defective image have been set to zero, the feature vectors extracted by the model are entirely derived from the unoccluded areas, resulting in higher identity fidelity.

[0103] Therefore, in this embodiment, a mask weighting strategy is preferably used to extract the identity feature vector of the unmasked area.

[0104] Furthermore, the specific implementation of identity-injected residual diffusion is as follows:

[0105] make This indicates a clean human face image. Indicates a missing image;

[0106] Define residual That is, the offset from a clean image to a defective image;

[0107] This invention constructs a Markov chain of length T for gradually... Towards Transfer; the transfer distribution of the forward process is defined as follows:

[0108] ;

[0109] ;

[0110] in, Let be the residual transfer rate at step t. It is from Monotonically increasing to The sequence controls the transfer progress from a clean image to a damaged image. These are hyperparameters that control noise intensity; this design allows for analytical calculation of the edge distribution of the forward process.

[0111] ;

[0112] when hour, Approximately follows a distribution This refers to a noisy, perturbed version of a defective image.

[0113] Reverse reconstruction process:

[0114] The inverse process aims to learn a conditional distribution. To reverse the forward process, parameterize it as a Gaussian distribution with a mean of By neural network For prediction, variance can be analytically calculated. To stabilize training and improve performance, the mean is parameterized as follows:

[0115] ;

[0116] in, It is a deep neural network with a U-Net structure, whose input is the current state. Defective images and time step The stitching process aims to predict the original clean image. The training objective is simplified to minimizing the prediction. With reality Mean square error between:

[0117] ;

[0118] When reasoning, first start from Medium sampling , and then from arrive Using a trained network calculate and gradually obtained through sampling ;go through After several iterations, the final result is This is the final reconstructed image.

[0119] The extracted identity feature vectors from the unoccluded regions are fed into a residual translation-diffusion network, as follows:

[0120] The residual translation diffusion network consists of an encoder, a bottleneck layer, and a decoder. Skip connections are introduced between the encoder and decoder. At the same time, a cross-attention injection module is embedded in each layer of the bottleneck layer and the decoder to fuse the identity feature vector into the diffusion reconstruction process.

[0121] The identity feature vector of the unoccluded region is injected through a cross-attention mechanism in the network encoder, bottleneck layer, and decoder, specifically as follows:

[0122] Let the feature map output by the current layer be... ,in, d represents the spatial size of the feature map, and d represents the number of channels.

[0123] identity feature vector Mapped to an identity key matrix through a linear transformation layer. and identity value matrix ;

[0124] At the same time, feature map Mapped to a query matrix through another linear transformation layer. ;

[0125] And by calculating cross-attention, we have:

[0126] ;

[0127] in, This represents the computation of cross attention. Represents the identity key matrix, Represents the identity value matrix. Represents the query matrix;

[0128] The output attention feature map is added to the original feature map by residual addition, and then passed through layer normalization and feedforward network to obtain the feature map after injecting identity information.

[0129] The residual map is predicted in the Markov chain iteration, and the final output is a thoroughly cleaned and high-fidelity face image. The specific implementation is as follows:

[0130] Let the number of iterations in the Markov chain of the diffusion process be T, and define the target clean face image as... The defective image is The residual between the two is ;

[0131] Build a from Towards The transition distribution of a forward Markov chain with transitions is defined as follows:

[0132] ;

[0133] in, , For the first The residual transfer rate of the step satisfies the formula , It is a hyperparameter for controlling noise intensity. It is the identity matrix;

[0134] In the reverse process, a neural network is trained. Predict clean images And define the mean of the inverse transition distribution as:

[0135] ;

[0136] in, For the front The cumulative transfer rate of the step;

[0137] In each iteration In the middle, the residual translation and diffusion network is based on the current state and conditional input Predict the residual map; the prediction result is the same as that of the clean image. The estimate is .

[0138] The final reconstructed image is determined based on the residual map prediction results, as follows:

[0139] from Starting with distribution Initial state of mid-sampling ;

[0140] for According to the reverse transition distribution Stepwise sampling;

[0141] go through After one iteration, the obtained For reconstructed facial images .

[0142] In the final reconstructed image, the adversarial textures of the malicious adversarial patch have been completely removed, the occluded areas have been filled with high-fidelity natural textures, and the overall facial features are highly consistent with the original unoccluded areas.

[0143] Furthermore, the final reconstructed face image is output and used as input for downstream tasks (such as face recognition and security verification). Under the 4-step sampling configuration, the average inference time of a single image can be compressed to the millisecond level, meeting the real-time processing requirements of industrial security video streams.

[0144] It should be noted that after identity-injected residual diffusion repair, the complete reconstruction from the defective image to a high-fidelity, non-aggressive face image was achieved, realizing the thorough purification of the adversarial patch and the perfect maintenance of identity consistency.

[0145] This invention employs a repair and defense method that innovatively introduces three core modules: spatial-frequency domain dual-stream intent awareness, intent-driven dynamic mask generation, and identity-injected ultra-fast residual diffusion repair. This achieves, for the first time, semantic-level differentiation and differential processing of physical adversarial attacks and natural occlusion. While effectively dismantling malicious adversarial patch attacks, this invention maximizes the preservation of identity features and natural textures in benignly occluded areas, successfully addressing the technical challenge of balancing robustness and fidelity in existing defense methods. Test results on multiple benchmark datasets demonstrate that this invention significantly outperforms existing technologies in defense success rate, image repair quality (LPIPS / FID), and identity preservation (ID Score), while its inference speed meets the real-time processing requirements of industrial security video streams, possessing significant industrial application value.

[0146] The above description is merely a preferred embodiment of the present invention and does not limit the patent scope of the present invention. Any equivalent structural or procedural transformations made based on the content of the present invention's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of the present invention.

[0147] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a processor-including system, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transmit programs for use by, or in conjunction with, an instruction execution system, apparatus, or device.

[0148] More specific examples of computer-readable media (a non-exhaustive list) include: electrical connections (electronic devices) having one or more wires, portable computer disk drives (magnetic devices), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CDROM). Furthermore, computer-readable media can even be paper or other suitable media on which the program can be printed, because the program can be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, interpreting, or otherwise processing as necessary, and then stored in computer memory.

[0149] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention, and all such modifications or substitutions should be covered within the scope of protection claimed by the present invention.

Claims

1. An adaptive adversarial patch inpainting and defense method based on intention perception, characterized in that: Includes the following steps: Intent classification based on spatial-frequency domain dual-stream perception is as follows: The system acquires an input image containing occlusions, extracts features based on a dual-stream sensing network, concatenates the features extracted by the dual-stream sensing network, determines the intent attribute based on the concatenated features, and outputs a classification result of benign or malicious occlusion. The mask generation strategy is dynamically switched based on the judgment result, and an image cutout operation is performed to determine the defective image, specifically as follows: The mask generation strategy is dynamically switched based on the judgment result. If it is determined to be benign occlusion, the high-precision segmentation model is called to generate the mask; if it is determined to be malicious occlusion, the adversarial anomaly perception model is called to generate the mask. The image is then cut out based on the generated mask to determine the missing image. And, residual diffusion repair based on cross-attention identity injection, specifically: After generating the defective image, the defective image and the original image are concatenated by channels and fed into the residual translation-diffusion network. The identity feature vectors of the unoccluded areas are extracted from the defective image and injected into the residual translation-diffusion network through a cross-attention mechanism. The residual map is predicted in the Markov chain iteration, and the final reconstructed image is determined based on the prediction results of the residual map.

2. The adaptive inpainting patch image inpainting and defense method based on intention perception of claim 1, wherein: The feature extraction based on the dual-stream sensing network is as follows: The dual-stream sensing network consists of a spatial stream and a frequency stream. It takes an input image containing occlusions and inputs it in parallel into both the spatial and frequency streams for feature extraction. Specifically: Spatial Flow uses a multi-layer multi-head self-attention mechanism to perform global semantic modeling of images, extracting semantic conflict information between local texture features of occluded areas and the overall facial topology. The frequency domain stream first performs a fast Fourier transform on the input image to convert the image from the spatial domain to the frequency domain, obtaining the frequency domain amplitude spectrum and phase spectrum. Then, it performs a high-pass filter on the amplitude spectrum to extract the high-frequency energy components, and uses an inverse Fourier transform to map the high-frequency energy distribution back to the spatial domain to generate a high-frequency energy heat map.

3. The adaptive inpainting patch image inpainting and defense method based on intention perception of claim 2, wherein: The determination of intent attributes based on the concatenated features is as follows: The spatial feature map output from the spatial stream and the high-frequency energy heat map output from the frequency domain stream are stitched together along the channel dimension to obtain a fused feature map. The fusion feature map is input into an intent classifier, which is composed of a global average pooling layer, a fully connected layer and a Softmax activation function, and outputs a classification probability distribution of the occlusion attribute for determining the intent attribute, specifically: If the output result of the intention classifier satisfies the formula , it indicates that the occlusion region in the fusion feature map and the global structure of the face present semantic coherence and normal high-frequency energy distribution, and it is determined as a benign occlusion. If the output of the intent classifier satisfies the formula If the occlusion region in the fused feature map has significant semantic breaks and abnormal high-frequency energy aggregation, it is determined to be malicious occlusion.

4. The intention-aware adaptive adversarial patching image inpainting and defense method as described in claim 3, characterized in that: The dynamic switching of the mask generation strategy based on the determination result is as follows: When the output of the intent classifier satisfies the formula At that time, the high-precision segmentation model SAM-2 is called to generate a mask; When the output of the intent classifier satisfies the formula At that time, a lightweight adversarial anomaly-aware two-node diffusion model is invoked to generate a mask.

5. The intention-aware adaptive adversarial patching image inpainting and defense method as described in claim 4, characterized in that: The process of using the high-precision segmentation model SAM-2 to generate the mask is as follows: The original image is input into the SAM-2 model, using the center point of the occluded region as a cue. The model outputs a high-precision instance segmentation mask through an encoder-decoder structure. The output instance segmentation mask can closely fit the geometric contours of natural occlusions, ensuring that benign occlusion areas are marked completely and accurately.

6. The intention-aware adaptive adversarial patching image inpainting and defense method as described in claim 5, characterized in that: The process of generating the mask by invoking a lightweight adversarial anomaly detection two-node diffusion model is as follows: Construct a two-branch diffusion model where the two branches share the same network structure but use different noise intensity parameters; First-level noise and second-level noise are injected into the input image respectively, and then denoising and reconstruction are performed separately to obtain two reconstructed images. The variance of the two reconstructed images at each pixel position is calculated. Set a variance threshold, mark pixel regions with variance values ​​greater than the variance threshold in the variance map as abnormal regions, and remove isolated noise points through morphological closing operations to obtain the malicious patch mask.

7. The intention-aware adaptive adversarial patching image inpainting and defense method as described in claim 6, characterized in that: The image cutout operation is performed to determine the defective image, as follows: Performing pixel-by-pixel operations on the original image and the mask, we get: ； in, Indicates pixel-by-pixel multiplication. This represents the output image with defects. This represents the original input image. The generated mask includes: instance segmentation mask and malicious patch mask.

8. The intention-aware adaptive adversarial patching image inpainting and defense method as described in claim 7, characterized in that: The method of injecting the identity feature vector of the unoccluded region into the residual translation-diffusion network through the cross-attention mechanism is as follows: Let the feature map output by the current layer be... ,in, d represents the spatial size of the feature map, and d represents the number of channels. Identity feature vector The feature map is mapped to an identity key matrix and an identity value matrix through a linear transformation layer, and simultaneously mapped to a query matrix through another linear transformation layer. And by calculating cross-attention, we have: ； in, This represents the computation of cross attention. Represents the identity key matrix, Represents the identity value matrix. Represents the query matrix. Dimensions representing attention mechanisms; The output attention feature map is added to the original feature map by residual addition, and then passed through layer normalization and feedforward network to obtain the feature map after injecting identity information.

9. The intention-aware adaptive adversarial patching image inpainting and defense method as described in claim 8, characterized in that: The prediction of the residual map in Markov chain iteration is as follows: Let the number of iterations in the Markov chain of the diffusion process be T, and define the target clean face image as... The defective image is The residual between the two is ; Build a from Towards The transition distribution of a forward Markov chain with transitions is defined as follows: ； in, , For the first The residual transfer rate of the step satisfies the formula , It is a hyperparameter for controlling noise intensity. It is the identity matrix; In the reverse process, a neural network is trained. Predict clean images And define the mean of the inverse transition distribution as: ； in, For the front The cumulative transfer rate of the step; In each iteration In the middle, the residual translation and diffusion network is based on the current state and conditional input Predict the residual map; the prediction result is the same as that of the clean image. The estimate is .

10. The intention-aware adaptive adversarial patching image inpainting and defense method as described in claim 9, characterized in that: The process of determining the final reconstructed image based on the residual map prediction results is as follows: from Starting with distribution Initial state of mid-sampling ; for According to the reverse transition distribution Stepwise sampling; go through After one iteration, the obtained For reconstructed facial images .

Images