End-to-end generation anti-Moire pattern attack method based on imbedding information

Abstract

The invention discloses an end-to-end generation anti-Moire pattern attack method based on imbedding information, which belongs to the technical field of image data processing and comprises the following steps: acquiring a data set D and a target tracker; constructing an end-to-end adversarial sample generation network which comprises a moire generation pipeline and a generative adversarial network; and training the network to obtain an adversarial sample generation model which is used for generating adversarial samples in an end-to-end manner. The confrontation disturbance form is designed for the visual effect for the first time, and the performance of the twin network tracker is greatly reduced under the superposition attack of double disturbances of Moire lines and confrontation disturbance. According to the method, the defect that when the U-Net network is used for image generation, the shape of the model cannot be changed after training is completed is overcome, so that the change of Moire lines is richer and more realistic. An end-to-end generation method is used, so that the generation speed of an adversarial sample meets the real-time requirement of target tracking, and the problem of high resource overhead of Moire pattern generation in a mathematical mode is solved.

Description

Technical Field

[0001] The present invention relates to the technical field of image data processing, and in particular to an end-to-end generation method for countering moiré attacks based on embedded information. Background Art

[0002] Single target tracking technology aims to extract feature information by analyzing the target area marked in the initial frame of the video sequence, and continuously predict the position change of the target in subsequent frames. In recent years, with the breakthrough progress of convolutional neural networks (CNN), tracking algorithms using twin neural network architecture have demonstrated excellent performance advantages, with significant improvements in tracking accuracy and real-time performance. Representative algorithms in this field include SiamFC, SiamRPN series, SiamMask, and Ocean. These algorithms are all based on the twin neural network architecture, which includes two sub-networks with shared weights, generally referred to as template branches and detection branches, which are used to process template frames and search area frames respectively, and locate the target by comparing their features. Therefore, the target tracker of the twin network can be simply divided into the following three parts: template branch, detection branch, similarity calculation and target positioning: Template branch: Responsible for extracting features from the template frame (also called the template region). This includes (1) Initial frame annotation: In the first frame of the video, the target location is determined by manual annotation and represented by a bounding box. (2) Cropped template frame: Based on the annotated bounding box, the target region is cropped from the initial frame as the template frame. The size of the template frame is usually fixed (e.g., 127×127 pixels). (3) Feature extraction: The template frame is input into the template branch of the Siamese network to extract the feature representation of the template.

[0003] Detection branch: Responsible for extracting features from the search region frame. (1) Cropping the search region: In each frame, a larger search region is cropped with the target's predicted position as the center. The size of the search region is usually larger than the template frame to cover the target's possible motion range. (2) Feature extraction: The search region is input into the detection branch of the Siamese network to extract the feature representation of the search region.

[0004] Similarity calculation and target positioning: By comparing the features of the template frame and the search area, the position of the target in the current frame is determined. Generally, classification maps and regression maps are output for the search area.

[0005] In recent years, research on adversarial examples in deep neural networks has demonstrated that by injecting tiny perturbations into images that are imperceptible to the human eye, state-of-the-art deep learning models can be successfully fooled. This phenomenon also exists in the field of object tracking. Researchers have designed specific objective functions and used network model optimization or iterative methods to generate visually imperceptible adversarial perturbations. These perturbations are then embedded into video frame sequences, ultimately causing advanced trackers to lose track of the target during normal tracking. Specifically, existing adversarial attack methods mainly include the following typical schemes: CSA (cooling shrinkage attack) effectively suppresses the response value of the target area in the heat map by designing a cooling shrinkage loss function, while inducing the tracker to continuously shrink the predicted bounding box, ultimately causing target loss; IoU (intersection over union attack) uses an orthogonal approximation method to iteratively search for the minimum noise perturbation in a black-box manner, thereby minimizing the tracker's IoU evaluation score and thus destabilizing tracking; DFA (decrementing feature attack) applies more subtle perturbations to the input frames, interfering with the feature map extraction and bounding box prediction process, ultimately rendering the tracker ineffective. These attack methods reveal the vulnerability of current tracking systems to adversarial examples from different perspectives. However, some problems still exist: (1) The existing target tracking adversarial attack algorithms often use disturbances such as noise or stickers. Existing forms of simulating natural disturbances such as motion blur and moiré require a lot of resource overhead, and their methods are difficult to meet the real-time application scenarios of target tracking.

[0006] (2) Current efforts to counteract disturbance concealment focus on reducing the degree of disturbance, with little attention paid to improving the concealment of disturbances from perspectives such as visual concealment effects. Furthermore, the concealment of methods that reduce the degree of disturbance is negatively correlated with the degree of countermeasures. Higher concealment often results in lower countermeasures, and the generation of disturbances cannot break through this limitation.

[0007] (3) After training, the Unet network used to generate adversarial samples for target tracking generates fixed adversarial perturbations, which cannot be changed by other means. Fixed perturbations are easily recognized by defense algorithms, and cannot reduce the similarity between frames to a greater extent. Summary of the Invention

[0008] The purpose of the present invention is to provide a method for generating an end-to-end countermeasure against moiré attacks based on embedded information to solve the above-mentioned defects, such as the large amount of resource consumption required for moiré, which makes it difficult to meet the real-time application scenarios of target tracking, the higher concealment makes the adversarial resistance poor, and the fixed disturbance is easily identified by the defense algorithm.

[0009] To achieve the above objectives, the present invention adopts the following technical solution: an end-to-end generation method for countering moiré attacks based on embedded information, comprising the following steps: S1, obtain the dataset D and the target tracker; The dataset D includes multiple video sequences, and the images in each video sequence are labeled with the target; The target tracker is pre-trained with the dataset D and is used to generate a classification map and a regression map of the search area for each frame image in the video sequence according to the template area and the search area. For a video sequence, the i-th image X i The search area is S i , ,H、W、3 are S respectively i Height, width, and number of channels; S2, construct an end-to-end adversarial sample generation network, including a moiré generation pipeline and a generative adversarial network; The moiré generation pipeline generates a moiré mask M on a blank image using a moiré feature parameter set (β, γ, η). Moire , and cover the search area according to the coverage rate α to obtain the real moiré search area , where β is the camera rotation angle, γ is the display distance scaling factor, and η is the display type; Magnify β, γ, and η to H×W respectively and compare them with S i Splicing along the channel dimension to obtain the spliced ​​search area , ; The generative adversarial network includes a generator and a discriminator, wherein the generator input , output S i Corresponding adversarial search area , the discriminator is used to input , output The mean μ and variance σ of 2 ; S3, training an end-to-end adversarial sample generation network to obtain an adversarial sample generation model, including steps S31 to S37; S31, randomly select a video sequence from the data set D, obtain the template area and the search area, where X i The search area is S i ; S32, for each search area S i , the moiré search area is generated by the moiré generation pipeline ; S33, the generator obtains the splicing search area , input generator to get adversarial search area , calculate the similarity loss L similar ; S34, will Input the discriminator to get the corresponding mean μ and variance σ2 , calculate the discriminator loss L NNLL ; S35, the template area and Input the target tracker and get Corresponding classification diagram C i and regression plot R i , calculate the adversarial deception loss L deception ; S36, calculate the total loss L total , and to minimize Adjust the network parameters of the generator and discriminator, L total =L deception +L similar +L NNLL ; S37, repeat S31 to S36 until the network converges to obtain the adversarial sample generation model; S4, obtaining a video sequence to be attacked, and attacking the search area and / or template area based on the adversarial sample generation model.

[0010] Preferably, the adversarial generative adversarial network is based on a U-Net network.

[0011] Preferably, the The calculation formula is, .

[0012] Preferably, the similarity loss Calculate according to the following formula; , , , , Where, L1, L LPIPS , L TV They are L1 loss, perceptual similarity loss, and total variation loss, 、 、 L1, L LPIPS , L TV The weight of is the L1 norm, for The eigenvalue at width w and height h, for The characteristic value at the width w and height h is 1≤h≤H, 1≤w≤W, is element-wise multiplication, is the square of the L2 norm, for The eigenvalue at width w and height h-1, for Eigenvalue at width w+1 and height h.

[0013] Preferably, the discriminator loss L NNLL Calculate according to the following formula; , Where B is the number of samples in a batch, d is the dth channel, 1≤d≤3, For very small numbers, for The variance of the value of the dth channel, for The mean value of the dth channel.

[0014] Preferably, the anti-deception loss Calculate according to the following formula; , Where, is the classification score threshold, AP is the classification graph C i The classification score is greater than The anchor points constitute a set, is the kth anchor point An in AP k The classification score of 、 An k In the regression plot R i The width regression value and height regression value in are λ1 and λ2, respectively, which are the classification graph weight and regression graph weight.

[0015] Preferably, attacking the search area in S4 includes steps Sa1 to Sa2; Sa1, the i-th image in the attack video sequence , get its search area , and the amplified β, γ, η are spliced ​​along the channel dimension to obtain the spliced ​​search area ; Sa2, will Input generator to get adversarial search area , and then input the target tracker to perform tracking tasks.

[0016] Preferably, attacking the template region in S4 includes steps Sb1 to Sb2: Sb1, obtain the template region T of the video sequence to be attacked z , zero-filled to the search area size, and spliced ​​with the enlarged β, γ, η along the channel dimension to obtain the splicing template area ; Sb2, The zero-filled area is cropped and restored to the template area size, and then input into the target tracker for tracking task.

[0017] Compared with the prior art, the advantages of the present invention are: (1) For the first time, we design an adversarial perturbation form for visual effects and propose an adversarial attack method that hides the adversarial perturbation under the moiré pattern. This adversarial sample is different from the previous visual concealment. Based on the selective ignoring mechanism and the visual masking effect, the adversarial perturbation is hidden under the moiré pattern that can be seen everywhere, making the adversarial perturbation difficult to detect. Under the superposition attack of the moiré pattern and the adversarial perturbation, the performance of the twin network tracker is greatly reduced.

[0018] (2) The present invention increases the dimension of the embedded information in the U-Net network input layer and adds a discriminator to process the U-Net network output samples during the training process. The discriminator can constrain the consistency of the generated image and the embedded information parameters, so that the trained model can change the perturbation form according to the embedded parameters, solving the problem of a single perturbation form. This method solves the problem that when using the U-Net network for image generation, the model cannot change its shape after training, making the changes in the moiré pattern richer and more realistic.

[0019] (3) An end-to-end generation method is used to ensure that the speed of adversarial sample generation meets the real-time requirements of target tracking. The design and use of the adversarial loss function can greatly reduce the tracking accuracy of the target tracking model. In addition, this end-to-end method can also solve the problem of high resource consumption when generating moiré patterns mathematically.

[0020] (4) The attack method proposed in the present invention can attack the template area, the search area, or both areas in the video sequence. The present invention shows good countermeasure effects in all three modes.

[0021] (5) The present invention can transfer its attack effectiveness to other trackers, including SiamRPN, SiamMask, and Ocean, in a plug-and-play manner.

[0022] (6) The improved U-Net network and discriminator can be applied to other image generation algorithms that require parameter changes. The loss function of the present invention can be used for optimization during the training process to change the image style according to the input information during the generation stage. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 This is a diagram of the architecture of the present invention; Figure 2 Schematic diagram of the generator structure based on the Unet architecture; Figure 3aFor a confrontation search area one generated by the present invention; Figure 3b for Figure 3a Image after removing moiré patterns; Figure 4a To counter the search area 2 generated by the present invention; Figure 4b for Figure 4a Image after removing moiré patterns; Figure 5a For a confrontation search area three generated by the present invention; Figure 5b for Figure 5a Image after removing moiré patterns; Figure 6a A fourth confrontation search area generated by the present invention; Figure 6b for Figure 6a Image after removing moiré. DETAILED DESCRIPTION

[0024] The present invention will be further described below with reference to the embodiments and accompanying drawings.

[0025] Example 1: See Figure 1 and Figure 2 , an end-to-end generation method for countering moiré attack based on embedded information, comprising the following steps: S1, obtain the dataset D and the target tracker; The dataset D includes multiple video sequences, and the images in each video sequence are labeled with the target; The target tracker is pre-trained with the dataset D and is used to generate a classification map and a regression map of the search area for each frame image in the video sequence according to the template area and the search area. For a video sequence, the i-th image X i The search area is S i , ,H、W、3 are S respectively i Height, width, and number of channels; S2, construct an end-to-end adversarial sample generation network, including a moiré generation pipeline and a generative adversarial network; The moiré generation pipeline generates a moiré mask M on a blank image using a moiré feature parameter set (β, γ, η). Moire , and cover the search area according to the coverage rate α to obtain the real moiré search area , where β is the camera rotation angle, γ is the display distance scaling factor, and η is the display type; Magnify β, γ, and η to H×W respectively and compare them with S i Splicing along the channel dimension to obtain the spliced ​​search area , ; The generative adversarial network includes a generator and a discriminator, wherein the generator input , output S i Corresponding adversarial search area , the discriminator is used to input , output The mean μ and variance σ of 2 ; S3, training an end-to-end adversarial sample generation network to obtain an adversarial sample generation model, including steps S31 to S37; S31, randomly select a video sequence from the data set D, obtain the template area and the search area, where X i The search area is S i ; S32, for each search area S i , the moiré search area is generated by the moiré generation pipeline ; S33, the generator obtains the splicing search area , input generator to get adversarial search area , calculate the similarity loss L similar ; S34, will Input the discriminator to get the corresponding mean μ and variance σ 2 , calculate the discriminator loss L NNLL ; S35, the template area and Input the target tracker and get Corresponding classification diagram C i and regression plot R i , calculate the adversarial deception loss L deception ; S36, calculate the total loss L total , and to minimize Adjust the network parameters of the generator and discriminator, L total =L deception +L similar +L NNLL ; S37, repeat S31 to S36 until the network converges to obtain the adversarial sample generation model; S4, obtaining a video sequence to be attacked, and attacking the search area and / or template area based on the adversarial sample generation model.

[0026] In this embodiment, the adversarial generative adversarial network is based on the U-Net network.

[0027] described The calculation formula is, .

[0028] The similarity loss Calculate according to the following formula; , , , , Where, L1, L LPIPS , L TV They are L1 loss, perceptual similarity loss, and total variation loss, 、 、 L1, L LPIPS , L TV The weight of is the L1 norm, for The eigenvalue at width w and height h, for The characteristic value at the width w and height h is 1≤h≤H, 1≤w≤W, is element-wise multiplication, is the square of the L2 norm, for The eigenvalue at width w and height h-1, for Eigenvalue at width w+1 and height h.

[0029] The discriminator loss L NNLL Calculate according to the following formula; , Where B is the number of samples in a batch, d is the dth channel, 1≤d≤3, For very small numbers, for The variance of the value of the dth channel, for The mean value of the dth channel.

[0030] The adversarial deception loss Calculate according to the following formula; , Where, is the classification score threshold, AP is the classification graph C i The classification score is greater than The anchor points constitute a set, is the kth anchor point An in AP k The classification score of 、 An k In the regression plot R i The width regression value and height regression value in are λ1 and λ2, respectively, which are the classification graph weight and regression graph weight.

[0031] Figure 2 A schematic diagram of the generator structure based on the Unet architecture is given. Assuming H=W=512, then , enlarge β, γ, and η to 512×512 respectively, and splice them with Si along the channel dimension to obtain the spliced ​​search area , , the image scale of the input generator is 512×512×6, such as Figure 2 As shown in the figure, the generator utilizes the existing U-Net framework, consisting of an encoder and decoder. The encoder consists of repeated dual convolutional units, each of which contains two 3×3 convolutional layers, a ReLU activation function, and a batch normalization layer. This is followed by a 2×2 max pooling layer (with a stride of 2) for downsampling. Each layer halves the spatial size of the feature map and doubles the number of channels, gradually extracting multi-scale abstract features of the input image and information. The decoder restores resolution through upsampling through deconvolutional layers. Each step doubles the size of the feature map and skips the map with the feature map from the corresponding encoder layer, fusing details and semantic information. The features are then optimized through dual convolutional units, while the number of channels is halved layer by layer. Finally, 1×1 convolutions are used to map the features to an output image of 512×512×3. The output of the generator is fed into the discriminator, which is based on existing technology.

[0032] Example 2: See Figure 1 and Figure 2 Based on Example 1, this example provides three specific attack methods: Method 1: Attack only the search area, including steps Sa1 and Sa2; Sa1, the i-th image in the attack video sequence , get its search area , and the amplified β, γ, η are spliced ​​along the channel dimension to obtain the spliced ​​search area ; Sa2, will Input generator to get adversarial search area , then input the target tracker to perform the tracking task. This is fed into the detection branch of the object tracker.

[0033] Method 2: Attack only the template area, including steps Sb1 and Sb2: Sb1, obtain the template region T of the video sequence to be attacked z, zero-filled to the search area size, and spliced ​​with the enlarged β, γ, η along the channel dimension to obtain the splicing template area ; Sb2, The zero-filled area is cropped, restored to the template area size, and then input into the template branch of the target tracker for tracking task.

[0034] Method 3: Attack the search area and template area simultaneously; Obtain the adversarial search area according to method 1 , fed into the detection branch of the target tracker; Method 2: Get the splicing template area And restore it back to the template area size and send it to the template branch of the target tracker.

[0035] Example 3: See Figure 3a to Figure 6b In this embodiment, the method of the present invention is used to generate four adversarial search areas, respectively as follows: Figure 3a 、 Figure 4a 、 Figure 5a 、 Figure 6a As shown, the mainstream de-moiré algorithm UHDM (Ultra-High-Definition Imaging Demoireing) is used to remove the moiré patterns on the four images, and the corresponding Figure 3b 、 Figure 4b 、 Figure 5b 、 Figure 6b .

[0036] contrast Figure 3a and Figure 3b It can be seen that using the de-noising algorithm based on this method not only fails to remove the moiré pattern well, but may also further degrade the image quality, such as Figure 3b The upper right center area also makes the moiré pattern more distinct.

[0037] contrast Figure 4a and 4b It can be seen that using the de-moiré algorithm based on this method cannot completely remove all moiré textures.

[0038] contrast Figure 5a and 5b It can be seen that the use of the de-moiré algorithm based on this method makes the overall color of the original image white, the image is more blurred, and the moiré texture is not reasonably removed.

[0039] contrast Figure 6a and 6b It can be seen that the de-moiré algorithm used on the basis of this method removes part of the moiré texture around the target, but the moiré disturbance covering the target is not removed well, which still affects the tracking performance.

[0040] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A method for generating an end-to-end anti-moire attack based on embedded information, characterized in that: The following steps are involved: S1, obtain the dataset D and the target tracker; The dataset D includes multiple video sequences, and the images in each video sequence are labeled with the target; The target tracker is pre-trained with the dataset D and is used to generate a classification map and a regression map of the search area for each frame image in the video sequence according to the template area and the search area. For a video sequence, the i-th image X i The search area is S i , ,H、W、3 are S respectively i Height, width, and number of channels; S2, construct an end-to-end adversarial sample generation network, including a moiré generation pipeline and a generative adversarial network; The moiré generation pipeline generates a moiré mask M on a blank image using a moiré feature parameter set (β, γ, η). Moire , and cover the search area according to the coverage rate α to obtain the real moiré search area , where β is the camera rotation angle, γ is the display distance scaling factor, and η is the display type; Magnify β, γ, and η to H×W respectively and compare them with S i Splicing along the channel dimension to obtain the spliced ​​search area , ; The generative adversarial network includes a generator and a discriminator, wherein the generator input , output S i Corresponding adversarial search area , the discriminator is used to input , output The mean μ and variance σ of 2 ; S3, training an end-to-end adversarial sample generation network to obtain an adversarial sample generation model, including steps S31 to S37; S31, randomly select a video sequence from the data set D, obtain the template area and the search area, where X i The search area is S i ; S32, for each search area S i , the moiré search area is generated by the moiré generation pipeline ; S33, the generator obtains the splicing search area , input generator to get adversarial search area , calculate the similarity loss L similar ; S34, will Input the discriminator to get the corresponding mean μ and variance σ 2 , calculate the discriminator loss L NNLL ; S35, the template area and Input the target tracker and get Corresponding classification diagram C i and regression plot R i , calculate the adversarial deception loss L deception ; S36, calculate the total loss L total , and to minimize Adjust the network parameters of the generator and discriminator, L total =L deception +L similar +L NNLL ; S37, repeat S31 to S36 until the network converges to obtain the adversarial sample generation model; S4, obtaining a video sequence to be attacked, and attacking the search area and / or template area based on the adversarial sample generation model.

2. The method for end-to-end generation of countermeasures against moiré attacks based on embedded information according to claim 1, characterized in that: The adversarial generative adversarial network is based on the U-Net network.

3. The method for end-to-end generation of countermeasures against moiré attacks based on embedded information according to claim 1, characterized in that: described The calculation formula is, .

4. The method for end-to-end generation of countermeasures against moiré attacks based on embedded information according to claim 1, characterized in that: The similarity loss Calculate according to the following formula; ， ， ， ， Where, L1, L LPIPS 、L TV They are L1 loss, perceptual similarity loss, and total variation loss, 、 、 L1, L LPIPS 、L TV The weight of is the L1 norm, for The eigenvalue at width w and height h, for The characteristic value at the width w and height h is 1≤h≤H, 1≤w≤W, is element-wise multiplication, is the square of the L2 norm, for The eigenvalue at width w and height h-1, for Eigenvalue at width w+1 and height h.

5. The method for end-to-end generation of countermeasures against moiré attacks based on embedded information according to claim 1, characterized in that: The discriminator loss L NNLL Calculate according to the following formula; ， Where B is the number of samples in a batch, d is the dth channel, 1≤d≤3, For a very small number, for The variance of the value of the dth channel, for The mean value of the dth channel.

6. The method for end-to-end generation of countermeasures against moiré attacks based on embedded information according to claim 1, characterized in that: The adversarial deception loss Calculate according to the following formula; ， Where, is the classification score threshold, AP is the classification graph C i The classification score is greater than The anchor points constitute a set, is the kth anchor point An in AP k The classification score of 、 An k In the regression plot R i The width regression value and height regression value in are λ1 and λ2, respectively, which are the classification graph weight and regression graph weight.

7. The method for end-to-end generation of countermeasures against moiré attacks based on embedded information according to claim 1, characterized in that: Attacking the search area in S4 includes steps Sa1 to Sa2; Sa1, the i-th image in the attack video sequence , get its search area , and the amplified β, γ, η are spliced ​​along the channel dimension to obtain the spliced ​​search area ; Sa2, will Input generator to get adversarial search area , and then input the target tracker to perform tracking tasks.

8. The method for end-to-end generation of countermeasures against moiré attacks based on embedded information according to claim 1, characterized in that: The attack on the template area in S4 includes steps Sb1~Sb2: Sb1, obtain the template region T of the video sequence to be attacked z , zero-filled to the search area size, and spliced ​​with the enlarged β, γ, η along the channel dimension to obtain the splicing template area ; Sb2, The zero-filled area is cropped and restored to the template area size, and then input into the target tracker for tracking task.

Images