An Attention-Guided Adversarial Attack Method for Video Object Segmentation

Abstract

The present invention discloses an attention-guided adversarial attack method for video target segmentation, firstly, a feature extractor is used to extract a feature map of a video frame; then a multi-level feature space attention module is constructed to obtain a spatial attention map; the spatial attention map is connected in series with the feature map to obtain an enhanced feature map, and the enhanced feature map is downsampled; a deconvolutional neural network is constructed to generate an adversarial perturbation of the original video sequence based on the downsampled enhanced feature map; the generated adversarial perturbation is added to the original video sequence to generate an adversarial sample; given an initial frame mask, the adversarial sample is input into a classic single-sample video target segmentation network to obtain a final prediction mask. The present invention provides an adversarial attack method that generates adversarial perturbations with the help of an attention mechanism and an enhanced feature map, and can perform adversarial attacks on video target segmentation tasks, so as to achieve misclassification of all pixels of the target.

Description

Technical Field

[0001] The invention relates to an attention-guided adversarial attack method for video target segmentation, and belongs to image processing technology. Background Art

[0002] Video object segmentation is a core task in computer vision and has a wide range of applications in video editing, video synthesis, and autonomous driving. Semi-supervised video object segmentation requires segmenting the target object in a video sequence given an initial mask. In recent years, with the advancement of deep learning, video object segmentation algorithms have made many breakthroughs.

[0003] Although deep neural networks have achieved excellent performance in video object segmentation, they are vulnerable to malicious adversarial attacks that aim to mislead the model with small perturbations that are imperceptible to humans. Several research works have shown that neural networks can be easily fooled. These works mainly focus on generating adversarial examples for computer vision classification tasks. The tiny perturbations added to the image, although usually imperceptible to humans, can easily fool deep neural network based classifiers, detectors, trackers and segmenters, leading to wrong results. This has attracted great attention, especially when deep neural networks are applied in safety and security critical scenarios. For specific tasks, specific domain research and understanding how adversarial attacks affect the performance of deep neural networks will be key to reduce these effects and further enhance robustness.

[0004] However, most of the existing work focuses on exploring adversarial examples for image classification models, and adversarial attacks on video segmentation models have rarely been studied. In fact, video object segmentation models based on deep neural networks are increasingly being deployed in real-world scenarios such as surveillance. Studying adversarial examples helps understand the working mechanism of deep models and provides opportunities to improve the robustness of the network. Therefore, it is crucial to study adversarial attack methods for video models. Summary of the invention

[0005] Purpose of the invention: In order to overcome the deficiencies in the prior art, the present invention provides an attention-guided adversarial attack method for video target segmentation, which invalidates the target model by finding adversarial samples with imperceptible differences from the video target segmentation model, while considering all targets and optimizing the overall loss. The adversarial attack method studied in this case helps to understand the working mechanism of the deep model to better identify the vulnerability of the segmentation algorithm, and provides an opportunity to improve the robustness of the video target segmentation algorithm.

[0006] Technical solution: To achieve the above purpose, the technical solution adopted by the present invention is:

[0007] An attention-guided adversarial attack method for video object segmentation includes the following steps:

[0008] (1) The video sequence X = {x 1 ,x 2 ,…,x n Input feature extractor F θ , get the video frame x i The feature map F i ; Where θ represents the parameter to be learned, i is the index of the video frame, i = 1, 2, ..., n, and n is the number of video frames;

[0009] (2) Construct a multi-level feature space attention module to obtain the feature map F i Spatial attention map

[0010] (3) The spatial attention map With the feature map F i Connect them in series to get the enhanced feature map E i , for the enhanced feature map E i Downsample to get E i ′;

[0011] (4) Construct a deconvolutional neural network based on the enhanced feature map E after downsampling i ′, produces adversarial perturbation ΔX of video sequence X;

[0012] (5) Add the adversarial perturbation ΔX to the video sequence X to generate the adversarial sample X adv ;

[0013] (6) Given the initial frame mask m, the adversarial sample X adv Input the single-sample video object segmentation network to obtain the final predicted mask.

[0014] Preferably, in step (1), the feature extractor F θ The VGG16 network in the single-sample video object segmentation model is used, and the last convolutional layer of the VGG16 network generates a video frame x i Feature map Where: H, W and C represent the feature map F i The height, width, and number of channels.

[0015] Preferably, in step (2), the multi-level feature space attention module first performs an operation on the feature map F i Perform average pooling and maximum pooling in a channel dimension to obtain two H×W×1 channel descriptions; then concatenate these two channel descriptions together; then pass through a 7×7 convolution layer with Sigmoid as the activation function, and use a Softmax layer to obtain the spatial attention map

[0016]

[0017] Where: σ(·) represents the Sigmoid function, f 7×7 Represents a convolution operation with a filter size of 7×7; AvgPool(F i ) represents the feature map F i Average pooling, the pooling result is MaxPool(F i ) represents the feature map F i Maximum pooling, the pooling result is

[0018] Preferably, in step (4), the deconvolution neural network is trained according to the following class loss function:

[0019]

[0020] in: represents the cross entropy loss function, y j is the video frame x i The pixel-level binary label, Y + and Y - is a pixel with positive or negative labels, β=|Y - | / |Y + |;

[0021] The deconvolutional neural network uses the following attention feature loss function:

[0022]

[0023] in: represents the Hadamard product, Represents video frame x i Adversarial examples, x r is a random predefined feature map; Represents the feature extractor F θ Extracted The feature map, F θ (x r ) indicates that the feature extractor F θ Extracted x r The feature map of ||·|| 2 represents the L2 norm;

[0024] The deconvolutional neural network uses the following total loss function:

[0025]

[0026] Where: λ represents the importance weight of controlling noise and cross entropy loss, μ represents the importance weight of the target in the video frame, ΔX = {Δx 1 ,Δx 2 ,…Δx n} represents the adversarial perturbation of the video sequence X, Δx i Represents video frame x i adversarial perturbation.

[0027] Preferably, in step (5), Where: ε is the maximum allowable adversarial perturbation threshold.

[0028] Preferably, in step (6), the attack on the single sample video target segmentation network is expressed as:

[0029]

[0030]

[0031] Where: S θ (X adv ,T τ (X adv )) indicates the use of adversarial sample X adv Single-sample video object segmentation network for segmentation, T τ (X adv ) represents the adversarial sample X adv The target model of the target perception output in the target model, τ represents the parameter of the integrated target model appearance information encoding, and τ is determined by the adversarial sample X adv The initial frame and its given initial frame mask m are determined, and the final predicted mask is output using target perception.

[0032] Beneficial effect: The attention-guided adversarial attack method for video target segmentation provided by the present invention generates adversarial perturbations with the help of attention mechanism and enhanced feature map, and can perform adversarial attacks on video target segmentation tasks to achieve the misclassification of all pixels of the target; the present invention studies how adversarial attacks affect model performance, thereby weakening the impact on the model and further enhancing the robustness of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 It is a flow chart of the implementation of the present invention;

[0034] Figure 2 It is a schematic diagram of the structure of the multi-level feature space attention module;

[0035] Figure 3 Schematic diagram of the obtained adversarial perturbations and adversarial samples;

[0036] Figure 4 A schematic diagram of the structure of a device for implementing the inventive method. DETAILED DESCRIPTION

[0037] The present invention is described in detail below with reference to the accompanying drawings and specific embodiments.

[0038] In the description of the present invention, it is necessary to understand that the terms "center", "longitudinal", "lateral", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside" and the like indicate positions or positional relationships based on the positions or positional relationships shown in the drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operate in a specific orientation, and therefore cannot be understood as limiting the present invention. In addition, the terms "first" and "second" are used for descriptive purposes only and cannot be understood as indicating or implying relative importance.

[0039] like Figure 1 The figure shows an implementation flow chart of an attention-guided adversarial attack method for video object segmentation, and each step is described in detail below.

[0040] Step S01: Video sequence X = {x 1 ,x 2 ,…,x n Input feature extractor F θ , get the video frame x i The feature map F i .

[0041] Feature Extractor F θ The VGG16 network in the single-sample video object segmentation model is used, and the last convolutional layer of the VGG16 network generates a video frame x i Feature map Where: i is the index of the video frame, i = 1, 2, ..., n, n is the number of video frames, θ represents the parameter to be learned, H, W and C represent the feature map F i The height, width and number of channels, Represents a three-dimensional matrix of H×W×C

[0042] Step S02: Construct a multi-level feature space attention module to obtain the feature map F i Spatial attention map

[0043] like Figure 2 As shown in Figure 2, when constructing a multi-level feature spatial attention module, we focus on spatial information with rich location information and use average pooling and maximum pooling to obtain a spatial attention map.

[0044]

[0045] Where: σ(·) represents the Sigmoid function, f 7×7 Represents a convolution operation with a filter size of 7×7; AvgPool(F i ) represents the feature map F i Average pooling, the pooling result is MaxPool(F i ) represents the feature map F i Maximum pooling, the pooling result is Represents a three-dimensional matrix of size H×W×1.

[0046] Specifically, first, the feature map F i Perform average pooling and maximum pooling in a channel dimension to obtain two H×W×1 channel descriptions; then concatenate these two channel descriptions together; then pass through a 7×7 convolution layer with Sigmoid as the activation function, and use a Softmax layer to obtain the spatial attention map Finally, the spatial attention map and feature map F i Multiplying them together gives you the scaled new features.

[0047] Step S03: Spatial attention map With the feature map F i Connect them in series to get the enhanced feature map E i , for the enhanced feature map E i Downsample to get E i ′.

[0048] Enhanced feature map E i Expressed as in, Represents matrix multiplication.

[0049] Step S04: Construct a deconvolutional neural network based on the enhanced feature map E after downsampling i ′, generate the adversarial perturbation ΔX of the video sequence X.

[0050] The deconvolutional neural network is trained according to the following class loss function:

[0051]

[0052] in: represents the cross entropy loss function, y j is the video frame x i The pixel-level binary label, Y + and Y - is a pixel with positive or negative labels, β=|Y- | / |Y + |; This is achieved by applying a Sigmoid function to the last activation layer of the deconvolutional neural network The σ(·) function in .

[0053] Deconvolutional neural networks often capture the shape of a specific category in the global context. In this case, a class loss function is designed to train the deconvolutional network for white-box attacks on single-sample video target segmentation models. Through deconvolution with class loss, activations closely related to the target class are reduced, while noise activations from other regions are effectively stimulated. Therefore, the goal of the deconvolutional neural network is to generate an adversarial perturbation ΔX = {Δx 1 ,Δx 2 ,…Δx n}.

[0054] To effectively generate adversarial perturbations, the deconvolutional neural network uses the following attention feature loss function:

[0055]

[0056] in: represents the Hadamard product, Represents video frame x i Adversarial examples, x r is a random predefined feature map (fixed during training); Represents the feature extractor F θ Extracted The feature map, F θ (x r ) indicates that the feature extractor F θ Extracted x r The feature map of ||·|| 2 represents the L2 norm.

[0057] In order to effectively generate adversarial samples, the deconvolutional neural network adopts the following total loss function:

[0058]

[0059] Where: λ represents the importance weight of controlling noise and cross entropy loss, μ represents the importance weight of the target in the video frame, ΔX = {Δx 1 ,Δx 2 ,…Δx n} represents the adversarial perturbation of the video sequence X, Δx i Represents video frame x i adversarial perturbation.

[0060] Step S05: Add the adversarial perturbation ΔX to the video sequence X to generate adversarial sample X adv :

[0061]

[0062] Where: ε is the maximum allowable adversarial perturbation threshold.

[0063] like Figure 3 Shown is a schematic diagram of the obtained adversarial samples.

[0064] Step S06: Given the initial frame mask m, the adversarial sample X adv Input into the single-sample video object segmentation network to obtain the final predicted mask.

[0065] The single-sample video target segmentation network based on a fully convolutional neural network is a matching-based segmentation method that uses the target binary mask given in the first frame to extract the target's appearance information, transfers the general semantic information to the foreground segmentation task, and then uses it to find similar objects in subsequent frames. The goal of the attention-guided adversarial attack method provided in this case is to invalidate the single-sample video target segmentation network by finding an adversarial sample that has an imperceptible difference from the single-sample video target segmentation network. This can be viewed as an optimization problem, that is, minimizing the loss function between the output of the single-sample video target segmentation network and the initial frame mask. Therefore, the attack on the single-sample video target segmentation network is expressed as:

[0066]

[0067]

[0068] Where: S θ (X adv ,T τ (X adv )) indicates the use of adversarial sample X adv Single-sample video object segmentation network for segmentation, T τ (X adv ) represents the adversarial sample X adv The target model of the target perception output in the target model, τ represents the parameter of the integrated target model appearance information encoding, and τ is determined by the adversarial sample X adv The initial frame and its given initial frame mask m are determined, and the final predicted mask is output using target perception.

[0069] In the description of the present invention, it should be noted that, unless otherwise clearly specified and limited, the terms "installed", "connected", and "connected" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be indirectly connected through an intermediate medium, or it can be the internal communication of two components. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0070] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "examples", "specific examples", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner.

[0071] The above shows and describes the basic principles, main features and advantages of the present invention. Those skilled in the art should understand that the above embodiments do not limit the present invention in any form, and any technical solution obtained by equivalent replacement or equivalent transformation falls within the protection scope of the present invention.

Claims

1. An attention-guided adversarial attack method for video object segmentation, characterized by: The steps include: (1) The video sequence X = {x1, x2, ..., x n Input feature extractor F θ , get the video frame x i The feature map F i ; Where θ represents the parameter to be learned, i is the index of the video frame, i = 1, 2, ..., n, and n is the number of video frames; (2) Construct a multi-level feature space attention module to obtain the feature map F i Spatial attention map (3) The spatial attention map With the feature map F i Connect them in series to get the enhanced feature map E i , for the enhanced feature map E i Downsample to get E i ′; (4) Construct a deconvolutional neural network based on the enhanced feature map E after downsampling i ′, produces adversarial perturbation ΔX of video sequence X; (5) Add the adversarial perturbation ΔX to the video sequence X to generate the adversarial sample X adv ; (6) Given the initial frame mask m, the adversarial sample X adv Input the single-sample video object segmentation network to obtain the final predicted mask.

2. The attention-guided adversarial attack method for video object segmentation according to claim 1, characterized in that: In step (1), the feature extractor F θ The VGG16 network in the single-sample video object segmentation model is used, and the last convolutional layer of the VGG16 network generates a video frame x i Feature map Where: H, W and C represent the feature map F i The height, width, and number of channels.

3. The attention-guided adversarial attack method for video object segmentation according to claim 1, characterized in that: In step (2), the multi-level feature space attention module first performs an operation on the feature map F i Perform average pooling and maximum pooling in a channel dimension to obtain two H×W×1 channel descriptions; The two channel descriptions are then concatenated together; then they pass through a 7×7 convolutional layer with a Sigmoid activation function and a Softmax layer to obtain a spatial attention map. Where: σ(·) represents the Sigmoid function, f 7×7 Represents a convolution operation with a filter size of 7×7; AvgPool(F i ) represents the feature map F i Average pooling, the pooling result is MaxPool(F i ) represents the feature map F i Maximum pooling, the pooling result is 4. The attention-guided adversarial attack method for video object segmentation according to claim 1, characterized in that: In step (4), the deconvolution neural network is trained according to the following class loss function: in: represents the cross entropy loss function, y j is the video frame x i The pixel-level binary label, Y + and Y - is a pixel with positive or negative labels, β=|Y - | / |Y + |; The deconvolutional neural network uses the following attention feature loss function: in: represents the Hadamard product, Represents video frame x i Adversarial examples, x r is a random predefined feature map; Represents the feature extractor F θ Extracted The feature map, F θ (x r ) indicates that the feature extractor F θ Extracted x r The feature map of , ||·||2 represents the L2 norm; The deconvolutional neural network uses the following total loss function: Where: λ represents the importance weight of controlling noise and cross entropy loss, μ represents the importance weight of the target in the video frame, ΔX={Δx1,Δx2,…Δx n } represents the adversarial perturbation of the video sequence X, Δx i Represents video frame x i adversarial perturbation.

5. The attention-guided adversarial attack method for video object segmentation according to claim 1, characterized in that: In the step (5), Where: ε is the maximum allowable adversarial perturbation threshold.

6. The attention-guided adversarial attack method for video object segmentation according to claim 1, characterized in that: In step (6), the attack on the single-sample video target segmentation network is expressed as: Where: S θ (X adv ,T τ (X adv )) indicates the use of adversarial sample X adv Single-sample video object segmentation network for segmentation, T τ (X adv ) represents the adversarial sample X adv The target model of the target perception output in the target model, τ represents the parameter of the integrated target model appearance information encoding, and τ is determined by the adversarial sample X adv The initial frame and its given initial frame mask m are determined, and the final predicted mask is output using target perception.

Images