Action detection query attack method based on temporal adversarial fusion

Abstract

The present invention discloses an action detection query attack method based on temporal adversarial fusion. The present invention first preprocesses the video into a frame sequence, inputs it into a spatiotemporal feature encoder, and obtains a spatiotemporal feature map; then decodes and obtains a target bounding box and an action category through a spatiotemporal action detection module; then obtains a temporal adversarial fusion sample through a temporal adversarial fusion module; finally, optimizes the temporal adversarial fusion sample through a temporal consistency attack module, and generates a powerful adversarial sample through a small number of queries on the model, which is used as the input of the spatiotemporal action detection model to make it misjudge the action category and detection. The present invention limits the adversarial attack to semantically related target areas through a temporal adversarial fusion module, thereby reducing the search space, and uses semantically irrelevant areas adjacent to the target area to attack the target area; optimizes the adversarial sample by aligning the attack directions of adjacent video clips through a temporal consistency attack, thereby improving the intensity of the attack on the action detection model.

Description

Technical Field

[0001] The present invention belongs to the field of computer vision technology, in particular to counterattack and defense and spatiotemporal motion detection, and relates to a motion detection query attack method based on temporal adversarial fusion. Background Art

[0002] In recent years, various video perception devices have been rapidly popularized, and a large amount of video data is generated all the time. How to accurately locate the time period and spatial position of a specific action from a large amount of video has become a challenging problem, namely the task of spatio-temporal action detection. This task aims to determine the category of actions contained in a video sequence, the start and end time of the action, and the spatial position of its associated target. It has broad application prospects in practical scenarios such as security monitoring, sports event analysis, and medical care. Due to the great success of deep learning methods in the field of computer vision, they have been widely used in the field of spatio-temporal action detection. However, studies have found that deep learning models are vulnerable, that is, adding perturbations that are difficult for the human eye to detect to the input will greatly reduce the performance of the model. These perturbations are called adversarial perturbations, and this attack is called adversarial attack. For example, in a security monitoring environment, the spatio-temporal action detection model can identify abnormal behaviors, such as illegal intrusions or emergencies, and respond quickly; but under the threat of adversarial attacks, the model may mistakenly mark normal behaviors or ignore real threats, resulting in serious security risks. To this end, an adversarial attack method is designed to attack the spatiotemporal motion detection model, aiming to evaluate the security and reliability of the model and provide a new research dimension to improve the performance of the model.

[0003] At present, adversarial attacks rely on the multi-layer structure of deep learning models, and research is concentrated in the field of image classification, while research on video attacks is relatively rare, and only a small number of researchers have explored the robustness of video classification models. In the field of image classification, the most common attack method is to use the gradient of the input image to generate an aggressive adversarial sample. For example, the Fast Gradient Sign Method (FGSM) projects the gradient into a symbolic tensor with the same dimension as the input image and takes the value of {-1, 0, 1}. Multiplying it by the hyperparameter used to control the size of the perturbation will obtain an adversarial sample with a certain degree of aggressiveness. The principle is to make the clean sample shift in the direction that maximizes the model loss function, so as to achieve the purpose of attacking the model and making it misclassify. In addition, when the gradient of the model to be attacked cannot be directly obtained, researchers have proposed a black-box attack method, which is mainly divided into migration attack and query attack. The former first generates adversarial samples on a white-box proxy model with known network structure and parameters, and then migrates them to the black-box model to be attacked for attack. The latter updates the adversarial perturbation by continuously querying the classification results returned by the model, and finally generates an aggressive adversarial sample. Since the attacker often cannot obtain all the information of the model to be attacked in reality, attacks on black-box models with unknown network structures and parameters have greater practical significance and have attracted the attention of more researchers. For example, some query attack methods use dimensionality reduction to reduce the search space, thereby improving search efficiency. In addition, many migration attack methods choose to train proxy models, and optimize the proxy models to have decision boundaries similar to those of the model to be attacked, thereby generating adversarial samples with good migration properties. In adversarial attacks against video models, some query attack methods add different color filters to consecutive frames to attack the temporal coherence of the video. There are also migration attack methods that first generate adversarial samples on the image classification model, and then attack the temporal relationship of the video by ensuring that the attack direction between adjacent frames is consistent.

[0004] The shortcomings of the above methods are mainly manifested in three aspects: 1) The existing attack methods are mainly aimed at image classification models. However, due to the dimensional difference between images and videos, the attack methods for images are often not applicable to videos and lack consideration of the time dimension; 2) The existing query attack methods have problems such as low search efficiency and weak attack strength when facing black box models due to unclear attack targets; 3) Videos contain more information than images. The existing query attack methods require multiple query overheads when facing video models, which greatly reduces the practical application value of query attacks. Based on these considerations, for practical application scenarios such as security monitoring and sports event analysis, it is urgent to design a query attack method that can not only efficiently generate adversarial samples, but also fully attack spatiotemporal action detection models and evaluate their reliability. Summary of the invention

[0005] The purpose of the present invention is to address the deficiencies of the prior art and propose an action detection query attack method based on temporal adversarial fusion. The method of the present invention constructs a temporal adversarial fusion module to extract the spatial position of the target in a video frame sequence from the video, obtain the spatiotemporal key area for temporal adversarial fusion; at the same time, it constructs a temporal consistency attack module to align the consistency of the attack direction of the coherent video clips, thereby improving the attack strength and reducing the query overhead of the black box attack.

[0006] The method of the present invention first obtains an original video data set, and then performs the following operations:

[0007] Step (1) preprocessing the video to obtain a video frame sequence, and using a two-way residual neural network to extract the video spatiotemporal feature map;

[0008] Step (2) constructs a spatiotemporal action detection model, takes the spatiotemporal feature map of the video as input, and obtains the spatial position and action category of all targets at different times;

[0009] Step (3) constructing a temporal adversarial fusion module, taking the video frame sequence, the spatial positions and action categories of all targets at different times as input, and obtaining a temporal adversarial fusion video frame;

[0010] Step (4) constructs a temporal consistency attack module, takes the temporal adversarial fusion video frame, the real action position of the video frame sequence, and the real action category as input, queries and iteratively optimizes the temporal adversarial fusion video frame, obtains the final adversarial sample, and then uses the adversarial sample to attack the spatiotemporal action detection model to cause it to misclassify and misdetect.

[0011] Further, step (1) is specifically:

[0012] (1-1) Taking N consecutive video frames as a single video segment, the original video frame sequence is divided into T sets of fast video segments 20≤N≤50; among which, It represents the tth video clip, H represents the height of the video frame, W represents the width of the video frame, 3 represents the RGB three channels, and the superscript fast represents the fast video frame relative to the slow video frame after sampling. is the field of real numbers;

[0013] (1-2) Collection of fast video clips Downsample and keep the first video frame every four frames to get a set of slow video clips in represents the tth slow video clip, and the superscript slow represents the slow video frame relative to the sampled fast video frame;

[0014] (1-3) Extracting a collection of fast video clips using a pre-trained two-way residual neural network consisting of a fast residual network and a slow residual network and slow motion video clip collection The fast and slow branches each contain five residual blocks; the fast video clip set As the input of the fast residual network, the fast feature maps of the five residual blocks are obtained and and Represent the spatiotemporal feature maps of the first, second, third, fourth and fifth residual blocks output of the t-th fast video clip respectively; the slow video clip set As the input of the slow residual network, the slow feature maps of the five residual blocks are obtained and and They represent the spatiotemporal feature maps output by the first, second, third, fourth, and fifth residual blocks of the t-th slow video clip, respectively.

[0015] Furthermore, step (2) is specifically:

[0016] (2-1) Downsample the spatiotemporal feature maps of the third and fourth fast residual blocks, retain the first frame of every four frames in the time dimension, and obtain the fast feature map obtained by downsampling the spatiotemporal feature maps of the third and fourth fast residual blocks. and

[0017] (2-2) Fast feature maps after the third and fourth downsampling And the third and fourth slow spatiotemporal feature maps By splicing along the channel dimension, we get the splicing features of the third and fourth layers, which are is the splicing feature of the third layer, is the concatenation feature of the fourth layer, concat(·) is the concatenation operation, and the superscript concat represents the concatenation feature;

[0018] (2-3). Construct a feature pyramid fusion module. The feature pyramid (Feature Pyramid Network) fusion module FPN (·) is expressed as FPN (·) = Conv2D (ReLU (Conv3D (·))); wherein Conv3D (·) is a three-dimensional convolutional layer, whose input channel number is 256, 288 or 576, the output channel number is 256, the convolution kernel size is 3×3×3, the step size is 1, and the padding is 1; ReLU (·) is a ReLU layer; Conv2D (·) is a two-dimensional convolutional layer, whose input channel number is 256, the output channel number is 256, the convolution kernel size is 1×1, the step size is 1, and the padding is 1;

[0019] (2-4). The fourth layer of the splicing feature map Input the feature pyramid fusion module to obtain the enhanced feature map of the fourth layer Enhanced feature map of the fourth layer Perform linear interpolation upsampling and pooling downsampling methods respectively to obtain the feature map after upsampling And the downsampled feature map The superscript up represents upsample, and the superscript down represents downsample;

[0020] The concatenated feature map of the third layer And the upsampled feature map Add and input into the feature pyramid fusion module to obtain the enhanced feature map of the third layer

[0021] The downsampled feature map Input the feature pyramid fusion module to obtain the enhanced feature map of the fifth layer

[0022] (2-5). Use the fully convolutional one-stage object detector (FCOS) (·) pre-trained on the large-scale dataset COCO to detect the object bounding box of the video frame sequence; the detector consists of four convolutional layers and three branches. The number of input channels of the four convolutional layers is 256, the number of output channels is 256, the convolution kernel size is 3×3, and the three branches are three convolutional layers, namely the classification branch, the center branch, and the bounding box regression branch. The number of input channels is 256, and the number of output channels is 2, 1, and 4 respectively;

[0023] The enhanced feature map of the third layer Enhanced feature map of the fourth layer Enhanced feature map of the fifth layer Input the first-order fully convolutional object detection head FCOS(·) to obtain classification feature maps of different scales Centrality feature map Bounding Box Regression Feature Map in The k+2th classification feature map representing the tth video clip, The k+2th layer centrality feature map representing the tth video clip, Represents the k+2th layer bounding box regression feature map of the tth video clip. The superscript class represents classification, the superscript center represents center, and the superscript regression represents regression.

[0024] (2-6). Decoding classification feature maps of different scales Centrality feature map Bounding Box Regression Feature Map Get the target bounding box sequence collection N t Indicates the target number of intermediate frames in the t-th video segment, represents the bounding box of the ith object in the middle frame of the tth video clip, and denote the horizontal and vertical coordinates of the upper left corner of the i-th target bounding box in the t-th video clip, respectively. denote the horizontal and vertical coordinates of the lower right corner of the i-th target bounding box in the middle frame of the t-th video clip, respectively, where β = 0 means that the bounding box does not contain a person, and β = 1 means that the bounding box contains a person;

[0025] (2-7) Construct the Region of Interest Head, which consists of the Region of Interest alignment layer and the action classification head, and convert the spatiotemporal feature map output by the fifth slow residual block Input, only keep the middle frame as the key frame, and get the key frame feature map of the fifth slow residual block Using target bounding box sequence collection Extract the spatial features of different targets and obtain the target spatial feature sequence of key frames represents the keyframe spatial features of the i-th target, H i represents the height of the keyframe spatial feature of the i-th target, W i Represents the width of the key frame spatial feature of the i-th target, and the superscript key represents the key frame;

[0026] (2-8) The target space feature sequences of key frames of different sizes Input the region of interest alignment layer and output a region of interest feature sequence with a height and width of 8×8 Represents the region of interest feature of the i-th target, and the superscript roi represents the region of interest;

[0027] Construct an action classification head consisting of a two-dimensional convolutional layer and a multi-layer perceptron layer to transform the feature sequence of the region of interest into Input the two-dimensional convolution layer to obtain the action feature sequence Used to determine the category to which the action belongs. represents the action category feature of the i-th target, and the superscript action represents the action; the number of input channels of the convolution layer is 1024, the number of output channels is 256, the convolution kernel size is 3×3, the step size is 1, and the padding is 1;

[0028] Action feature sequence After pooling, flatten into a sequence of action vectors express The expanded action vector, pool(·) represents the average pooling layer, which averages the width and height of 8×8 into 2×2, flatten(·) represents the flattening operation, 1024 is the dimension of the vector, which is obtained by multiplying the width of the pooled tensor by the height and the number of channels;

[0029] The action vector sequence Input the multi-layer perceptron layer to obtain the action category probability sequence represents the action classification probability of the i-th target in the t-th video clip. The number of input channels of the multilayer perceptron layer is 1024 and the number of output channels is N. c , N c Represents the total number of action categories.

[0030] Furthermore, step (3) is specifically:

[0031] (3-1) Construct a temporal adversarial fusion module, which consists of a bounding box union module and an adversarial fusion module;

[0032] (3-2) Construct a bounding box union module to combine the target bounding box sequence of step (2-7) β=1 is used as input and output is adversarial fusion region mask It represents the union of all the positions where the i-th target appears in the t-th video segment. 1 indicates whether the target appears at this pixel. When this channel is 0, it means that the target does not appear in this video segment. When it is 1, it means that the target appears.

[0033] (3-3). Construct an adversarial fusion module and mask the adversarial fusion area and a collection of quick video clips As input, the adversarial fusion region mask Replicate along the time dimension and channel dimension to obtain the temporal adversarial fusion mask Through the dot multiplication operation, we get the Hadamard product Represents the use of adversarial fusion region mask The tth video clip containing the ith target captured in the spatial dimension;

[0034] (3-4).Yes Taking the average along the first dimension, we get Perform confrontation fusion to obtain is the temporal adversarial fusion video clip obtained by adversarial fusion of the i-th target in the t-th video clip. The superscript adv means adversarial. - '(·) is a truncation function that limits the output to The part beyond the range is directly truncated, the truncation factor ∈=16 / 255 is used to control the output range, and the adversarial fusion factor η=0.8 is used to control the strength of adversarial fusion;

[0035] (3-5). For all targets i=1,2,...,N t Repeat steps (3-1) to (3-4) to obtain j temporal adversarial fusion video clip sequences. j indicates that the target being fused this time is the jth target.

[0036] Furthermore, step (4) is specifically:

[0037] (4-1) Construct a temporal consistency attack module, build a spatiotemporal action detection model consisting of steps (1-2) to (2-7), and obtain a real target bounding box sequence set β = 1 and the true action category probability sequence in represents the action category probability of the i-th target in the t-th video clip, and the overline represents the true label;

[0038] For j adversarial fusion video clip sequences Mix, the tth adversarial sequence fragment Random from Select from and obtain a total of m mixed adversarial fusion video clip sequences The superscript T indicates mathematical power;

[0039] (4-2). As the input of the spatiotemporal action detection model, we obtain a sequence of m bounding boxes predicted by the model. m action category probability sequences m means that this output is based on the mth adversarial fusion video clip sequence, represents the action classification probability of the i-th target in the t-th video clip based on the m-th adversarial fusion video clip sequence;

[0040] (4-3). Constructing multi-target adversarial attack score, S total =S bbx +S class -λS consist ; where λ = 0.5 is used to balance the weight of the attack consistency score; S total is the multi-target adversarial attack score Score, the subscript total represents the overall loss function; S bbx is the IoU score of target detection, the subscript bbx represents the bounding box; S class is the cross entropy loss score of action recognition, the subscript class represents classification; S consist is the attack consistency score, the subscript “consist” indicates consistency, which is used to measure the consistency of adversarial attacks in time sequence, namely:

[0041]

[0042] Where cos(·) is the cosine function, Motion(·) is the displacement function and represents the vector displacement between the bounding box position predicted by the model under attack and the true bounding box position;

[0043] (4-4). Calculate m mixed adversarial fusion video clip sequences respectively Multi-target adversarial attack scores Select the smallest κ in ascending order, κ = 10 multi-target adversarial attack scores and its corresponding hybrid adversarial fusion video clip sequence

[0044] (4-5). Randomly generate adversarial samples Δ∈{-1,0,1} T×N×H×W×3 , the mixed adversarial fusion video clip sequence Add Δ with different intensities to obtain a new round of iterative adversarial fusion video clip sequence The subscript iter indicates the number of iterations, ε indicates the size of the perturbation applied this time, ε=0,1,2,...,16, repeat steps (4-2) to (4-4) to obtain the minimum κ multi-target adversarial attack scores for the new round and its corresponding hybrid adversarial fusion video clip sequence when Or iter ≥ 50 stops iteration and outputs the minimum multi-target adversarial attack score Corresponding hybrid adversarial fusion video clip sequence

[0045] (4-6) The adversarial fusion video clip sequence obtained by the final iteration Feeding a spatiotemporal action detection model causes it to misclassify action categories and obtain incorrect object bounding box locations.

[0046] The present invention uses a temporal adversarial fusion module and a temporal consistency attack module to attack the spatiotemporal action detection model. The method has the following characteristics: 1) Through the temporal adversarial fusion module, the target subject of the action can be effectively attacked; 2) The temporal consistency attack module is used to generate adversarial samples, which improves the strength of the adversarial attack from the perspective of aligning the attack direction; 3) Using the query method to attack the black box spatiotemporal action detection model is more practical.

[0047] The present invention is aimed at the task of spatiotemporal motion detection of videos under adversarial attacks, and the beneficial effects include: 1) the attack is limited to the semantic area through the temporal adversarial fusion module, avoiding the attack on the semantically irrelevant area, and improving the search efficiency by reducing the query space; 2) the background area of ​​the adjacent frame is used to attack the target area of ​​the target frame through the adversarial fusion method, blurring the boundary between the semantic area and the non-semantic area to improve the attack intensity; 3) the adversarial sample is optimized through the temporal consistency attack module, the attack direction between adjacent video clips is aligned, and the spatiotemporal motion detection model is used to optimize the attack by relying on the temporal information, thereby improving the attack intensity. This lays the foundation for practical application scenarios such as monitoring security, video content review, and traffic safety detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Figure 1 is a flow chart of the method of the present invention. DETAILED DESCRIPTION

[0049] The present invention will be further described below in conjunction with the accompanying drawings.

[0050] like Figure 1, an action detection query attack method based on temporal adversarial fusion, first samples a given video, encodes it using a spatiotemporal feature encoder composed of a dual-path residual neural network, and obtains a spatiotemporal feature map; uses a spatiotemporal action detection module composed of a fully connected first-order target detection head and an action classification head to decode the target's bounding box and action classification from the spatiotemporal feature map; uses the temporal adversarial fusion module to expand the position where the obtained target bounding box appears in the video frame into an adversarial fusion region, and obtains the initial adversarial perturbation by averaging along the time dimension. This adversarial perturbation is adversarially fused with the adversarial fusion region to obtain a temporal adversarial fusion sample; uses the temporal consistency attack module to optimize the adversarial sample, and selects the adversarial sample with the smallest adversarial attack score by repeatedly querying the adversarial attack score of the black-box spatiotemporal action detection model to be attacked. This method uses the spatiotemporal feature encoder to extract spatiotemporal features from the fast and slow dual-path residual network, and then uses the spatiotemporal action detection module to obtain the target's bounding box and action classification. It then uses the temporal adversarial fusion module to generate temporal adversarial fusion samples, and uses the temporal consistency attack module to optimize the temporal adversarial fusion samples, and finally generates a more aggressive adversarial sample. The details are as follows:

[0051] First, obtain the original video data set, and then perform the following operations:

[0052] Step (1) preprocesses the video to obtain a video frame sequence, and uses a two-way residual neural network to extract the video spatiotemporal feature map; specifically:

[0053] (1-1) Taking N consecutive video frames as a single video segment, the original video frame sequence is divided into T sets of fast video segments 20≤N≤50; among which, It represents the tth video clip, H represents the height of the video frame, W represents the width of the video frame, 3 represents the RGB three channels, and the superscript fast represents the fast video frame relative to the slow video frame after sampling. is the field of real numbers;

[0054] (1-2) Collection of fast video clips Downsample and keep the first video frame every four frames to get a set of slow video clips in represents the tth slow video clip, and the superscript slow represents the slow video frame relative to the sampled fast video frame;

[0055] (1-3) Extracting a collection of fast video clips using a pre-trained two-way residual neural network consisting of a fast residual network and a slow residual network and slow motion video clip collection The fast and slow branches each contain five residual blocks; the fast video clip set As the input of the fast residual network, the fast feature maps of the five residual blocks are obtained and and Represent the spatiotemporal feature maps of the first, second, third, fourth and fifth residual blocks output of the t-th fast video clip respectively; the slow video clip set As the input of the slow residual network, the slow feature maps of the five residual blocks are obtained and and They represent the spatiotemporal feature maps output by the first, second, third, fourth, and fifth residual blocks of the t-th slow video clip, respectively.

[0056] Step (2) constructs a spatiotemporal action detection model, takes the spatiotemporal feature map of the video as input, and obtains the spatial position and action category of all targets at different times; specifically:

[0057] (2-1) Downsample the spatiotemporal feature maps of the third and fourth fast residual blocks, retain the first frame of every four frames in the time dimension, and obtain the fast feature map obtained by downsampling the spatiotemporal feature maps of the third and fourth fast residual blocks. and

[0058] (2-2) Fast feature maps after the third and fourth downsampling And the third and fourth slow spatiotemporal feature maps By splicing along the channel dimension, we get the splicing features of the third and fourth layers, which are is the splicing feature of the third layer, is the concatenation feature of the fourth layer, concat(·) is the concatenation operation, and the superscript concat represents the concatenation feature;

[0059] (2-3). Construct a feature pyramid fusion module. The feature pyramid fusion module FPN(·) is expressed as FPN(·)=Conv2D(ReLU(Conv3D(·))); wherein Conv3D(·) is a three-dimensional convolution layer, whose input channel number is 256, 288 or 576, whose output channel number is 256, whose convolution kernel size is 3×3×3, whose step size is 1, and whose padding is 1; ReLU(·) is a ReLU layer; Conv2D(·) is a two-dimensional convolution layer, whose input channel number is 256, whose output channel number is 256, whose convolution kernel size is 1×1, whose step size is 1, and whose padding is 1;

[0060] (2-4). The fourth layer of the splicing feature map Input the feature pyramid fusion module to obtain the enhanced feature map of the fourth layer Enhanced feature map of the fourth layer Perform linear interpolation upsampling and pooling downsampling methods respectively to obtain the feature map after upsampling And the downsampled feature map The superscript up indicates upsampling, and the superscript down indicates downsampling;

[0061] The concatenated feature map of the third layer And the upsampled feature map Add and input into the feature pyramid fusion module to obtain the enhanced feature map of the third layer

[0062] The downsampled feature map Input the feature pyramid fusion module to obtain the enhanced feature map of the fifth layer

[0063] (2-5). Use the fully convolutional single-stage object detector FCOS(·) pre-trained on the large-scale dataset COCO to detect the object bounding box of the video frame sequence; the detector consists of four convolutional layers and three branches. The number of input channels of the four convolutional layers is 256, the number of output channels is 256, and the convolution kernel size is 3×3. The three branches are three convolutional layers, namely the classification branch, the center branch, and the bounding box regression branch. The number of input channels is 256, and the number of output channels is 2, 1, and 4 respectively;

[0064] The enhanced feature map of the third layer Enhanced feature map of the fourth layer Enhanced feature map of the fifth layer Input the first-order fully convolutional object detection head FCOS(·) to obtain classification feature maps of different scales Centrality feature map Bounding Box Regression Feature Map in The k+2th classification feature map representing the tth video clip, The k+2th layer centrality feature map representing the tth video clip, Represents the k+2th layer bounding box regression feature map of the tth video clip. The superscript class represents classification, the superscript center represents center, and the superscript regression represents regression.

[0065] (2-6). Decoding classification feature maps of different scales Centrality feature map Bounding Box Regression Feature Map Get the target bounding box sequence collection N t Indicates the target number of intermediate frames in the t-th video segment, represents the bounding box of the ith object in the middle frame of the tth video clip, and denote the horizontal and vertical coordinates of the upper left corner of the i-th target bounding box in the t-th video clip, respectively. denote the horizontal and vertical coordinates of the lower right corner of the i-th target bounding box in the middle frame of the t-th video clip, respectively, where β = 0 means that the bounding box does not contain a person, and β = 1 means that the bounding box contains a person;

[0066] (2-7). Construct the region of interest head, which consists of the region of interest alignment layer and the action classification head, and convert the spatiotemporal feature map output by the fifth slow residual block Input, only keep the middle frame as the key frame, and get the key frame feature map of the fifth slow residual block Using target bounding box sequence collection Extract the spatial features of different targets and obtain the target spatial feature sequence of key frames represents the keyframe spatial features of the i-th target, H i represents the height of the keyframe spatial feature of the i-th target, W i Represents the width of the key frame spatial feature of the i-th target, and the superscript key represents the key frame;

[0067] (2-8) The target space feature sequences of key frames of different sizes Input the region of interest alignment layer and output a region of interest feature sequence with a height and width of 8×8 It represents the region of interest feature of the i-th target, and the superscript roi represents the region of interest;

[0068] Construct an action classification head consisting of a two-dimensional convolutional layer and a multi-layer perceptron layer to transform the feature sequence of the region of interest Input the two-dimensional convolution layer to obtain the action feature sequence Used to determine the category to which the action belongs. represents the action category feature of the i-th target, and the superscript action represents the action; the number of input channels of the convolution layer is 1024, the number of output channels is 256, the convolution kernel size is 3×3, the step size is 1, and the padding is 1;

[0069] Action feature sequence After pooling, flatten into a sequence of action vectors express The expanded action vector, pool(·) represents the average pooling layer, which averages the width and height of 8×8 into 2×2, flatten(·) represents the flattening operation, 1024 is the dimension of the vector, which is obtained by multiplying the width of the pooled tensor by the height and the number of channels;

[0070] The action vector sequence Input the multi-layer perceptron layer to obtain the action category probability sequence represents the action classification probability of the i-th target in the t-th video clip. The number of input channels of the multilayer perceptron layer is 1024 and the number of output channels is N. c , N c Represents the total number of action categories.

[0071] Step (3) constructs a temporal adversarial fusion module, takes the video frame sequence, the spatial positions of all targets at different times and the action categories as input, and obtains a temporal adversarial fusion video frame; specifically:

[0072] (3-1) Construct a temporal adversarial fusion module, which consists of a bounding box union module and an adversarial fusion module;

[0073] (3-2) Construct a bounding box union module to combine the target bounding box sequence of step (2-7) β=1 is used as input and output is adversarial fusion region mask It represents the union of all the positions where the i-th target appears in the t-th video segment. 1 indicates whether the target appears at this pixel. When this channel is 0, it means that the target does not appear in this video segment. When it is 1, it means that the target appears.

[0074] (3-3). Construct an adversarial fusion module and mask the adversarial fusion area and a collection of quick video clips As input, the adversarial fusion region mask Replicate along the time dimension and channel dimension to obtain the temporal adversarial fusion mask Through the dot multiplication operation, we get the Hadamard product Represents the use of adversarial fusion region mask The tth video clip containing the ith target captured in the spatial dimension;

[0075] (3-4).Yes Taking the average along the first dimension, we get Perform confrontation fusion to obtain is the temporal adversarial fusion video clip obtained by adversarial fusion of the i-th target in the t-th video clip. The superscript adv represents adversarial. - '(·) is a truncation function that limits the output to The part beyond the range is directly truncated, the truncation factor ∈=16 / 255 is used to control the output range, and the adversarial fusion factor η=0.8 is used to control the strength of adversarial fusion;

[0076] (3-5). For all targets i=1,2,...,N t Repeat steps (3-1) to (3-4) to obtain j temporal adversarial fusion video clip sequences. j indicates that the target being fused this time is the jth target.

[0077] Step (4) constructs a temporal consistency attack module, takes the temporal adversarial fusion video frame, the real action position of the video frame sequence, and the real action category as input, queries and iteratively optimizes the temporal adversarial fusion video frame, obtains the final adversarial sample, and then uses the adversarial sample to attack the spatiotemporal action detection model to cause it to misclassify and misdetect; specifically:

[0078] (4-1) Construct a temporal consistency attack module, build a spatiotemporal action detection model consisting of steps (1-2) to (2-7), and obtain a real target bounding box sequence set β = 1 and the true action category probability sequence in represents the action category probability of the i-th target in the t-th video clip, and the overline represents the true label;

[0079] For j adversarial fusion video clip sequences Mix, the tth adversarial sequence fragment Random from Select from and obtain a total of m mixed adversarial fusion video clip sequences The superscript T indicates mathematical power;

[0080] (4-2). As the input of the spatiotemporal action detection model, we obtain a sequence of m bounding boxes predicted by the model. m action category probability sequences m means that this output is based on the mth adversarial fusion video clip sequence, represents the action classification probability of the i-th target in the t-th video clip based on the m-th adversarial fusion video clip sequence;

[0081] (4-3). Constructing multi-target adversarial attack score, S total =S bbx +S class-λS consist ; where λ = 0.5 is used to balance the weight of the attack consistency score; S total is the multi-target adversarial attack score, the subscript total represents the overall loss function; S bbx is the IoU score of target detection, the subscript bbx represents the bounding box; S class is the cross entropy loss score of action recognition, the subscript class indicates classification; S consist is the attack consistency score, the subscript “consist” indicates consistency, which is used to measure the consistency of adversarial attacks in time sequence, namely: Where cos(·) is the cosine function, Motion(·) is the displacement function and represents the vector displacement between the bounding box position predicted by the model under attack and the true bounding box position;

[0082] (4-4). Calculate m mixed adversarial fusion video clip sequences respectively Multi-target adversarial attack scores Select the smallest κ in ascending order, κ = 10 multi-target adversarial attack scores and its corresponding hybrid adversarial fusion video clip sequence

[0083] (4-5). Randomly generate adversarial samples Δ∈{-1,0,1} T×N×H×W×3 , the mixed adversarial fusion video clip sequence Add Δ with different intensities to obtain a new round of iterative adversarial fusion video clip sequence The subscript iter indicates the number of iterations, ε indicates the size of the perturbation applied this time, ε=0,1,2,...,16, repeat steps (4-2)~(4-4) to obtain the minimum κ multi-target adversarial attack scores for a new round and its corresponding hybrid adversarial fusion video clip sequence when Or iter ≥ 50 stops iteration and outputs the minimum multi-target adversarial attack score Corresponding hybrid adversarial fusion video clip sequence

[0084] (4-6) The adversarial fusion video clip sequence obtained by the final iteration Feeding a spatiotemporal action detection model causes it to misclassify action categories and obtain incorrect object bounding box locations.

[0085] The contents described in this embodiment are merely an enumeration of the implementation forms of the inventive concept. The protection scope of the present invention should not be regarded as being limited to the specific forms described in the embodiment. The protection scope of the present invention also extends to the equivalent technical means that can be conceived by those skilled in the art based on the inventive concept.

Claims

1. The action detection query attack method based on temporal adversarial fusion is characterized by: First, obtain the original video data set, and then perform the following operations: Step (1) preprocessing the video to obtain a video clip set, the video clip set includes a fast video clip set and a slow video clip set, and using a two-way residual neural network to extract a spatiotemporal feature map of the video clip set; Step (2) using the spatiotemporal action detection model and the spatiotemporal feature map as input, to obtain a set of target bounding box sequences and action category probability sequences for all targets in different video clips; Step (3) constructing a temporal adversarial fusion module, which is composed of a bounding box union module and an adversarial fusion module; using the target bounding box sequence set as the input of the bounding box union module to obtain an adversarial fusion region mask; using the adversarial fusion region mask and the fast video clip set as the input of the adversarial fusion module to obtain a temporal adversarial fusion video clip sequence; Step (4) constructs a temporal consistency attack module, takes the temporal adversarial fusion video clip sequence, the real target bounding box sequence set, and the real action category probability sequence as input, obtains the final adversarial sample by querying and iteratively optimizing the temporal adversarial fusion video clip sequence, and then uses the final adversarial sample to attack the spatiotemporal action detection model, causing it to incorrectly determine the action category and obtain the wrong target bounding box position.

2. The action detection query attack method based on temporal adversarial fusion according to claim 1, characterized in that: Step (1) specifically includes: (1-1) Taking N consecutive video frames as a single video segment, the original video frame sequence is divided into T sets of fast video segments 20≤N≤50; among which, It represents the t-th fast video clip, H represents the height of the video frame, W represents the width of the video frame, 3 represents the RGB three channels, and the superscript fast represents the fast video frame relative to the slow video frame after sampling. is the field of real numbers; (1-2) Collection of fast video clips Downsample and keep the first video frame every four frames to get a set of slow video clips in, represents the tth slow video clip, and the superscript slow represents the slow video frame relative to the sampled fast video frame; (1-3) Extracting a collection of fast video clips using a pre-trained two-way residual neural network consisting of a fast residual network and a slow residual network and slow motion video clip collection The fast and slow branches each contain five residual blocks; the fast video clip set As the input of the fast residual network, the spatiotemporal feature maps of five fast residual blocks are obtained and and Respectively represent the spatiotemporal feature maps output by the first, second, third, fourth and fifth fast residual blocks of the t-th fast video clip; the slow video clip set As the input of the slow residual network, the feature maps of five slow residual blocks are obtained and and Respectively represent the spatiotemporal feature maps output by the first, second, third, fourth, and fifth slow residual blocks of the t-th slow video clip.

3. The action detection query attack method based on temporal adversarial fusion as claimed in claim 2, characterized in that: Step (2) is specifically: (2-1) Downsample the spatiotemporal feature maps output by the third and fourth fast residual blocks, retain the first frame of every four frames in the time dimension, and obtain the fast feature map obtained by downsampling the spatiotemporal feature maps output by the third and fourth fast residual blocks. and (2-2) Fast feature maps after the third and fourth downsampling And the spatiotemporal feature maps output by the third and fourth slow residual blocks By splicing along the channel dimension, we get the spliced ​​feature maps of the third and fourth layers, which are is the concatenated feature map of the third layer, is the concatenated feature map of the fourth layer, concat(·) is the concatenation operation, and the superscript concat represents the concatenated feature; (2-3). Construct a feature pyramid fusion module. The feature pyramid fusion module FPN(·) is expressed as FPN(·)=Conv2D(ReLU(Conv3D(·))); wherein Conv3D(·) is a three-dimensional convolution layer, whose input channel number is 256, 288 or 576, whose output channel number is 256, whose convolution kernel size is 3×3×3, whose step size is 1, and whose padding is 1; ReLU(·) is a ReLU layer; Conv2D(·) is a two-dimensional convolution layer, whose input channel number is 256, whose output channel number is 256, whose convolution kernel size is 1×1, whose step size is 1, and whose padding is 1; (2-4). The fourth layer of the splicing feature map Input the feature pyramid fusion module to obtain the enhanced feature map of the fourth layer Enhanced feature map of the fourth layer Perform linear interpolation upsampling and pooling downsampling methods respectively to obtain the upsampled feature map And the downsampled feature map The superscript up indicates upsampling, and the superscript down indicates downsampling; The concatenated feature map of the third layer And the upsampled feature map Add and input into the feature pyramid fusion module to obtain the enhanced feature map of the third layer The downsampled feature map Input the feature pyramid fusion module to obtain the enhanced feature map of the fifth layer (2-5). Use the fully convolutional single-stage object detector FCOS(·) pre-trained on the large-scale dataset COCO to detect the object bounding box of the video frame sequence; the detector consists of four convolutional layers and three branches. The number of input channels of the four convolutional layers is 256, the number of output channels is 256, and the convolution kernel size is 3×3. The three branches are three convolutional layers, namely the classification branch, the center branch, and the bounding box regression branch. The number of input channels is 256, and the number of output channels is 2, 1, and 4 respectively; The enhanced feature map of the third layer Enhanced feature map of the fourth layer Enhanced feature map of the fifth layer Input the first-order fully convolutional target detection head to obtain classification feature maps of different scales Centrality feature map Bounding Box Regression Feature Map in The k+2th classification feature map representing the tth video clip, The k+2th layer centrality feature map representing the tth video clip, Represents the k+2th layer bounding box regression feature map of the tth video clip. The superscript class represents classification, the superscript center represents center, and the superscript regression represents regression. (2-6). Decoding classification feature maps of different scales Centrality feature map Bounding Box Regression Feature Map Get the target bounding box sequence collection N t Indicates the target number of intermediate frames in the t-th video segment, represents the bounding box of the ith object in the middle frame of the tth video clip, and They represent the horizontal and vertical coordinates of the upper left corner of the i-th target bounding box in the middle frame of the t-th video clip, respectively. denote the horizontal and vertical coordinates of the lower right corner of the i-th target bounding box in the middle frame of the t-th video clip, respectively, where β = 0 means that the bounding box does not contain a person, and β = 1 means that the bounding box contains a person; (2-7). Construct the region of interest head, which consists of the region of interest alignment layer and the action classification head, and convert the spatiotemporal feature map output by the fifth slow residual block Input, only keep the middle frame as the key frame, and get the key frame feature map of the fifth slow residual block Using target bounding box sequence collection Extract the spatial features of different targets and obtain the target spatial feature sequence of key frames represents the key frame spatial features of the i-th target, H i represents the height of the keyframe spatial feature of the i-th target, W i Represents the width of the key frame spatial feature of the i-th target, and the superscript key represents the key frame; (2-8) The target space feature sequences of key frames of different sizes Input the region of interest alignment layer and output a region of interest feature sequence with a height and width of 8×8 It represents the region of interest feature of the i-th target, and the superscript roi represents the region of interest; Construct an action classification head consisting of a two-dimensional convolutional layer and a multi-layer perceptron layer to transform the feature sequence of the region of interest Input the two-dimensional convolution layer to obtain the action feature sequence Used to determine the category to which the action belongs. represents the action category feature of the i-th target, and the superscript action represents the action; the number of input channels of the convolution layer is 1024, the number of output channels is 256, the convolution kernel size is 3×3, the step size is 1, and the padding is 1; Action feature sequence After pooling, flatten into a sequence of action vectors express The expanded action vector, pool(·) represents the average pooling layer, which averages the width and height of 8×8 into 2×2, flatten(·) represents the flattening operation, 1024 is the dimension of the vector, which is obtained by multiplying the width of the pooled tensor by the height and the number of channels; The action vector sequence Input the multi-layer perceptron layer to obtain the action category probability sequence represents the action classification probability of the i-th target in the t-th video clip. The number of input channels of the multilayer perceptron layer is 1024 and the number of output channels is N. c , N c Represents the total number of action categories.

4. The action detection query attack method based on temporal adversarial fusion as claimed in claim 3, characterized in that: Step (3) is specifically: (3-1) Construct a temporal adversarial fusion module, which consists of a bounding box union module and an adversarial fusion module; (3-2) Construct a bounding box union module to combine the target bounding box sequence of step (2-6) As input, output is adversarial fusion region mask It represents the union of all positions where the i-th target appears in the t-th video clip. 1 means that the target appears in this video clip. When this channel is 0, it means that the target does not appear in this video clip. (3-3). Construct an adversarial fusion module and mask the adversarial fusion area and a collection of quick video clips As input, the adversarial fusion region mask Replicate along the time dimension and channel dimension to obtain the temporal adversarial fusion mask Through the dot multiplication operation, we get the Hadamard product Represents the use of adversarial fusion region mask The tth video clip containing the ith target captured in the spatial dimension; (3-4).Yes Taking the average along the first dimension, we get Perform confrontation fusion to obtain is the temporal adversarial fusion video clip obtained by adversarial fusion of the i-th target in the t-th video clip. The superscript adv represents adversarial, and Clip-∈,∈(·) is the truncation function, which limits the output to The part beyond the range is directly truncated, and the truncation factor ∈=16 / 255 is used to control the output range, and the adversarial fusion factor η=0.8 is used to control the strength of adversarial fusion; (3-5). For all targets i=1,2,...,N t Repeat steps (3-1) to (3-4) to obtain j temporal adversarial fusion video clip sequences. in represents the adversarial sequence segment of the t-th video segment.

5. The action detection query attack method based on temporal adversarial fusion as claimed in claim 4, characterized in that: The details of step (4) are: (4-1) Construct a temporal consistency attack module, build a spatiotemporal action detection model consisting of steps (1-2) to (2-7), and obtain a real target bounding box sequence set and the true action category probability sequence in represents the true action category probability of the i-th target in the t-th video clip, and the overline represents the true label; For j temporal adversarial fusion video clip sequences Mix the adversarial sequence fragment of the tth video clip Random from Select from and obtain a total of m mixed adversarial fusion video clip sequences The superscript T indicates mathematical power; (4-2). As the input of the spatiotemporal action detection model, we obtain a sequence of m bounding boxes predicted by the model. m action category probability sequences m means that this output is based on the mth adversarial fusion video clip sequence, represents the action classification probability of the i-th target in the t-th video clip based on the m-th adversarial fusion video clip sequence; (4-3). Constructing multi-target adversarial attack score, S total =S bbx +S class -λS consist ; where λ = 0.5 is used to balance the weight of the attack consistency score; S total is the multi-target adversarial attack score, the subscript total represents the overall loss function; S bbx is the IoU score of target detection, the subscript bbx represents the bounding box; S class is the cross entropy loss score of action recognition, the subscript class indicates classification; S consist is the attack consistency score, the subscript “consist” indicates consistency, which is used to measure the consistency of adversarial attacks in time sequence, namely: Where cos(·) is the cosine function, Motion(·) is the displacement function and represents the vector displacement between the bounding box position predicted by the model under attack and the true bounding box position; (4-4). Calculate m mixed adversarial fusion video clip sequences respectively Multi-target adversarial attack scores Select the smallest k multi-target adversarial attack scores in ascending order and its corresponding hybrid adversarial fusion video clip sequence Where, κ = 10, d = 1, 2, ..., 10; (4-5). Randomly generate adversarial samples Δ∈{-1,0,1} T×N×H×W×3 , the mixed adversarial fusion video clip sequence Add Δ with different intensities to obtain a new round of iterative adversarial fusion video clip sequence The subscript iter indicates the number of iterations, ε indicates the size of the perturbation applied this time, ε = 0, 1, 2, ..., 16, and steps (4-2) to (4-4) are repeated to obtain the minimum κ multi-target adversarial attack scores for a new round. and its corresponding hybrid adversarial fusion video clip sequence Where d=1,2,...,10; when Or iter ≥ 50 stops iteration and outputs the minimum multi-target adversarial attack score Corresponding hybrid adversarial fusion video clip sequence (4-6) The hybrid adversarial fusion video clip sequence obtained by the final iteration As the final adversarial sample, the spatiotemporal action detection model is input, causing it to incorrectly determine the action category and obtain the wrong target bounding box position.

Images