Action recognition defense method, device, equipment and storage medium based on multi-source adversarial training of spatial and frequency domain streams

Abstract

Action recognition defense method, device, equipment and storage medium based on multi-source adversarial training of spatial-frequency domain streams. First, the original video is preprocessed to obtain the original video sample; through the spatial-frequency domain stream mask generation module, the input spatial domain and input frequency domain mask of the original video sample are used as optimization targets using a genetic algorithm to obtain the spatial-frequency domain stream mask; then, through the spatial-frequency domain stream adversarial sample generation module, the spatial-frequency domain stream mask and the original video sample are input, and the projected gradient descent algorithm is used to generate the video adversarial sample; then, through the multi-source adversarial training module, the video adversarial sample and the original video sample are input to balance the data distribution between the two, and an action recognition model with strong adversarial robustness is obtained. The present invention not only generates adversarial samples based on the key areas of the video to learn the data distribution of the real adversarial samples, but also alleviates the problem of data distribution differences between the adversarial samples and the original video samples, thereby improving the adversarial robustness of the action recognition model.

Description

Technical Field

[0001] The present invention belongs to the field of adversarial attack and defense in video analysis, and specifically relates to an action recognition defense method, device, equipment and storage medium based on multi-source adversarial training of spatial and frequency domain streams. Background Art

[0002] In modern society, video surveillance equipment and personal cameras have been widely deployed in various daily scenarios - from fitness records captured by smartphones to security systems in public places such as supermarkets and streets. These video data carry rich information about human behavior (such as running, waving, falling, etc.). Action recognition, as an important task in the field of computer vision, aims to enable computers to simulate human visual understanding capabilities. By analyzing continuous frame sequences in videos, it automatically identifies the specific actions performed by individuals within a specific time period (such as "shooting" or "dribbling" in a "playing basketball" scene). This technology is widely used in key scenarios such as smart home interaction, public safety monitoring, and emergency warning. It is of great significance to improving the level of human-computer interaction intelligence and social security prevention and control capabilities. Current research has found that the deep learning model used in action recognition models is extremely vulnerable to adversarial attacks. Adversarial attack refers to the attacker adding artificially designed tiny adversarial perturbations (noise of the same size as the input) to the input data (images, videos or other types of data) to generate adversarial samples, and then inputting the adversarial samples into the model to be attacked (also called the victim model or target model) to make it recognize incorrectly, such as Figure 1 However, the current mainstream defense methods against adversarial attacks are mainly applied in the field of image classification, and there is a lack of research on defense methods in the video field. With the increasing popularity of self-media, since videos contain temporal information, attacks are more complex and closer to real-world application scenarios. Therefore, there is an urgent need to develop defense methods for action recognition models against video adversarial samples to improve the adversarial robustness of action recognition models.

[0003] Currently, most action recognition methods are based on convolutional methods and ViViT (Video Vision Transformer)-based methods. Among convolutional methods, early work used a "2D+1D" paradigm, applying 2D convolution to each video frame to extract spatial features, followed by a "1D" module to integrate spatial and temporal features. Because 3D convolution is more suitable for processing video data, researchers proposed inflated 3D convolutional neural networks (I3D), which encode spatiotemporal features by expanding the 2D convolution kernel to 3D. Inspired by the rhythmic changes in human vision, researchers proposed the SlowFast Network, which uses the Slow network branch to capture motion information at fine temporal resolution and the Fast network branch to capture rapidly changing spatial semantic information. Among ViViT-based methods, researchers proposed the TimeSformer Network, which replaces traditional convolution operations with a 3D self-attention mechanism, eliminating the inductive bias of convolution operations and capturing long-range dependencies in the video. In order to better handle subtle changes in the temporal dimension, researchers proposed the Motionformer network, which uses a trajectory attention mechanism to establish trajectory connections between different video frames, which is conducive to processing videos with motion changes over a long span of time.

[0004] The aforementioned video action recognition methods fail to consider the robustness of action recognition models to video adversarial examples, specifically the model's vulnerability to adversarial attack scenarios. Current mainstream defense methods utilize adversarial training (AT). This method first uses the original input video to artificially generate video adversarial examples, and then uses both the original input video and the video adversarial examples to train the action recognition model. Among these methods, the projected gradient descent (PGD) algorithm is primarily used to generate adversarial examples. This method iteratively adds adversarial perturbations to clean samples, shifting the adversarial examples along the gradient ascent direction. The updated variables are then projected onto the constrained region to generate the adversarial examples.

[0005] Existing adversarial training methods primarily focus on image classification, lacking research specifically targeting video. In image classification, researchers have proposed a robustness-accuracy balance method to balance adversarial robustness with natural accuracy. This method designs the robust error as the sum of the natural error and the boundary error, and devises a loss function that explicitly balances accuracy and robustness, achieving a controllable trade-off. In the field of action recognition, researchers have proposed an adaptive adversarial training method that seeks an adaptive step size during training and combines it with a generative adversarial network to improve adversarial robustness. However, this method fails to account for temporal redundancy in video.

[0006] The shortcomings of the above-mentioned adversarial training defense methods are mainly manifested in the following three aspects: (1) There is a distribution difference between clean samples and adversarial samples: Current research shows that there is a large difference between clean samples and adversarial samples in the adversarial defense scenario. Compared with image data, the distribution difference between clean samples and adversarial samples becomes more obvious with the increase of the time series dimension. The limited knowledge capacity of the trained model makes it difficult to force fit the two data distributions, resulting in difficulty in balancing the standard accuracy and adversarial robustness of clean samples; (2) Redundant perturbations hinder the adversarial training effect: Compared with image data, adjacent video frames contain a lot of redundant information. Since the model pays different attention to the foreground and background areas of the video data, the adversarial samples generated based on redundant information irrelevant to the video content are less aggressive (research shows that under the condition of a fixed perturbation threshold, the more aggressive the adversarial samples are, the more conducive they are to improving the adversarial robustness of the model in the adversarial training stage). As a result, it is difficult for the model to learn the true adversarial sample data distribution in the adversarial training stage. Based on the above considerations, there is an urgent need to design a defense method and device specifically for action recognition models, an action recognition defense method that can reduce redundant adversarial perturbations to learn the true adversarial sample distribution and balance the data distribution of clean samples and adversarial samples. Summary of the Invention

[0007] The purpose of the present invention is to address the shortcomings of the existing technology and provide an action recognition defense method based on multi-source adversarial training of spatial and frequency domain streams, which can not only reduce redundant adversarial perturbations to learn the real adversarial sample data distribution, but also balance the clean sample data distribution and the adversarial sample data distribution, thereby improving the adversarial robustness of the action recognition model and maintaining a high recognition rate when facing adversarial attacks.

[0008] The action recognition defense method based on multi-source adversarial training of spatial and frequency domain streams includes the following steps:

[0009] Step (1) obtaining action recognition video data as the original video sample and the action recognition model to be defended, replicating the model to be defended three times and initializing the weights and naming them as robust model, clean model and adversarial model respectively;

[0010] Step (2) construct a spatial-frequency domain flow mask generation module, the input is the original video sample and the robust model, and the output is the spatial-frequency domain flow mask;

[0011] Step (3) construct a spatial-frequency domain flow adversarial sample generation module, the input is the original video sample and the spatial-frequency domain flow mask, and the output is the video adversarial sample;

[0012] Step (4) constructs a multi-source adversarial training module, the input of which is the original video sample, the video adversarial sample, the robust model, the clean model and the adversarial model, and the output is the robust model;

[0013] Step (5) iteratively optimizes the action recognition defense model composed of the spatial-frequency domain flow mask generation module, the spatial-frequency domain flow adversarial sample generation module and the multi-source adversarial training module, and uses the final output robust model as the action recognition defense model.

[0014] Preferably, in step (1), obtaining the original video sample comprises the following steps: uniformly sampling the original video in the action recognition dataset to obtain the original video sample; and the action recognition model to be defended is the SlowFast model. The above model is a widely used action recognition model. The present invention improves the above model to reflect its adaptability to application in actual scenarios.

[0015] Preferably, in step (2), the spatial domain frequency domain flow mask generation module is composed of an input spatial domain mask generation module and a frequency domain mask generation module; the spatial domain frequency domain flow mask is composed of an input spatial domain mask set and a frequency domain mask set; this preferred solution provides a specific solution for step (2);

[0016] The processing of the input spatial domain mask generation module includes the following steps:

[0017] Input the original video sample into the foreground-background separation model to obtain the foreground and background parts.

[0018] Randomly generate a plurality of input space mask sets consisting of input space domain masks in the background part, each of the input space masks consists of a plurality of rectangular mask blocks of the same size and with a pixel value of 0;

[0019] Inputting the original video sample and the input spatial mask set into the mask set selection method to obtain the final input spatial domain mask set;

[0020] The processing process of the frequency domain mask generation module includes the following steps:

[0021] Perform discrete cosine transform on the original video sample to obtain the low-frequency part and the high-frequency part.

[0022] Randomly generate a frequency domain mask set consisting of frequency domain masks in the low-frequency part, each of the frequency domain masks consists of a number of rectangular mask blocks with the same size and pixel value of 0;

[0023] Inputting the original video sample and the frequency domain mask set into the mask set selection method to obtain the final frequency domain mask set;

[0024] Preferably, the mask set selection method comprises the following steps:

[0025] S1 performs Hadamard product on the original video sample and each of the mask sets to obtain a mask video sample corresponding to each of the mask sets;

[0026] Each masked video sample is input into the robust model, and the masked adversarial sample corresponding to each masked video sample is obtained using the fast gradient sign method, wherein the fast gradient sign method includes: perturbing the gradient direction of the input sample along the loss function;

[0027] Input all masked adversarial samples into the clean model to obtain the mask set score corresponding to each masked adversarial sample; each mask set corresponds to a mask set score;

[0028] S2 takes out the mask set with the highest mask set score from the input mask set, takes out a part of the remaining mask sets, randomly selects two mask sets for cross fusion, and obtains multiple cross fusion masks.

[0029] Then, the mask set with the highest mask set score is cross-fused with each cross-fusion mask to obtain the first part of the child mask set;

[0030] The cross-fusion operation means that if there is a mask block at the same position in both mask sets or no mask block exists, the pixel value at the same position is set to 1;

[0031] If only one of the two mask sets has a mask block at the same position, and the other mask set does not have a mask block at the same position, the pixel value at the same position is set to 1 with a preset probability; the preset probability in S2 is preferably 50%.

[0032] S3 takes out another part from the remaining mask set in step S2, performs a mutation operation on the mask set, and obtains a second part of the child mask set. The mutation operation means that the pixel value in a part of the mask set changes to 0 or 1 with a preset probability; the preset probability in S3 is preferably 30%.

[0033] The sub-mask sets of the first and second parts are respectively Hadamard-producted with the original video samples, and input into the clean model to obtain the sub-mask set scores corresponding to each sub-mask set, and the sub-mask set with the highest sub-mask set score is taken as the final mask set.

[0034] This preferred solution provides a specific solution for the mask set selection method.

[0035] Preferably, in step (3), the spatial-frequency domain flow mask is composed of the final input spatial-domain mask set and the final frequency-domain mask set; and the processing process of the spatial-frequency domain flow adversarial sample generation module includes the following steps:

[0036] The original video sample and the spatial-frequency domain flow mask are input into the robust model, and the video adversarial sample is generated using the iterative fast gradient sign method.

[0037] The iterative fast gradient sign method refers to adding adversarial perturbations to the input samples in each iteration, and recalculating the gradient based on the current adversarial perturbation sample at each step. When the adversarial perturbation threshold is exceeded, a clipping operation is performed to ensure that the adversarial perturbation does not exceed the perturbation threshold.

[0038] Preferably, in step (4), the processing of the multi-source adversarial training module includes the following steps:

[0039] Input the original video sample into the clean model and get the output of the clean model;

[0040] Input the video adversarial sample obtained in step (3) into the adversarial model to obtain the output of the adversarial model;

[0041] Input the original video sample and the video adversarial sample into the robust model respectively, and obtain the original video output and adversarial sample output of the robust model respectively;

[0042] Based on the output of the adversarial model and the original video output of the robust model, the clean model loss is constructed and optimized;

[0043] Based on the output of the clean model and the adversarial sample output of the robust model, the loss of the adversarial model is constructed and the adversarial model is optimized;

[0044] Based on the output of the adversarial model and the output of the clean model, the loss of the robust model is constructed and the robust model is optimized.

[0045] The present invention also provides an action recognition defense device based on multi-source adversarial training of spatial and frequency domain streams, the device comprising:

[0046] The acquisition unit is used to: acquire action recognition video data as original video samples and the action recognition model to be defended,

[0047] The processing unit is used to: copy the model to be defended three times and randomly initialize the weights and name them as a robust model, a clean model and an adversarial model respectively; construct a spatial-frequency-domain flow mask generation module, which takes the original video sample and the robust model as input and outputs the spatial-frequency-domain flow mask; construct a spatial-frequency-domain flow adversarial sample generation module, which takes the original video sample and the spatial-frequency-domain flow mask as input and outputs the video adversarial sample; construct a multi-source adversarial training module, which takes the original video sample, the video adversarial sample, the robust model, the clean model and the adversarial model as input and outputs the robust model; iteratively optimize the action recognition defense model composed of the spatial-frequency-domain flow mask generation module, the spatial-frequency-domain flow adversarial sample generation module and the multi-source adversarial training module, and use the final output robust model as the action recognition defense model;

[0048] The detection unit is used to perform action recognition defense using the action recognition defense model.

[0049] The present invention also provides an action recognition defense device based on multi-source adversarial training of spatial and frequency domain streams. The device includes: a memory, a processor, and a computer program stored on the memory and runnable on the processor. The computer program is configured to implement the steps of the action recognition defense method based on multi-source adversarial training of spatial and frequency domain streams.

[0050] The present invention also provides a storage medium, which is a computer-readable storage medium. A computer program is stored on the storage medium. When the computer program is executed by a processor, the steps of the action recognition defense method based on multi-source adversarial training of spatial and frequency domain streams are implemented.

[0051] The present invention proposes an action recognition defense method based on spatial-frequency domain multi-source adversarial training. This method has the following characteristics: 1) Utilizing the spatial-frequency domain flow mask generation module and the spatial-frequency domain flow adversarial sample generation module, the key areas of the video input spatial domain and input frequency domain are located, the spatial-frequency domain flow mask is obtained, and strong adversarial samples are generated based on the spatial-frequency domain flow mask; 2) Utilizing the multi-source adversarial training module, different types of data distribution perspectives are provided to balance the data distribution of original samples and adversarial samples, thereby improving the adversarial robustness of the action recognition model without affecting the classification accuracy of the original samples.

[0052] The present invention addresses the problem of large differences in data distribution between clean samples and adversarial samples in action recognition adversarial defense scenarios, and has the following advantages: 1) Through a spatial-frequency domain flow mask generation method based on a genetic algorithm, the mask is used as the optimization target of the genetic algorithm, and the mask is used to locate the key areas of the input spatial and frequency domains, preventing the action recognition model from over-focusing on the redundant areas of the video time series; 2) Based on the positioning of multiple groups of key areas, a projected gradient descent method is used to generate strong adversarial samples, alleviating the problem that the existing action recognition model generates adversarial samples that are unrelated to the key areas of the video under the condition of a fixed perturbation threshold; 3) Through a multi-source adversarial training method, the original video samples and adversarial video samples are regarded as different perspectives, and the data distribution between the two is balanced through the consistency and difference between the two; this provides an adversarial training defense method for the security of action recognition models for practical application scenarios such as smart homes, virtual reality, and health monitoring. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] Figure 1 Schematic diagram of the process of countering attacks;

[0054] Figure 2 is a flow chart of the method of the present invention;

[0055] Figure 3This is a schematic diagram of the structure of the action recognition defense device based on multi-source adversarial training of spatial and frequency domain streams provided by the present invention;

[0056] Figure 4 It is a structural diagram of the action recognition defense device based on multi-source adversarial training of spatial and frequency domain streams provided by the present invention. DETAILED DESCRIPTION

[0057] The present invention will be further described below with reference to the accompanying drawings.

[0058] Figure 1 Reflecting the normal process of adversarial attacks, in the figure, a group of videos of the "playing baseball" action category are identified as "playing baseball" by the action recognition model with a probability of 91.8%. However, when a tiny adversarial perturbation is artificially added to the video clip, the original video clip is identified as a "bicycle riding" video by the action recognition model with a probability of 96.2%.

[0059] like Figure 2 This action recognition defense method based on multi-source adversarial training of spatial and frequency domain streams first samples the original input video frame samples. The original video samples are input into the spatial and frequency domain stream mask generation module, where a genetic algorithm is used to optimize the localization mask to obtain the spatial and frequency domain stream mask (input spatial and frequency domain masks). The spatial and frequency domain stream mask and the original video samples are then input into the spatial and frequency domain stream adversarial sample generation module, where projected gradient descent is used to generate video adversarial samples for adversarial training. Finally, the original video samples and video adversarial samples are input into the multi-source adversarial training module, resulting in an action recognition model with strong adversarial robustness. This method uses the spatial and frequency domain stream mask generation module to locate key regions of the original video samples from the input spatial and frequency domains, preventing the model from focusing on temporally redundant regions of the original video samples. The spatial and frequency domain stream adversarial sample generation module then generates strong adversarial samples, improving the quality of adversarial samples for adversarial training and learning the data distribution of real adversarial samples. The multi-source adversarial training module balances the data distribution of original video samples and adversarial samples, improving the adversarial robustness of the action recognition model.

[0060] The action recognition defense method based on multi-source adversarial training of spatial and frequency domain streams includes the following steps:

[0061] Step (1) obtain the original input video sample set and initialize the training model:

[0062] From a public action recognition video dataset, such as the Kinetics-400 action recognition data, obtain action recognition video data as raw video samples, obtain the action recognition model to be defended, and copy the model to be defended three times and randomly initialize the weights. They are named robust model, clean model, and adversarial model respectively.

[0063] Step (2) constructs a spatial-frequency domain flow mask generation module, which takes the original video sample and the robust model as input and outputs the spatial-frequency domain flow mask;

[0064] Step (3) construct a spatial-frequency domain flow adversarial sample generation module, the input is the original video sample and the spatial-frequency domain flow mask, and the output is the video adversarial sample;

[0065] Step (4) constructs a multi-source adversarial training module, the input of which is the original video sample, the video adversarial sample, the robust model, the clean model and the adversarial model, and the output is the robust model;

[0066] Step (5) using a stochastic gradient descent algorithm to iteratively optimize the action recognition defense method consisting of a spatial-frequency domain flow mask generation module, a spatial-frequency domain flow adversarial sample generation module, and a multi-source adversarial training module, and using the final output robust model as the action recognition defense model, using the defense model to generate video adversarial samples, and inputting the video adversarial samples into the defense model to obtain the adversarial robustness of the defense model;

[0067] Preferably, step (1) specifically includes the following steps:

[0068] (1-1) Select some original videos from the training set of the action recognition dataset and sample the original videos evenly video frames, the sampled video is the original video sample ; Among them, the three-dimensional tensor For the Frame video frame, is the field of real numbers, 、 、 are the width, height, and number of channels of the video frame respectively; Corresponding action category ;in, is the number of action categories;

[0069] The video data set described in step (1-1) is the Kinetics-400 action recognition dataset, with a video frame sampling of 32 frames, a video frame width, a height, and a channel number of 224, 224, and 3, respectively; and a total of 400 action categories.

[0070] (1-2) The action recognition model to be defended (For example, SlowFast model) copy three times and name them as robust models , clean model and adversarial models , randomly initialize the weights of the above three models.

[0071] The action recognition model to be defended in step (1-2) is a SlowFast model, wherein the backbone used by the source proxy model is a ResNet50 network.

[0072] The SlowFast model is a dual-channel model for video recognition tasks that improves action recognition accuracy by simultaneously processing both spatial and temporal information in videos. The core idea of this model is to divide the video into two distinct pathways: a slow pathway and a fast pathway. The slow pathway focuses on capturing spatial semantic information in the video, extracting relatively static content at a lower frame rate. The fast pathway focuses on temporal information in the video, capturing rapidly changing actions at a higher frame rate. The slow and fast pathways fuse information via lateral connections. This one-way information fusion allows information from the fast pathway to flow into the slow pathway, while the slow pathway does not receive any information from the fast pathway.

[0073] Preferably, in step (2), the construction of the spatial-frequency-domain flow mask generation module includes the following steps:

[0074] (2-1) Construct a spatial-frequency domain flow mask generation module, which consists of an input spatial-domain mask generation module and a frequency-domain mask generation module (the spatial-frequency domain flow mask consists of an input spatial-domain mask and a frequency-domain mask);

[0075] (2-2) The specific process of inputting the spatial domain mask generation module is as follows: the original video sample Input to (Contrastive learning for

[0076] Class-agnostic Activation Map) model, It is a foreground-background separation model that uses an unsupervised method to separate the foreground and background of the input sample and the original video sample Divided into foreground and background parts, ,in For video The foreground area, For video The background area, Random area generation A set of input spatial domain masks , ,in, , is the qth input spatial domain mask set, is the i-th video frame The corresponding input spatial mask matrix, Depend on equally divided mask blocks composition, is the width of the input spatial mask block, is the length of the input spatial mask block, Represents the i-th mask matrix The input spatial mask block in row j and column k, where the value 0 indicates mask and 1 indicates no mask, is used to convert the original video sample into respectively and The input spatial domain mask set is masked to obtain the input spatial domain mask video sample, ,in represents the qth input spatial domain mask video sample, represents Hadamard Product;

[0077] (2-3) Replace the Input spatial domain mask video samples Input to the robust model In the fast gradient sign method, we get Input spatial domain adversarial samples, input all input spatial domain adversarial samples into the clean model Get The mask set score, the fast gradient sign method is as follows, , , , sorted in descending order according to the mask set score, where is the cross entropy loss, y is the original video sample The true category action label, Indicates that Input to the robust model Obtained through the back propagation algorithm Gradient , , is a symbolic function, is the iteration step length, is the disturbance threshold, To truncate the function to ensure that the generated adversarial samples are within a limited range, represents the mask set ranked sth, , ;

[0078] (2-4) Set the mask with the highest score (The mask set with the highest score) and two random mask sets other than the mask set with the highest score 、 Perform cross fusion and repeat the above operations Second-rate( is an integer), we get A set of child input spatial masks , , , , the cross fusion formula first performs , then and conduct , specifically expressed as

[0079] ,in, Indicates that if the element values of the corresponding mask blocks are the same (all 1 or all 0), they become 1, if they are different, they are all 0. If the values of the corresponding mask blocks are all 0, the output is 0; if they are all 1, the output is 1; if they are different (i.e., the element value of one mask block is 1 and the other element value is 0), the probability The corresponding mask block value becomes 0. The probability corresponding to the mask block value becomes 1, represents the set of spatial masks of the t-th child input, ;

[0080] (2-5) Perform a "mutation" operation on the input space mask set that does not participate in the cross fusion in step (2-4), and obtain A set of child input spatial masks “Mutation” refers to the change of the mask block value in the input spatial mask set with probability From 0 to 1 or from 1 to 0, Descendant input spatial mask Use steps (2-3) to get Mask set score , select the mask set with the highest score As the input spatial domain mask set;

[0081] The input spatial domain mask generation module in steps (2-2) to (2-5) first separates the foreground and background areas of the original video sample, adds a randomly initialized input spatial domain mask to the background area of the original video sample, and continuously optimizes the input spatial domain mask using a genetic algorithm; unlike the traditional method of generating adversarial perturbations based on the global video area, the proposed input spatial domain mask generation module adaptively selects a suitable input spatial mask for each input video sample based on the training model knowledge and the input sample, avoiding the model from generating irrelevant adversarial perturbations based on the temporal redundant areas of the input sample, which is beneficial for the model to generate higher quality adversarial samples based on the key areas of the input samples during the adversarial training stage, thereby learning the data distribution of real adversarial samples.

[0082] (2-6) The frequency domain mask generation module is specifically: the original video sample Perform discrete cosine transform, define the upper left triangle area of the frequency domain matrix as the low-frequency area, and the lower right triangle area as the high-frequency area, and only mask the low-frequency area. ,in, is the Fourier-based discrete cosine transform, where For video The low-frequency region, For video In the high frequency region, Random generation of high-frequency areas Frequency domain mask set , ;in, , is the qth frequency domain mask set, is the i-th video frame The corresponding frequency domain mask matrix, Depend on equally divided mask blocks composition, is the width of the frequency domain mask block, is the length of the frequency domain mask block, Represents the i-th mask matrix The frequency domain mask block in row j and column k, where all values in the mask block are 0 or 1, converts the original video sample respectively and The frequency domain mask set is masked to obtain the frequency domain mask video sample, ,in represents the qth frequency domain mask video sample;

[0083] (2-7) Replace the Frequency domain mask video samples Input to the robust model In the fast gradient sign method, we get Frequency domain adversarial samples, all frequency domain adversarial samples are input into the clean model Get Mask set score , , , sorted in descending order according to the mask set score, where represents the mask set ranked sth, , ;

[0084] (2-8) Set the mask with the highest score (The mask set with the highest score) and two random mask sets other than the mask set with the highest score 、 Perform cross fusion and repeat the above operations Second-rate( is an integer), we get A set of child frequency domain masks , , , , the cross fusion formula first performs , then and conduct , specifically expressed as follows ,

[0085] represents the tth frequency domain mask set;

[0086] (2-9) The remaining steps (2-8) The mask set "mutates", and we get A set of child frequency domain masks ,Will Child frequency domain mask Use steps (2-7) to get Mask set score , select the mask set with the highest score As the input spatial domain mask set;

[0087] The frequency domain mask generation module in steps (2-6) to (2-9) locates the high-frequency and low-frequency areas of the original video sample from the perspective of the frequency domain, adds a randomly initialized frequency domain mask to the low-frequency area of the original video, and then continuously optimizes the frequency domain mask using a genetic algorithm; similar to the input spatial domain mask generation module, in order to more accurately locate the key area of the original video sample, the frequency domain areas irrelevant to the original video sample and the key area content are removed from the perspective of the frequency domain, thereby improving the quality of the adversarial samples used by the model in the adversarial training stage.

[0088] The step (3) specifically includes the following steps:

[0089] (3-1) Constructing the spatial and frequency domain stream adversarial sample generation module: Input the original video sample and the input spatial domain mask set , frequency domain mask set , and then use the iterative fast gradient attack method to generate video adversarial samples. The formula is as follows:

[0090] ,

[0091] in, , , is the discrete cosine transform, is the inverse discrete cosine transform, , is the number of iterations, is the disturbance threshold, is the iteration step length, Indicates cropping to , the final video adversarial sample is .

[0092] The spatial-frequency domain flow adversarial sample generation module uses the original video sample and the spatial-frequency domain flow mask obtained in step (2) to generate video adversarial samples using an iterative fast gradient attack method; the function of this module is to provide video adversarial samples as input for the multi-source adversarial training module in step (4).

[0093] The step (4) specifically includes the following steps:

[0094] (4-1) Construct a multi-source adversarial training module to convert the original video samples and video adversarial examples Input to clean model , adversarial model and robust models ,use 、 and The three models mentioned above are jointly optimized, and the loss is specifically expressed as follows:

[0095] ;

[0096]

[0097]

[0098] in, is a hyperparameter, is the cross entropy loss, is the KL divergence (Kullback-LeiblerDivergence), Represents input Sample robust model For real mark The predicted probability of To enhance the prediction sharpness of the hyperparameter, use Loss Optimization Clean Model ,use Loss-Optimized Adversarial Model ,use Loss Optimization Robust Model .

[0099] The multi-source adversarial training module in step (4) is different from the traditional adversarial training that uses the original video samples and the adversarial samples to train the defense model. This module trains three models with the same structure at the same time. Each model uses different data samples. The three models provide different data perspectives to each other. The remaining two models distill knowledge to another model to balance the data distribution of adversarial samples and original clean samples. It is similar to the idea of adversarial distillation but does not require the provision of additional pre-trained teacher models.

[0100] The step (5) specifically includes the following steps:

[0101] (5-1) Construct an action recognition adversarial training method consisting of a spatial-frequency domain flow mask generation module, an adversarial sample generation module, and a multi-source adversarial training module. Repeat steps (1-1) to (4-1) and use the stochastic gradient descent algorithm to optimize the action recognition model. Finally, the robust model As the final action recognition model.

[0102] (5-2) Testing the robust model The adversarial robustness and classification accuracy of the original samples are calculated using the original video samples in the test set. Generate video adversarial samples using the iterative fast gradient sign method in step (3-1) , the original video sample and video adversarial examples Input to Get the recognition rate.

[0103] The following describes implementations of the apparatus, device, and storage medium of the present invention.

[0104] The embodiment of the present invention provides an action recognition defense device 600 based on multi-source adversarial training of spatial and frequency domain streams, such as Figure 3The device specifically includes: an acquisition unit 601, a processing unit 602, and a detection unit 603. The functions of each component module are as follows:

[0105] The acquisition unit is used to: acquire action recognition video data as original video samples and the action recognition model to be defended,

[0106] The processing unit is used to: copy the model to be defended three times and randomly initialize the weights and name them as a robust model, a clean model and an adversarial model respectively; construct a spatial-frequency-domain flow mask generation module, which takes the original video sample and the robust model as input and outputs the spatial-frequency-domain flow mask; construct a spatial-frequency-domain flow adversarial sample generation module, which takes the original video sample and the spatial-frequency-domain flow mask as input and outputs the video adversarial sample; construct a multi-source adversarial training module, which takes the original video sample, the video adversarial sample, the robust model, the clean model and the adversarial model as input and outputs the robust model; iteratively optimize the action recognition defense model composed of the spatial-frequency-domain flow mask generation module, the spatial-frequency-domain flow adversarial sample generation module and the multi-source adversarial training module, and use the final output robust model as the action recognition defense model;

[0107] The detection unit is used to perform action recognition defense using the action recognition defense model.

[0108] The embodiment of the present application also provides a structural diagram of an action recognition defense device 500 based on multi-source adversarial training of spatial and frequency domain streams, as shown in FIG. Figure 4 The device specifically includes: one or more central processing units; a memory communicatively connected to the one or more central processing units; wherein the memory stores instructions executable by the one or more central processing units, and the instructions are executed by at least one central processing unit, so that the one or more central processing units can execute the action recognition defense method based on multi-source adversarial training of spatial and frequency domain streams according to one of the above-mentioned embodiments.

[0109] The action recognition defense device based on multi-source adversarial training of spatial and frequency domain streams in the embodiment of the present application may include but is not limited to mobile terminals such as mobile phone terminals, tablet computers, portable computers, etc.

[0110] In one embodiment, the motion recognition defense device based on multi-source adversarial training of spatial and frequency domain streams specifically includes a CPU device 505 (e.g., a central processing unit or graphics processing unit), which is configured to read and execute computer programs to implement corresponding processing functions. A read-only memory (ROM) 503 can be used to store program code, and a storage device 506 can load this code into a random access memory (RAM) 504. RAM 504 stores the instructions and data sets required for device operation. CPU device 505, ROM 503, RAM 504, and input / output (I / O) interface 502 are interconnected via a bus 501. The I / O interface 502 typically connects input devices (e.g., a keyboard, mouse, image acquisition device, microphone, etc.), output devices (e.g., an LCD screen, speaker, vibrator, etc.), a storage device 506 (e.g., a tape drive, hard disk, etc.), and a communication module 507. The communication module 507 enables the motion recognition defense device based on multi-source adversarial training of spatial and frequency domain streams to communicate with other systems via wired or wireless networks. Although the figure shows a complete system structure, it should be understood that some modules and functional components can be expanded, reduced or replaced as needed during actual deployment.

[0111] The present invention also provides an embodiment in which the process described above with reference to the flowchart is implemented as a computer software program. For example, the embodiments disclosed herein include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for executing the method shown in the flowchart. In the above implementation case, the computer program can be downloaded and installed from the storage device 506 over a network. When the computer program is executed by the CPU device 505, the above-mentioned functions defined in the method of the embodiment disclosed herein are performed.

[0112] It should be understood that the various parts disclosed in this application can be implemented using hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials, or characteristics can be combined in any one or more embodiments or examples in a suitable manner.

[0113] An embodiment of the present application also provides a computer-readable storage medium having computer-readable program instructions (i.e., a computer program) stored thereon, and the computer-readable program instructions are used to execute the action recognition defense method based on multi-source adversarial training of spatial and frequency domain streams in the above embodiment.

[0114] The computer-readable storage medium involved in this application may include a USB flash drive or other forms of semiconductor systems, modules or devices. The form of such media may include, but is not limited to: an electrical connection device with one or more wires, a portable disk, a hard disk drive, a random access memory (RAM), a read-only memory (ROM), an optical fiber, a magnetic storage device, or a combination of any of the above. In this application, the computer-readable medium may be any of the above-mentioned individual media or a combination thereof, and the computer program stored therein may be read and executed by the relevant system or device, or work in conjunction with it. In addition, the program stored in the medium can also be transmitted through a variety of transmission methods, such as wires, optical cables and optical fibers, or a combination of any of the above.

[0115] The above-mentioned computer-readable storage medium can be included in the action recognition defense device based on spatial-frequency-domain flow multi-source adversarial training; or it can exist independently without being assembled into the action recognition defense device based on spatial-frequency-domain flow multi-source adversarial training.

[0116] The contents described in this embodiment are merely an enumeration of the implementation forms of the inventive concept. The scope of protection of the present invention should not be regarded as being limited to the specific forms described in the embodiment. The scope of protection of the present invention also extends to equivalent technical means that can be conceived by those skilled in the art based on the inventive concept.

Claims

1. An action recognition defense method based on multi-source adversarial training of spatial and frequency domain streams, characterized by: The following steps are involved: Step (1) obtaining action recognition video data as the original video sample and the action recognition model to be defended, replicating the model to be defended three times and initializing the weights and naming them as robust model, clean model and adversarial model respectively; Step (2) constructing a spatial-domain frequency-domain flow mask generation module, wherein the spatial-domain frequency-domain flow mask generation module is composed of an input spatial-domain mask generation module and a frequency-domain mask generation module; the spatial-domain frequency-domain flow mask is composed of an input spatial-domain mask set and a frequency-domain mask set; The processing of the input spatial domain mask generation module includes the following steps: Input the original video sample into the foreground-background separation model to obtain the foreground and background parts. Randomly generate a plurality of input space mask sets consisting of input space domain masks in the background part, each of the input space masks consists of a plurality of rectangular mask blocks of the same size and with a pixel value of 0; Inputting the original video sample and the input spatial mask set into the mask set selection method to obtain the final input spatial domain mask set; The processing process of the frequency domain mask generation module includes the following steps: Perform discrete cosine transform on the original video sample to obtain the low-frequency part and the high-frequency part. Randomly generate a frequency domain mask set consisting of frequency domain masks in the low-frequency part, each of the frequency domain masks consists of a number of rectangular mask blocks with the same size and pixel value of 0; Inputting the original video sample and the frequency domain mask set into the mask set selection method to obtain the final frequency domain mask set; The mask set selection method comprises the following steps: Performing Hadamard products on the original video sample and each of the mask sets to obtain mask video samples corresponding to each of the mask sets; Input each masked video sample into the robust model respectively, use the fast gradient sign method to obtain the masked adversarial sample corresponding to each masked video sample, input all masked adversarial samples into the clean model to obtain the mask set score corresponding to each masked adversarial sample; each mask set corresponds to a mask set score; The original video sample is input into the spatial and frequency domain flow mask generation module, and the positioning mask is optimized using a genetic algorithm to obtain the spatial and frequency domain flow mask; Step (3) construct a spatial-frequency domain flow adversarial sample generation module, the input is the original video sample and the spatial-frequency domain flow mask, and the output is the video adversarial sample; Step (4) constructs a multi-source adversarial training module, inputs the original video samples and the video adversarial samples into the clean model, the adversarial model and the robust model, jointly optimizes the above three models, and outputs the robust model; Step (5) iteratively optimizes the action recognition defense model composed of the spatial-frequency domain flow mask generation module, the spatial-frequency domain flow adversarial sample generation module and the multi-source adversarial training module, and uses the final output robust model as the action recognition defense model.

2. The action recognition defense method based on multi-source adversarial training of spatial and frequency domain streams according to claim 1 is characterized in that: In step (1), The acquisition of the original video samples comprises the following steps: uniformly sampling the original videos in the action recognition data set to obtain the original video samples; The action recognition model to be defended is the SlowFast model.

3. The action recognition defense method based on multi-source adversarial training of spatial and frequency domain streams according to claim 1 is characterized in that: The method of using a genetic algorithm to optimize the positioning mask to obtain a spatial domain and frequency domain flow mask includes the following steps: S2 takes out the mask set with the highest mask set score from the input mask set, takes out a part of the remaining mask sets, randomly selects two mask sets for cross fusion, and obtains multiple cross fusion masks. Then, the mask set with the highest mask set score is cross-fused with each cross-fusion mask to obtain the first part of the child mask set; The cross-fusion operation means that if there is a mask block at the same position in both mask sets or no mask block exists, the pixel value at the same position is set to 1; If only one of the two mask sets has a mask block at the same position, and the other mask set does not have a mask block at the same position, then the pixel value at the same position is set to 1 with a preset probability; S3 takes out another part from the remaining mask set in step S2, and performs a mutation operation on the mask set to obtain a second part of the child mask set, wherein the mutation operation refers to changing the pixel values in some areas of the mask set to 0 or 1 with a preset probability; The sub-mask sets of the first and second parts are respectively Hadamard-producted with the original video samples, and input into the clean model to obtain the sub-mask set scores corresponding to each sub-mask set, and the sub-mask set with the highest sub-mask set score is taken as the final mask set.

4. The action recognition defense method based on multi-source adversarial training of spatial and frequency domain streams according to claim 1 is characterized in that: In step (3), The spatial-frequency-domain flow mask is composed of a final input spatial-domain mask set and a final frequency-domain mask set; The processing process of the spatial-frequency stream adversarial sample generation module includes the following steps: The original video sample and the spatial-frequency domain flow mask are input into the robust model, and the video adversarial sample is generated using the iterative fast gradient sign method.

5. The action recognition defense method based on multi-source adversarial training of spatial and frequency domain streams according to claim 1 is characterized in that: In step (4), the processing process of the multi-source adversarial training module includes the following steps: Input the original video sample into the clean model and get the output of the clean model; Input the video adversarial sample obtained in step (3) into the adversarial model to obtain the output of the adversarial model; Input the original video sample and the video adversarial sample into the robust model respectively, and obtain the original video output and adversarial sample output of the robust model respectively; Based on the output of the adversarial model and the original video output of the robust model, the clean model loss is constructed and optimized; Based on the output of the clean model and the adversarial sample output of the robust model, the loss of the adversarial model is constructed and the adversarial model is optimized; Based on the output of the adversarial model and the output of the clean model, the loss of the robust model is constructed and the robust model is optimized.

6. An action recognition defense device based on multi-source adversarial training of spatial and frequency domain streams, characterized in that: The device comprises: The acquisition unit is used to: acquire action recognition video data as original video samples and the action recognition model to be defended, The processing unit is used to: copy the model to be defended three times and randomly initialize the weights and name them as robust model, clean model and adversarial model respectively; construct a spatial frequency domain flow mask generation module, the spatial frequency domain flow mask generation module is composed of an input spatial domain mask generation module and a frequency domain mask generation module; the spatial frequency domain flow mask is composed of an input spatial domain mask set and a frequency domain mask set; the processing process of the input spatial domain mask generation module includes the following steps: input the original video sample into the foreground and background separation model to obtain the foreground part and the background part, and randomly generate a number of input spatial domain masks in the background part. The method comprises the following steps: performing discrete cosine transform on the original video sample to obtain a low-frequency part and a high-frequency part, randomly generating a frequency domain mask set composed of frequency domain masks in the low-frequency part, each of the frequency domain masks consists of a number of rectangular mask blocks of the same size and a pixel value of 0; inputting the original video sample and the input spatial mask set into the mask set selection method to obtain a final input spatial domain mask set; the processing process of the frequency domain mask generation module comprises the following steps: performing discrete cosine transform on the original video sample to obtain a low-frequency part and a high-frequency part, randomly generating a frequency domain mask set composed of frequency domain masks in the low-frequency part, each of the frequency domain masks consists of a number of rectangular mask blocks of the same size and a pixel value of 0; inputting the original video sample and the frequency domain mask set into the mask set selection method to obtain a final input spatial domain mask set; the processing process of the frequency domain mask generation module comprises the following steps: performing discrete cosine transform on the original video sample to obtain a low-frequency part and a high-frequency part, randomly generating a frequency domain mask set composed of frequency domain masks in the low-frequency part, Input the mask set selection method to obtain the final frequency domain mask set; the mask set selection method includes the following steps: perform Hadamard product on the original video sample and each mask set respectively to obtain the mask video sample corresponding to each mask set; input each mask video sample into the robust model respectively, use the fast gradient sign method to obtain the mask adversarial sample corresponding to each mask video sample, input all the mask adversarial samples into the clean model to obtain the mask set score corresponding to each mask adversarial sample; each mask set corresponds to a mask set score; input the original video sample into the spatial domain frequency domain flow mask generator The algorithm is used to generate a spatial-frequency-domain adversarial sample generation module, which takes the original video sample and the spatial-frequency-domain flow mask as input and outputs the video adversarial sample. The algorithm is used to construct a multi-source adversarial training module, which inputs the original video sample and the video adversarial sample into the clean model, the adversarial model, and the robust model, and jointly optimizes the three models to output the robust model. The action recognition defense model composed of the spatial-frequency-domain flow mask generation module, the spatial-frequency-domain flow adversarial sample generation module, and the multi-source adversarial training module is iteratively optimized, and the final output robust model is used as the action recognition defense model. The detection unit is used to perform action recognition defense using the action recognition defense model.

7. An action recognition defense device based on multi-source adversarial training of spatial and frequency domain streams, characterized in that: The device includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the action recognition defense method based on multi-source adversarial training of spatial-frequency streams as described in any one of claims 1 to 5.

8. A storage medium, characterized in that: The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by the processor, the steps of the action recognition defense method based on multi-source adversarial training of spatial and frequency domain streams as described in any one of claims 1 to 5 are implemented.

Images