Sparse video adversarial attack method and device based on spatiotemporal reversible neural network

Abstract

The application relates to a sparse video adversarial attack method and device based on a space-time reversible neural network. The method comprises the following steps: acquiring a guide video of a target class; learning a learnable target feature tensor under the guidance of the guide video of the target class and a guide target classification model by adopting a guide target video learning module; and performing space-time feature space information exchange between the target feature tensor learned by the guide target video learning module and a clean video by adopting a space-time reversible neural network module to generate an adversarial video and a residual video; the space-time reversible neural network module is used for utilizing the information preservation characteristic of the reversible neural network, performing information exchange between the target feature tensor and the clean video at a space-time feature level under the drive of an adversarial loss function, and generating the adversarial video and the residual video. The method fully considers the space-time features of the video, and generates a robust adversarial video sample which is difficult to detect.

Description

Technical Field

[0001] This application relates to the field of video adversarial attack technology, and in particular to a sparse video adversarial attack method and apparatus based on a spatiotemporally reversible neural network. Background Technology

[0002] Due to the spatiotemporal characteristics of video, robust and difficult-to-detect adversarial attacks on video are challenging. Existing video adversarial attack methods mainly employ gradient-based approaches to generate adversarial videos with malicious perturbations.

[0003] Recent advances in Deep Neural Networks (DNNs) have propelled the development of image and video content. Simultaneously, the security issues of DNNs have garnered significant attention. Szegedy et al. discovered that DNN-based image classifiers are easily fooled by adding slight adversarial noise to clean images. Since then, various adversarial attack methods have been proposed to investigate the security of DNNs. Existing image domain adversarial attack methods can be broadly categorized into two types: adding adversarial perturbations to clean images or removing semantic features of specific categories from clean images. Recently, Chen et al. proposed an image adversarial attack method based on Invertible Neural Networks (AdvINNs). By utilizing Invertible Neural Networks (INNs) for feature extraction and feature exchange, while simultaneously removing semantic information from clean images and adding category features of the target image, robust and imperceptible adversarial examples are generated.

[0004] Videos consist of a series of images, encompassing both spatial and temporal dimensions, and exhibit high temporal consistency between adjacent frames. While adversarial attacks targeting images can be naturally extended to videos, the enormous search space and information redundancy pose challenges to effectively generating adversarial examples for videos. To improve the imperceptibility of adversarial video examples and reduce computational costs, Wei et al. proposed a sparse attack method for videos, utilizing l 2,1The norm regularizes adversarial perturbations to temporal sparsity and propagates the generated perturbations to other frames through interactions in the temporal dimension of the target model. Inka et al. proposed a video adversarial attack method based on the Fast Gradient Sign Method (FGSM) for optical flow-based action recognition. To accelerate video adversary generation, Mu et al. proposed DeepSAVA, which efficiently alternates between keyframe selection and optimization of spatial transformation perturbations. By utilizing temporal dimension information, Jiang et al. proposed a video attack framework called V-DSA, which uses optical flow to select perturbation pixels for each frame. Although it reduces the search space, the adversarial perturbations are perceptible to human observers. Summary of the Invention

[0005] Therefore, it is necessary to provide a method and apparatus for sparse video adversarial attacks based on spatiotemporal reversible neural networks to address the aforementioned technical problems.

[0006] A sparse video adversarial attack method based on a spatiotemporally reversible neural network, the method comprising:

[0007] Retrieve the introductory video for the target class.

[0008] A guided target video learning module is used to learn the learnable target feature tensor under the guidance of the guided video of the target class and the target classification model.

[0009] A spatiotemporal reversible neural network module is used to exchange spatiotemporal feature space information between the target feature tensor learned by the target video learning module and the clean video, generating adversarial video and residual video. The spatiotemporal reversible neural network module is used to utilize the information preservation characteristics of reversible neural networks to exchange information between the target feature tensor and the clean video at the spatiotemporal feature level under the drive of the adversarial loss function, generating adversarial video and residual video.

[0010] In one embodiment, the guided target video learning module includes a target classification model.

[0011] A guided target video learning module is used to learn the learnable target feature tensor under the guidance of guided videos of the target class, including:

[0012] Set the target feature tensor as a learnable variable.

[0013] The target class's guiding video is input into the target classification model to obtain the target class's prediction result.

[0014] Construct a guided loss function; the guided loss function includes: the cross-entropy loss between the target feature tensor and the prediction results of the target class, and the MSE loss between the guided video of the target feature tensor and the target class.

[0015] The target feature tensor is updated based on the prediction results of the target class, the guiding video of the target class, the target feature tensor, and the guiding loss function.

[0016] In one embodiment, the guiding loss function is:

[0017]

[0018] in, To guide losses, For the target feature tensor X t The prediction result y of the target class t The cross-entropy loss function between them, F φ (·) represents the target classification model, X g For the target class of introductory video, λ b ,β b For regularization parameters, N = C * T * W * H, where C, T, W, and H are the number of channels, the number of video frames, the width of the video frame, and the height of the video frame, respectively.

[0019] In one embodiment, the spatiotemporally reversible neural network module includes: a 3D discrete wavelet transform layer for video feature decomposition, M ST-ACB layers for feature updating and prediction, and a 3D discrete wavelet inverse transform layer for video reconstruction; where M is an integer greater than 1.

[0020] A spatiotemporal reversible neural network module is used to exchange spatiotemporal feature spatial information between the target feature tensor learned by the guiding target video learning module and the clean video, generating adversarial videos and residual videos, including:

[0021] The target feature tensor learned by the guided target video learning module and the clean video are respectively decomposed into low-frequency and high-frequency components by a 3D discrete wavelet transform layer, resulting in the low-frequency and high-frequency components of the target feature tensor and the clean video.

[0022] The low-frequency and high-frequency components of the target feature tensor, as well as the low-frequency and high-frequency components of the clean video, are updated and predicted using M sequentially connected spatiotemporal affine coupling blocks.

[0023] The output features of the Mth empty affine coupling block are used to reconstruct the image using a 3D discrete wavelet inverse transform layer to obtain the predicted adversarial video and the predicted residual video.

[0024] Construct an adversarial loss function, which includes: cross-entropy loss between the predicted adversarial video and the clean video, and a factor used to adjust the generated perturbation to be more sparsity-based. The sparse loss of the norm and the low-frequency loss used to constrain the perturbation added to the target content in the high-frequency division of the 3D discrete wavelet transform.

[0025] Based on the predicted adversarial video, the predicted residual video, the high-frequency components of the clean video, the high-frequency components of the output of the Mth empty affine coupling block corresponding to the clean video, and the adversarial loss function, the parameters of the spatiotemporal reversible neural network module are optimized until the preset optimization target is reached, thus obtaining the adversarial video and the residual video.

[0026] In one embodiment, the forward process of the i-th ST-ACB can be represented as:

[0027]

[0028]

[0029] Among them, w i Let be the input feature of the i-th spatiotemporal affine coupling block, ⊙ denotes element-wise multiplication, and α is the Sigmoid function multiplied by a constant factor. Γ(·) represents the wavelet coefficients, Γ(·)∈R 8C ×T / 2×W / 2×H / 2 C, T, W, and H represent the number of channels, video frames, video frame width, and video frame height, respectively, while ψ(·), φ(·), ρ(·), and η(·) represent dense network architectures extended to 3D convolutional layers.

[0030] In one embodiment, the adversarial loss function is:

[0031]

[0032] Where N = C * T * W * H, λ a β a γ a F is the regularization parameter. φ (·) represents the target classification model, and the subscript (lll) represents the low-low-low frequency subband of 3D-DWT. CE (·) represents the cross-entropy loss function, and the target classification model F... φ (·), X g For target-oriented introductory videos, y t Let T be the target class, T be the number of video frames, and X be the value of the target class. a For adversarial videos, X c For clean videos, X c(lll) X a(lll) These are the low-low-low frequency subbands for clean video and adversarial video 3D-DWT, respectively.

[0033] A sparse video adversarial attack device based on a spatiotemporally reversible neural network, the device comprising:

[0034] The target class's guide video acquisition module is used to acquire the guide video for the target class.

[0035] The target feature tensor learning module is used to learn the learnable target feature tensor under the guidance of the guided target video learning module and the target classification model.

[0036] The adversarial video and residual video generation modules are used to exchange spatiotemporal feature space information between the target feature tensor learned by the guiding target video learning module and the clean video using a spatiotemporal reversible neural network module, thereby generating adversarial videos and residual videos. The spatiotemporal reversible neural network module is used to utilize the information preservation characteristics of reversible neural networks to exchange information between the target feature tensor and the clean video at the spatiotemporal feature level under the drive of the adversarial loss function, thereby generating adversarial videos and residual videos.

[0037] In one embodiment, the guided target video learning module includes: a target classification model; a target feature tensor learning module, further configured to set the target feature tensor as a learnable variable; inputting the guided video of the target class into the target classification model to obtain the prediction result of the target class; constructing a guided loss function; the guided loss function includes: the cross-entropy loss between the target feature tensor and the prediction result of the target class and the MSE loss between the target feature tensor and the guided video of the target class; and updating the target feature tensor based on the prediction result of the target class, the guided video of the target class, the target feature tensor, and the guided loss function.

[0038] In one embodiment, the guidance loss function in the guidance target video learning module is:

[0039]

[0040] in, To guide losses, For the target feature tensor X t The prediction result y of the target class t The cross-entropy loss function between them, F φ (·) represents the target classification model, X g For the target class of introductory video, λ b ,β b For regularization parameters, N = C * T * W * H, where C, T, W, and H are the number of channels, the number of video frames, the width of the video frame, and the height of the video frame, respectively.

[0041] In one embodiment, the spatiotemporally reversible neural network module includes: a 3D discrete wavelet transform layer for video feature decomposition, M ST-ACB layers for feature updating and prediction, and a 3D discrete wavelet inverse transform layer for video reconstruction; where M is an integer greater than 1; the adversarial video and residual video generation module is further configured to decompose the spatial and temporal dimensions of the video into low-frequency and high-frequency components using the 3D discrete wavelet transform layer on the target feature tensor learned by the guided target video learning module and the clean video, respectively, to obtain the low-frequency and high-frequency components of the target feature tensor and the low-frequency and high-frequency components of the clean video; update and predict the spatiotemporal features of the low-frequency and high-frequency components of the target feature tensor and the low-frequency and high-frequency components of the clean video using M sequentially connected spatiotemporal affine coupling blocks; reconstruct the image using the output features of the Mth empty affine coupling block using the 3D discrete wavelet inverse transform layer to obtain the predicted adversarial video and the predicted residual video; construct an adversarial loss function, which includes: cross-entropy loss between the predicted adversarial video and the clean video, and a perturbation-based loss function for adjusting the generated perturbation to be more sparse. The sparsity loss of the norm and the low-frequency loss used to constrain the perturbation added to the target content in the high-frequency component of the 3D discrete wavelet transform are used. Based on the predicted adversarial video, the predicted residual video, the high-frequency component of the clean video, the high-frequency component of the output of the Mth empty affine coupling block corresponding to the clean video, and the adversarial loss function, the parameters of the spatiotemporal reversible neural network module are optimized until the preset optimization target is reached, and the adversarial video and residual video are obtained.

[0042] The aforementioned sparse video adversarial attack method and apparatus based on spatiotemporal reversible neural networks include: acquiring a guiding video of the target class; using a guided target video learning module to learn a learnable target feature tensor under the guidance of the guiding video of the target class and a target classification model; using a spatiotemporal reversible neural network module to exchange spatiotemporal feature spatial information between the target feature tensor learned by the guided target video learning module and a clean video, generating adversarial videos and residual videos; the spatiotemporal reversible neural network module is used to utilize the information preservation characteristics of reversible neural networks to exchange information between the target feature tensor and the clean video at the spatiotemporal feature level under the drive of an adversarial loss function, generating adversarial videos and residual videos. This method fully considers the spatiotemporal characteristics of the video, generating robust adversarial video samples that are difficult to detect. Attached Figure Description

[0043] Figure 1 This is a flowchart illustrating a sparse video adversarial attack method based on a spatiotemporally reversible neural network in one embodiment.

[0044] Figure 2 This is a sparse adversarial video attack framework based on a spatiotemporally reversible neural network in one embodiment;

[0045] Figure 3 This is a structural block diagram of a sparse video adversarial attack device based on a spatiotemporal reversible neural network in one embodiment. Detailed Implementation

[0046] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0047] A sparse video adversarial attack model based on a spatiotemporally reversible neural network, abbreviated as SVASTIN.

[0048] The Guided Target Video Learning (GTVL) module.

[0049] The Spatio-Temporal Invertible Neural Network (STIN) module.

[0050] Spacetime affine coupling block, abbreviated as ST-ACB.

[0051] Three-dimensional discrete wavelet transform, abbreviated as 3D-DWT.

[0052] Three-dimensional discrete wavelet inverse transform, abbreviated as 3D-IDWT.

[0053] Reversible neural networks, or INNs for short.

[0054] In one embodiment, such as Figure 1 As shown, a spatiotemporal-based method is provided.

[0055] A sparse video adversarial attack method using inverse neural networks, comprising the following steps:

[0056] Step 100: Obtain the introductory video for the target class.

[0057] Specifically, the guide video is selected from the target class.

[0058] Step 102: The guided target video learning module is used to learn the learnable target feature tensor under the guidance of the guided video of the target class and the target classification model.

[0059] Specifically, for image adversarial attacks, Chen et al. proposed three methods for selecting target images: Highest Confidence Target Image (HCT), Universal Adversarial Perturbation Image (UAP), and Classifier Guided Target Image (CGT). However, these methods cannot effectively and efficiently learn the target information for video adversarial attacks. Specifically, HCT carries a large amount of redundant information unrelated to the target class, the UAP method cannot successfully perturb all target classes, and CGT requires optimization in a search space much larger than the image. To improve the effectiveness and efficiency of target video information learning, this application proposes a guided target video learning module to provide a suitable target feature tensor (i.e., target information) for the spatiotemporally reversible neural network module.

[0060] The guided target video learning module is used to effectively learn the target feature tensor and update the learnable target feature tensor under the guidance of the target class's guided video and the target classification model.

[0061] Target feature tensor X t The source information for the target class.

[0062] Step 104: The spatiotemporal reversible neural network module is used to exchange spatiotemporal feature space information between the target feature tensor learned by the target video learning module and the clean video to generate adversarial video and residual video. The spatiotemporal reversible neural network module is used to utilize the information preservation characteristics of reversible neural networks to exchange information between the target feature tensor and the clean video at the spatiotemporal feature level under the drive of the adversarial loss function, thereby generating adversarial video and residual video.

[0063] Specifically, reversible neural networks (INNs) with information-preserving properties have been used to employ image adversarial attacks that are difficult to detect and robust. Compared to images, videos possess an additional temporal dimension, a crucial feature that should be leveraged. To facilitate better spatiotemporal information exchange, a reversible neural network with spatiotemporal affine coupling blocks is proposed. The spatiotemporal reversible neural network module is used to apply three-dimensional discrete wavelet transform to clean video X. c and target feature tensor X t The video is decomposed, and then a reversible neural network with spatiotemporal affine coupling blocks is used to capture motion information from both spatial and temporal dimensions. Finally, the residual video X is reconstructed using three-dimensional discrete wavelet inverse transform. r adversarial video X a .

[0064] The Spacetime Reversible Neural Network (STIN) module utilizes its information preservation properties to process clean video X as input. c and a target feature tensor X tThe spatiotemporal characteristic spatial information exchange between them, the spatiotemporal reversible neural network module can simultaneously extract and process spatiotemporal information, and this module is completely reversible.

[0065] A framework for sparse adversarial video attack models based on spatiotemporal reversible neural networks, such as... Figure 2 As shown, the sparse adversarial video attack model based on spatiotemporal reversible neural networks generates adversarial videos through the exchange of spatiotemporal feature space information. The model consists of a guided target video learning module and a spatiotemporal reversible neural network module. The guided target video learning module is used to balance the perturbation size and optimize the speed, while the spatiotemporal reversible neural network module is used to exchange spatiotemporal feature space information between the clean video and the target feature tensor learned by the GTVL module.

[0066] This framework generates adversarial video examples based on a Spatiotemporally Reversible Neural Network (STIN) and Guided Target Video Learning (GTVL) module. Leveraging the information-preserving properties of INNs, and driven by an adversarial loss function, information exchange occurs at the spatiotemporal feature level. This allows adversarial video samples to be generated by adding class-specific features of the target feature tensor learned by GTVL to the clean video while simultaneously removing the discriminative information from the clean video.

[0067] The aforementioned sparse video adversarial attack method based on spatiotemporal reversible neural networks includes: acquiring a guiding video for the target class; using a guided target video learning module to learn a learnable target feature tensor under the guidance of the guiding video and a target classification model; and using a spatiotemporal reversible neural network module to exchange spatiotemporal feature spatial information between the target feature tensor learned by the guided target video learning module and a clean video, generating adversarial videos and residual videos. The spatiotemporal reversible neural network module utilizes the information preservation characteristics of reversible neural networks to exchange information between the target feature tensor and the clean video at the spatiotemporal feature level, driven by an adversarial loss function, to generate adversarial videos and residual videos. This method fully considers the spatiotemporal characteristics of the video, generating robust adversarial video samples that are difficult to detect.

[0068] In one embodiment, the guided target video learning module includes a target classification model. Step 102 includes: setting the target feature tensor as a learnable variable; inputting the guided video of the target class into the target classification model to obtain the prediction result of the target class; constructing a guidance loss function; the guidance loss function includes: the cross-entropy loss between the target feature tensor and the prediction result of the target class and the MSE loss between the target feature tensor and the guided video of the target class; updating the target feature tensor according to the prediction result of the target class, the guided video of the target class, the target feature tensor, and the guidance loss function.

[0069] Specifically, the target feature tensor X tSet as a learnable variable, and in the target class's guide video X g And target classification model F φ Updates are performed under the guidance of (·). Compared to directly using HCT video as the target feature tensor, the GTVL module can mitigate the impact of the large amount of redundant information in HCT video. Compared to the CGT method, the GTVL module utilizes a guiding video and a guiding loss, facilitating fast convergence. Therefore, the GTVL module effectively generates target feature tensors with high confidence.

[0070] In one embodiment, the guiding loss function is:

[0071]

[0072] in, To guide losses, For the target feature tensor X t The prediction result y of the target class t The cross-entropy loss function between them, F φ (·) represents the target classification model, X g For the target class of introductory video, λ b ,β b For regularization parameters, N = C * T * W * H, where C, T, W, and H are the number of channels, the number of video frames, the width of the video frame, and the height of the video frame, respectively.

[0073] Specifically, in order to effectively embed target content for information exchange, this application uses bootstrapping loss. To learn the target feature tensor X t The loss includes the target feature tensor X. t The cross-entropy loss between the predicted class and the target class, and the target feature tensor X t And the target class's introductory video X g MSE loss between The guiding loss function is shown in Equation (1).

[0074] In one embodiment, the spatiotemporally reversible neural network module includes: a 3D discrete wavelet transform layer for video feature decomposition, M ST-ACB layers for feature updating and prediction, and a 3D discrete wavelet inverse transform layer for video reconstruction; where M is an integer greater than 1.

[0075] Step 104 includes: decomposing the spatial and temporal features of the target feature tensor learned by the guided target video learning module and the clean video into low-frequency and high-frequency components using a 3D discrete wavelet transform layer, respectively, to obtain the low-frequency and high-frequency components of the target feature tensor and the clean video; updating and predicting the low-frequency and high-frequency components of the target feature tensor and the clean video using M sequentially connected spatiotemporal affine coupling blocks; reconstructing the image using the output features of the Mth empty affine coupling block using a 3D discrete wavelet inverse transform layer, to obtain the predicted adversarial video and the predicted residual video; constructing an adversarial loss function, which includes: cross-entropy loss between the predicted adversarial video and the clean video, and a perturbation-based loss function to adjust the generated perturbation to be more sparse. The sparsity loss of the norm and the low-frequency loss used to constrain the perturbation added to the target content in the high-frequency component of the 3D discrete wavelet transform are used. Based on the predicted adversarial video, the predicted residual video, the high-frequency component of the clean video, the high-frequency component of the output of the Mth empty affine coupling block corresponding to the clean video, and the adversarial loss function, the parameters of the spatiotemporal reversible neural network module are optimized until the preset optimization target is reached, and the adversarial video and residual video are obtained.

[0076] Specifically, perturbations are added to the high-frequency components obtained from 3D-DWT decomposition, thereby producing adversarial videos with greater imperceptibility.

[0077] In one embodiment, the forward process of the i-th ST-ACB can be represented as:

[0078]

[0079]

[0080] Among them, w i Let be the input feature of the i-th spatiotemporal affine coupling block, ⊙ denotes element-wise multiplication, and α is the Sigmoid function multiplied by a constant factor. Γ(·) represents the wavelet coefficients, Γ(·)∈R 8C ×T / 2×W / 2×H / 2 C, T, W, and H represent the number of channels, video frames, video frame width, and video frame height, respectively, while ψ(·), φ(·), ρ(·), and η(·) represent dense network architectures extended to 3D convolutional layers.

[0081] Specifically, a three-dimensional discrete wavelet transform layer (3D-DWT layer): Wavelet transform is widely used to analyze images and signals with different frequency components. In this application, 3D-DWT is used to transform video... The spatial and temporal dimensions are decomposed into low-frequency and high-frequency components, resulting in eight sub-bands. Its wavelet coefficients can be expressed as... It is worth noting that, compared to 2D-DWT, 3D-DWT further decomposes the time dimension into low-frequency and high-frequency subbands, which can help capture motion information in the video.

[0082] Spatiotemporal Affine Coupling Block: Affine Coupling Blocks (ACBs) are key building blocks of reversible neural networks, but their design for processing two-dimensional images fails to capture temporal information effectively in video. This application proposes a Spatiotemporal Affine Coupling Block with 3D convolutional layers (ST-ACB) to facilitate temporal information modeling. i As the input feature of the i-th spatiotemporal affine coupling block, and using and The forward process of the i-th ST-ACB can be represented by formulas (2) to (3).

[0083] Given the output of the Mth ST-ACB, apply the 3D discrete wavelet inverse transform to the 3D-IDWT layer. and This is used to reconstruct the adversarial and residual videos. Therefore, the STIN module is completely reversible, outputting the video (X... a ,X r ) and input video (X c ,X t They contain the same amount of information.

[0084] In one embodiment, the adversarial loss function is:

[0085]

[0086] Where N = C * T * W * H, λ a β a γ a F is the regularization parameter. φ (·) represents the target classification model, and the subscript (lll) represents the low-low-low frequency subband of 3D-DWT. CE (·) represents the cross-entropy loss function, and the target classification model F... φ (·), X g For target-oriented introductory videos, y t Let T be the target class, T be the number of video frames, and X be the value of the target class. a For adversarial videos, X c For clean videos, X c(lll) X a(lll) These are the low-low-low frequency subbands for clean video and adversarial video 3D-DWT, respectively.

[0087] Specifically, using adversarial loss This guides the optimization of STIN module parameters. Specifically, Including adversarial video X a With target class y t The cross-entropy loss between them, as shown in Equation (4), is used to adjust the generated perturbation to be sparser based on The sparse loss of the norm, along with the low-frequency loss used to constrain the perturbation added to the target content in the high-frequency subband of 3D-DWT, results in more difficult-to-detect adversarial examples.

[0088] It should be understood that, although Figure 1 The steps in the flowchart are shown sequentially as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order in which these steps are executed, and they can be performed in other orders. Figure 1 At least some of the steps in the process may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least some of the sub-steps or stages of other steps.

[0089] In a validation embodiment, quantitative and qualitative results of SVASTIN are compared with those of other state-of-the-art sparse video adversarial methods. The STIN module, GTVL module, and low-frequency loss are validated. The effectiveness.

[0090] (1) Experimental setup

[0091] Datasets and Models: The performance of the comparative methods was evaluated on Kinetics-400 and UCF-101. Under the same experimental settings, a clean input video was randomly selected from each category of the dataset and correctly classified by the target classification model. Three commonly used video action recognition models, MVIT, SLOWFAST, and TSN, were used as the target classification models.

[0092] Settings: For the STIN module, use a learning rate of 1e. -4 The Adam optimizer optimizes its parameters relative to equation (4). Based on experience, λ is set accordingly. a =0.3, β a =0.4 and γ a =10. For GTVL, the Adam optimizer with a learning rate of 1 / 255 is used for Equation (3) on X. t Optimize and adjust parameter λ. b and β bThe regularization values ​​were set to 0.01 and 1, respectively.

[0093] Comparison Method: This embodiment includes a comparison of three white-box attack methods: traditional adversarial attack (C&W), sparse attack method (Sparse-Attack), and DeepSAVA.

[0094] Evaluation metrics: Five metrics were used to evaluate the performance of different methods, including Mean Square Error (MSE), Structural Similarity Index (SSIM), Peak Signal-to-Noise Ratio (PSNR), Fréchet Inception Distance (FID), and Fooling rate (FR).

[0095] (2) Targeted attack assessment

[0096] Table 1 shows the target attack performance of different methods on UCF-101 and Kinetics-400. A generated adversarial example is considered successful if the confidence score of the target class is greater than 90%. The results show that, compared with other methods, our method achieves the highest overall frame rate (FR) and generates the best quality adversarial examples, especially in perceptual metrics. For example, our method achieves 100% FR when fooling the MVIT model on the Kinect-400 dataset, while also exhibiting higher structural and perceptual similarity compared to the comparison methods. In terms of PSNR, our method is 9.41 and 14.54 higher than Sparse-Attack and DeepSAVA, respectively. In terms of the perceptual metric FID, our method is 181.54 and 74.83 lower than Sparse-Attack and DeepSAVA, respectively. Therefore, our method achieves the best visual quality and the highest imperceptibility.

[0097] Table 1. Comparison of different targeted attack methods against the benchmark dataset and model

[0098]

[0099] Note: Higher values ​​for MSE, SSIM, PSNR, and FRT are better, and vice versa; The lower the FID value, the better, and vice versa.

[0100] (3) Ablation test

[0101] Effectiveness of the STIN Module: The STIN module utilizes the spatiotemporal information of the clean video feature tensor and the target video feature tensor to exchange feature space information. Improper processing of the video's temporal features will limit the performance of video adversarial attack methods. Table 2 shows the attack performance of 3D-DWT and ST-ACB using 2D-DWT and 2D convolution in AdvINN, as well as the STIN module of IIEM. The results show that the STIN module outperforms IIEM in both the quality of adversarial videos and the optimization speed.

[0102] Table 2. Ablation Experiment: Effectiveness of the STIN Module

[0103]

[0104] Note: MSE, FID, The smaller the value, the better, and vice versa; the larger the values ​​of SSIM and PSNR, the better, and vice versa.

[0105] Effectiveness of the GTVL Module: Table 3 presents the performance of three methods for learning the target feature tensor. CGT learns the target feature tensor only from the classifier, while HCT utilizes natural videos selected from the target class as the target feature tensor. For the GTVL module, the target feature tensor X is initialized with a learnable variable. t Then, randomly select a guide video X from the target class. g Further fine-tuning of the guiding video X g This induces misclassification into the target class. It can be seen that the GTVL module plays a crucial role in SVASTIN, effectively improving the imperceptibility of adversarial perturbations and the convergence speed.

[0106] Table 3. Ablation Experiment: Effectiveness of the GTVL Module

[0107]

[0108] Note: MSE, FID, The smaller the Epochs value, the better, and vice versa; the larger the SSIM and PSNR values, the better, and vice versa.

[0109] Low frequency loss Effectiveness: 3D-DWT decomposes the spatial and temporal dimensions of the input video into eight high- and low-frequency subbands. To improve the imperceptibility of adversarial examples, this application proposes a low-frequency loss to constrain the correction primarily applied to high-frequency coefficients. It can help achieve smoothness in both space and time. Table 4 compares the results with and without considering the low-frequency loss. Experiments were conducted using the MVIT model. Table 4 shows that... The SVASTIN model in MSE, SSIM, PSNR, It exhibits better qualitative performance in terms of FID and has a slightly improved optimization speed.

[0110] Table 4. Ablation Experiment: Effectiveness of Low-Frequency Loss

[0111]

[0112] Note: MSE, FID, The smaller the Epochs value, the better, and vice versa; the larger the SSIM and PSNR values, the better, and vice versa.

[0113] In one embodiment, such as Figure 3 As shown, a sparse video adversarial attack device based on a spatiotemporally reversible neural network is provided, comprising: a target class guided video acquisition module, a target feature tensor learning module, and an adversarial video and residual video generation module, wherein:

[0114] The target class's guide video acquisition module is used to acquire the guide video for the target class.

[0115] The target feature tensor learning module is used to learn the learnable target feature tensor under the guidance of the guided target video learning module and the target classification model.

[0116] The adversarial video and residual video generation modules are used to exchange spatiotemporal feature space information between the target feature tensor learned by the guiding target video learning module and the clean video using a spatiotemporal reversible neural network module, thereby generating adversarial videos and residual videos. The spatiotemporal reversible neural network module is used to utilize the information preservation characteristics of reversible neural networks to exchange information between the target feature tensor and the clean video at the spatiotemporal feature level under the drive of the adversarial loss function, thereby generating adversarial videos and residual videos.

[0117] In one embodiment, the guided target video learning module includes: a target classification model; a target feature tensor learning module, further configured to set the target feature tensor as a learnable variable; inputting the guided video of the target class into the target classification model to obtain the prediction result of the target class; constructing a guided loss function; the guided loss function includes: the cross-entropy loss between the target feature tensor and the prediction result of the target class and the MSE loss between the target feature tensor and the guided video of the target class; and updating the target feature tensor based on the prediction result of the target class, the guided video of the target class, the target feature tensor, and the guided loss function.

[0118] In one embodiment, the guidance loss function in the guidance target video learning module is as shown in formula (1).

[0119] In one embodiment, the spatiotemporally reversible neural network module includes: a 3D discrete wavelet transform layer for video feature decomposition, M ST-ACB layers for feature updating and prediction, and a 3D discrete wavelet inverse transform layer for video reconstruction; where M is an integer greater than 1; the adversarial video and residual video generation module is further configured to decompose the spatial and temporal dimensions of the video into low-frequency and high-frequency components using the 3D discrete wavelet transform layer on the target feature tensor learned by the guided target video learning module and the clean video, respectively, to obtain the low-frequency and high-frequency components of the target feature tensor and the low-frequency and high-frequency components of the clean video; update and predict the spatiotemporal features of the low-frequency and high-frequency components of the target feature tensor and the low-frequency and high-frequency components of the clean video using M sequentially connected spatiotemporal affine coupling blocks; reconstruct the image using the output features of the Mth empty affine coupling block using the 3D discrete wavelet inverse transform layer to obtain the predicted adversarial video and the predicted residual video; construct an adversarial loss function, which includes: cross-entropy loss between the predicted adversarial video and the clean video, and a perturbation-based loss function for adjusting the generated perturbation to be more sparse. The sparsity loss of the norm and the low-frequency loss used to constrain the perturbation added to the target content in the high-frequency component of the 3D discrete wavelet transform are used. Based on the predicted adversarial video, the predicted residual video, the high-frequency component of the clean video, the high-frequency component of the output of the Mth empty affine coupling block corresponding to the clean video, and the adversarial loss function, the parameters of the spatiotemporal reversible neural network module are optimized until the preset optimization target is reached, and the adversarial video and residual video are obtained.

[0120] In one embodiment, the forward process of the i-th ST-ACB in the spatiotemporally reversible neural network module is shown in Equations (2) and (3).

[0121] In one embodiment, the adversarial loss function in the spatiotemporally reversible neural network module is shown in Equation (4).

[0122] Specific limitations regarding the sparse video adversarial attack device based on spatiotemporal reversible neural networks can be found in the limitations of the sparse video adversarial attack method based on spatiotemporal reversible neural networks mentioned above, and will not be repeated here. Each module in the aforementioned sparse video adversarial attack device based on spatiotemporal reversible neural networks can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the corresponding operations of each module.

[0123] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0124] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A sparse video adversarial attack method based on a spatiotemporally reversible neural network, characterized in that, The method includes: Retrieve the introductory video for the target class; The learnable target feature tensor is learned by using a guided target video learning module under the guidance of the guided video of the target class and the target classification model. A spatiotemporal reversible neural network module is used to exchange spatiotemporal feature spatial information between the target feature tensor learned by the guiding target video learning module and the clean video to generate adversarial video and residual video. The spatiotemporal reversible neural network module is used to utilize the information preservation characteristics of reversible neural networks to exchange information between the target feature tensor and the clean video at the spatiotemporal feature level under the drive of the adversarial loss function, thereby generating adversarial video and residual video. The spatiotemporal reversible neural network module includes: a 3D discrete wavelet transform layer for video feature decomposition, M ST-ACB layers for feature updating and prediction, and a 3D discrete wavelet inverse transform layer for video reconstruction; where M is an integer greater than 1. A spatiotemporal reversible neural network module is used to exchange spatiotemporal feature spatial information between the target feature tensor learned by the guided target video learning module and the clean video, generating adversarial videos and residual videos, including: The target feature tensor learned by the guided target video learning module and the clean video are respectively decomposed into low-frequency and high-frequency components in the spatial and temporal dimensions of the video using a 3D discrete wavelet transform layer, resulting in the low-frequency and high-frequency components of the target feature tensor and the low-frequency and high-frequency components of the clean video. The low-frequency and high-frequency components of the target feature tensor, as well as the low-frequency and high-frequency components of the clean video, are updated and predicted using M sequentially connected spatiotemporal affine coupling blocks. The output features of the Mth empty affine coupling block are used to reconstruct the image using the 3D discrete wavelet inverse transform layer to obtain the predicted adversarial video and the predicted residual video. Construct an adversarial loss function, which includes: a cross-entropy loss between the predicted adversarial video and the clean video, and a factor for adjusting the generated perturbation to be more sparsity-based. The sparsity loss of the norm and the low-frequency loss used to constrain the perturbation added to the target content in the high-frequency division of the 3D discrete wavelet transform; Based on the predicted adversarial video, the predicted residual video, the high-frequency components of the clean video, the high-frequency components of the output of the Mth empty affine coupling block corresponding to the clean video, and the adversarial loss function, the parameters of the spatiotemporal reversible neural network module are optimized until a preset optimization target is reached, thereby obtaining the adversarial video and the residual video. No. The forward process of each ST-ACB is as follows: in, For the first Input features of a spatiotemporal affine coupling block, This represents element-wise multiplication. It is the Sigmoid function multiplied by a constant factor. , , These are wavelet coefficients. C, T, W, and H represent the number of channels, the number of video frames, the width of the video frame, and the height of the video frame, respectively. , , , This indicates a dense network architecture extended to 3D convolutional layers.

2. The method according to claim 1, characterized in that, The guided target video learning module includes: a target classification model; The guided target video learning module learns the learnable target feature tensor under the guidance of the guided video of the target class, including: Set the target feature tensor as a learnable variable; The guiding video of the target class is input into the target classification model to obtain the prediction result of the target class; Construct a guiding loss function; the guiding loss function includes: cross-entropy loss between the target feature tensor and the prediction result of the target class, and MSE loss between the target feature tensor and the guiding video of the target class; The target feature tensor is updated based on the prediction result of the target class, the guiding video of the target class, the target feature tensor, and the guiding loss function.

3. The method according to claim 2, characterized in that, The guiding loss function is: in, To guide losses, For target feature tensor Prediction results for target class The cross-entropy loss function between them For target classification models, Introductory videos for target audiences, , For regularization parameters, N = C * T * W * H, where C, T, W, and H are the number of channels, the number of video frames, the width of the video frame, and the height of the video frame, respectively.

4. The method according to claim 1, characterized in that, The adversarial loss function is: in , , , For regularization parameters, For target classification models, subscript For the low-low-low frequency subband of 3D-DWT, The cross-entropy loss function is used in the target classification model. , Introductory videos for target audiences, Let T be the target class and T be the number of video frames. For adversarial videos, For clean videos, , These are the low-low-low frequency subbands for clean video and adversarial video 3D-DWT, respectively.

5. A sparse video adversarial attack device based on a spatiotemporally reversible neural network, characterized in that, The device includes: The target class's guide video acquisition module is used to acquire the target class's guide video; The target feature tensor learning module is used to learn the learnable target feature tensor under the guidance of the guided target video learning module and the target classification model of the target class. The adversarial video and residual video generation module is used to exchange spatiotemporal feature spatial information between the target feature tensor learned by the guiding target video learning module and the clean video using a spatiotemporal reversible neural network module, thereby generating adversarial video and residual video. The spatiotemporal reversible neural network module is used to utilize the information preservation characteristics of reversible neural networks to exchange information between the target feature tensor and the clean video at the spatiotemporal feature level under the drive of the adversarial loss function, thereby generating adversarial video and residual video. The spatiotemporal reversible neural network module includes: a 3D discrete wavelet transform layer for video feature decomposition, M ST-ACB layers for feature updating and prediction, and a 3D discrete wavelet inverse transform layer for video reconstruction; where M is an integer greater than 1. The adversarial video and residual video generation module is further configured to decompose the spatial and temporal dimensions of the video into low-frequency and high-frequency components using the 3D discrete wavelet transform layer on the target feature tensor learned by the guided target video learning module and the clean video, respectively, to obtain the low-frequency and high-frequency components of the target feature tensor and the clean video; update and predict the spatiotemporal features of the target feature tensor and the clean video using M sequentially connected spatiotemporal affine coupling blocks; reconstruct the image using the output features of the Mth empty affine coupling block using the 3D discrete wavelet inverse transform layer to obtain the predicted adversarial video and the predicted residual video; construct an adversarial loss function, which includes: cross-entropy loss between the predicted adversarial video and the clean video, and a perturbation-based loss function for adjusting the generated perturbation to be more sparse. The sparse loss of the norm and the low-frequency loss used to constrain the perturbation added to the target content in the high-frequency component of the 3D discrete wavelet transform; based on the predicted adversarial video, the predicted residual video, the high-frequency component of the clean video, the high-frequency component of the output of the Mth empty affine coupling block corresponding to the clean video, and the adversarial loss function, the parameters of the spatiotemporal reversible neural network module are optimized until the preset optimization target is reached, and the adversarial video and the residual video are obtained; No. The forward process of each ST-ACB is as follows: in, For the first Input features of a spatiotemporal affine coupling block, This represents element-wise multiplication. It is the Sigmoid function multiplied by a constant factor. , , These are wavelet coefficients. C, T, W, and H represent the number of channels, the number of video frames, the width of the video frame, and the height of the video frame, respectively. , , , This indicates a dense network architecture extended to 3D convolutional layers.

6. The apparatus according to claim 5, characterized in that, The guided target video learning module includes: a target classification model; a target feature tensor learning module, which is further used to set the target feature tensor as a learnable variable; inputting the guided video of the target class into the target classification model to obtain the prediction result of the target class; constructing a guided loss function; the guided loss function includes: cross-entropy loss between the target feature tensor and the prediction result of the target class and MSE loss between the target feature tensor and the guided video of the target class; updating the target feature tensor according to the prediction result of the target class, the guided video of the target class, the target feature tensor, and the guided loss function.

7. The apparatus according to claim 6, characterized in that, The guidance loss function described in the guided target video learning module is: in, To guide losses, For target feature tensor Prediction results for target class The cross-entropy loss function between them For target classification models, Introductory videos for target audiences, , For regularization parameters, N = C * T * W * H, where C, T, W, and H are the number of channels, the number of video frames, the width of the video frame, and the height of the video frame, respectively.

Images