Image-to-video cross-modal adversarial sample generation method based on key frame recognition

Abstract

The present invention discloses a method for generating cross-modal adversarial samples from images to videos based on key frame recognition, including the following steps: obtaining a video frame sequence; obtaining key video segments through a sliding window strategy and video segment contribution; selecting key frames from key video segments based on an optical flow method; forming a key frame set from all key frames in the key video segments; generating a corresponding adversarial sample for each key frame in the key frame set; replacing the corresponding key frame in a video frame sequence X with the adversarial sample to obtain an adversarial video sequence of the video frame sequence X. The present invention effectively overcomes the defects of the prior art such as inaccurate key frame positioning, failure to fully consider timing information when extracting key frames, and poor robustness of the perturbation generation method by improving the key frame extraction method and the adversarial perturbation generation technology, and can significantly improve the accuracy and robustness of video adversarial attacks, and better meet the needs of practical applications.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to a method for generating cross-modal adversarial samples from images to videos based on key frame recognition. Background Art

[0002] In the study of video adversarial attacks, the technology has been extended from the perturbation of a single image to video data, targeting tasks such as video classification and action recognition. These tasks usually use time-aware neural network architectures, such as 3D convolutional networks (C3D), I3D, and SlowFast models, to capture dynamic features between frames. Therefore, an effective adversarial attack strategy must consider both single-frame perturbations and their cumulative effects on time series. Currently, the key technologies for implementing video adversarial attacks include: technology (1), using video models to perform gradient analysis on input videos to generate perturbations and selectively apply perturbations to key frames, and technology (2), using image models such as I2V or BTC to map perturbations to video frames. These methods have jointly promoted the development of video adversarial attack technology and enhanced the depth and breadth of robustness testing of deep learning models.

[0003] About technology (1): First, identify key frames based on the Grad-CAM (Gradient-weighted Class Activation Mapping) idea, and then apply perturbations to the key frames. When identifying key frames, Grad-CAM calculates the gradient of the convolutional layer output relative to the classification label y, and calculates the weight by the global average of the gradient, thereby assigning a weight to each frame of the image and arranging them in descending order according to these weights, thereby dynamically determining the key frames. This method can adaptively adjust the number of key frames. This method not only improves the flexibility of adversarial attacks, but also achieves effective selection of key frames through accurate evaluation of gradient contributions. However, this type of method has the following problems:

[0004] (1) Temporal information is not considered. In the extraction of key frames, key frame extraction methods based on gradient calculation, such as class activation mapping (G-CAM), usually only focus on the features of a single frame image and independently evaluate the contribution of local frames. The correlation between video frames is ignored, resulting in the inability to fully capture dynamic information across frames, which is crucial for action recognition models.

[0005] (2) Key frame positioning is inaccurate. Action recognition videos do not contain important information globally. Key frame extraction based on optical flow cannot accurately locate the position of key frames that affect the model output results only through the information changes between adjacent frames. It is very likely that video frames that do not contain valid information but have drastic changes will be extracted.

[0006] Regarding the technique (2) I2V attack method, this method is based on the image model pre-trained by ImageNet. By generating adversarial samples on all frames of the video, it improves the transferability between heteromodal models and realizes the attack on the black-box video model. The core idea of ​​the I2V method is to perturb the intermediate features of the image model by inputting the video frames into the image model frame by frame, thereby perturbing the intermediate features of the video model with a high probability. In this method, I2V optimizes the adversarial perturbation of each frame so that the features of the perturbed frame and the normal frame are as orthogonal as possible in the feature space of the image model. However, the robustness of this method is poor. This is because action recognition in the video depends not only on the content of a single frame, but also on the continuity and temporal correlation between frames. The I2V method generates adversarial perturbations across modalities through the image model, but only by maximizing the difference between the perturbation map and the adversarial image feature map. Obviously, the temporal relationship between consecutive video frames is not taken into account, resulting in the perturbation only affecting a single frame and failing to produce a continuous attack effect on the entire action sequence. This limitation makes the attack perform poorly when facing complex temporal models.

[0007] Glossary: ​​Typical examples of ladder class activation mapping methods include CAM (Class Activation Mapping or ClassActivation Map) algorithm and Grad-CAM (Gradient-weighted ClassActivation Mapping) algorithm. CAM proposed the concept of class activation mapping, and Grad-CAM is an improvement on CAM. Grad-CAM generates a heat map that highlights the key areas of the image by analyzing the gradients flowing into the last convolutional layer of CNN. By calculating the gradient of the predicted class score and the feature map of the last convolutional layer, Grad-CAM determines the importance of each feature map to a specific class.

[0008] Optical flow is the instantaneous speed of the pixel movement of a moving object in space on the imaging plane. The optical flow method is a method that uses the change of pixels in the time domain in an image sequence and the correlation between adjacent frames to find the corresponding relationship between the previous frame and the current frame, thereby calculating the motion information of the object between adjacent frames. In space, motion can be described by a motion field, and on an image plane, the motion of an object is often reflected by the different grayscale distributions of different images in an image sequence. Therefore, the motion field in space is transferred to the image and represented as an optical flow field. The optical flow field is a two-dimensional vector field that reflects the trend of grayscale changes at each point on the image. It can be regarded as the instantaneous velocity field generated by the movement of grayscale pixels on the image plane. The information it contains is the instantaneous motion velocity vector information of each image point. In short, the optical flow field is a collection of optical flows for each pixel in the image. Summary of the invention

[0009] The purpose of the present invention is to provide a method for generating cross-modal adversarial samples from images to videos based on key frame recognition, which can solve the above problems, fully consider timing information, accurately recognize key frames, and generate more aggressive adversarial samples.

[0010] In order to achieve the above-mentioned purpose, the technical solution adopted by the present invention is as follows: a method for generating cross-modal adversarial samples from images to videos based on key frame recognition, comprising the following steps;

[0011] S1, get the video frame sequence , A is the total number of images in the video frame sequence, X a For the a-th frame image in X, the maximum sliding window length L is preset max , minimum value L min And stride, let the current sliding window length L=L max ;

[0012] S2, obtaining key video clips, including steps S21 to S26;

[0013] S21, traverse X according to the current sliding window length L to perform sliding window sampling. If there are K samplings in total, the kth sampling obtains the video segment P k , P k It contains L images, of which the lth image is marked as ,but , 1≤k≤K, 1≤l≤L;

[0014] S22, for P k For each image in the image, extract its channel average feature vector, and then average all channel average feature vectors to obtain P k The comprehensive feature vector of ;

[0015] S23, calculated based on ladder-class activation mapping method Contribution to classification w k ;

[0016] S24, process each video segment in sequence according to steps S22~S23 to obtain P1~P K The corresponding contribution w1~w K , and w1~w K The maximum value is marked as w max ;

[0017] S25, preset the current maximum contribution Max;

[0018] If max ≥Max, w max The corresponding video segment is marked as a key video segment, otherwise the current window length is reduced by 1, and the process goes to step S26;

[0019] S26, if the current sliding window length L ≥ L min , repeat S21~S25, if L<L min , then L=L min Time max The corresponding video clips are marked as key video clips;

[0020] S3, select key frames from key video clips, if P k For a key video segment, extracting key frames includes steps S31-S32;

[0021] S31, calculate P k The optical flow field and difference between two adjacent frames of images, where , The optical flow field is , the difference is ;

[0022] S32, preset difference threshold θ, if , then Mark as keyframe;

[0023] S4, forming a key frame set from all key frames in the key video segment;

[0024] S5, generating a corresponding adversarial sample for each key frame in the key frame set;

[0025] S6, replacing the corresponding key frames in the video frame sequence X with the adversarial sample to obtain an adversarial video sequence of the video frame sequence X.

[0026] As a preferred embodiment: in S22,k The first image in , extract its channel average feature vector The method is;

[0027] Extracted by convolutional neural network Get a feature map of dimension C×W×H , where C, W, and H are The number of channels, width and height, and then the channel average feature vector is obtained according to the following formula ;

[0028] ,

[0029] In the formula, for The value of the cth channel, 1≤c≤C;

[0030] In S22, .

[0031] As a preferred embodiment: in S23, , where y is the category label of the target in the image, is the L2 norm.

[0032] As a preference: Optical flow field , Difference According to the following formula:

[0033] ,

[0034] ,

[0035] In the formula, is the FlowNet2 optical flow algorithm, N is the total number of pixels in the image, x and y are the horizontal and vertical coordinates of the pixels respectively, is the L2 norm.

[0036] As a preferred embodiment: in S5, for any key frame in the key frame set Generating Adversarial Examples The steps include:

[0037] S51, based on random optical flow, simulates two different motion trajectories respectively. Generate two optical flow warped images , ;exist Add random perturbations to generate random perturbation samples ;

[0038] S52, Add initial adversarial perturbations to generate initial adversarial samples ;

[0039] S53, set time loss function , spatial loss function ;

[0040] S54, first minimize the time loss function update , and then minimize the spatial loss function to update ,get .

[0041] The main idea of ​​the present invention is to identify key video clips from a video frame sequence, select key frames from the key video clips, and then generate an adversarial sample for each key frame, which is used to replace the corresponding key frame in the video frame sequence, thereby generating an adversarial video sequence.

[0042] When identifying key video clips, for each video clip, the comprehensive feature vector calculated in step S22 is , integrated P k The feature information of all frames and channels in the image can more comprehensively reflect the features of the clip. Contribution to classification w k , and quantify the contribution of each segment. However, the present invention may have multiple rounds of calculation and comparison operations when identifying key video segments, and adjust the sliding window length according to the calculation and comparison results. The length of the key video segment finally obtained is not fixed, but is adjusted according to the contribution, resulting in the number of key frames for subsequent screening is also not fixed, so that the number of key frames can be adaptively adjusted under different video inputs.

[0043] When selecting key frames, we further perform optical flow calculation on the frames in the clip to find key frames with significant changes between frames. This step aims to detect the motion information between frames through optical flow and select frames with differences greater than a preset threshold as key frames.

[0044] When generating adversarial samples, we further optimize the keyframes to generate adversarial samples with strong destructive power. The entire optimization process is divided into two steps, generating different optimization targets through optical flow and random perturbations, aiming to destroy the temporal consistency of the video and the stability of frame features. One of the optimization targets corresponds to the temporal loss function Loss1, which is used to destroy temporal consistency and minimized during the optimization process. and and The cosine similarity between As different as possible from the one generated by optical flow and This is because the optical flow frame represents continuous motion in the time dimension. Destroying this similarity means that the adversarial sample breaks the temporal continuity of the video, making the generated adversarial sample more effective in destroying action recognition in the video. Another optimization goal corresponds to the spatial loss function Loss2, which is used to destroy the stability of frame features and minimize the adversarial sample during the optimization process. and The cosine similarity of More different from Through these two steps of optimization, adversarial samples can not only destroy the temporal information, but also break the stability of local features in key frames, thereby producing a stronger attack effect at the level of each frame.

[0045] When generating an adversarial video sequence, the adversarial samples generated in the above steps are used to replace the corresponding key frames in the video frame sequence. The resulting adversarial video sequence will be greatly damaged in terms of coherence and recognizability.

[0046] Compared with the prior art, the advantages of the present invention are:

[0047] (1) Improving keyframe positioning accuracy: The present invention effectively solves the problem of traditional methods locating keyframes in non-core areas through a sliding window strategy and dynamic adjustment based on feature contribution. Compared with existing methods based on single-frame gradient calculation, the present invention can fully consider the contribution of video frames and ensure that keyframes can accurately reflect important information in the video. This method makes keyframe extraction more accurate and avoids the problem of inaccurate keyframe positioning caused by ignoring timing information, thereby improving the effectiveness of countermeasures.

[0048] (2) Enhanced robustness: The sliding window-based segment selection method proposed in this paper not only considers the feature information of a single frame during feature extraction, but also combines the comprehensive features of all frames in the segment. Compared with the traditional optical flow method and single-frame gradient calculation method, this method is more adaptable to the changes of different video segments, improves the robustness and accuracy of key frame selection, and thus performs better when processing complex video data.

[0049] (3) Improved adversarial perturbation generation method: This paper proposes a new adversarial perturbation generation method by combining optical flow information and random perturbation. This method not only considers the motion information between frames, but also introduces random perturbations, thereby generating powerful adversarial samples in both the temporal dimension and the frame level. Compared with the existing I2V method, the perturbation generation method of the present invention can more effectively break the temporal consistency and frame feature stability of the video, providing a more comprehensive and lasting attack effect.

[0050] (4) Comprehensive optimization of attack effects: In the process of generating adversarial samples, the present invention effectively improves the comprehensive effect of adversarial attacks through the dual strategies of optical flow optimization and frame feature optimization. By minimizing the cosine similarity between optical flow frame versions and the cosine similarity between randomly perturbed samples, it can not only destroy the temporal continuity of the video, but also weaken the stability of the intra-frame features. This dual optimization strategy makes the generated adversarial samples more aggressive overall and can significantly improve the success rate of attacks on action recognition models.

[0051] In summary, the present invention effectively overcomes the main defects in the prior art by improving the key frame extraction method and adversarial perturbation generation technology, significantly improves the accuracy and robustness of video adversarial attacks, and can better meet the needs in practical applications. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] Figure 1 It is a flow chart of the present invention;

[0053] Figure 2 Flowchart for obtaining key video clips;

[0054] Figure 3 Flowchart for generating adversarial examples. DETAILED DESCRIPTION

[0055] The present invention will be further described below in conjunction with the accompanying drawings.

[0056] Example 1: See Figures 1 to 3 ,A method for generating cross-modal adversarial samples from image to video based on key frame recognition, comprises the following steps;

[0057] S1, get the video frame sequence , A is the total number of images in the video frame sequence, X a For the a-th frame image in X, the maximum sliding window length L is preset max , minimum value L min And stride, let the current sliding window length L=L max ;

[0058] S2, obtaining key video clips, including steps S21 to S26;

[0059] S21, traverse X according to the current sliding window length L to perform sliding window sampling. If there are K samplings in total, the kth sampling obtains the video segment P k , P k It contains L images, of which the lth image is marked as ,but , 1≤k≤K, 1≤l≤L;

[0060] S22, for P kFor each image in the image, extract its channel average feature vector, and then average all channel average feature vectors to obtain P k The comprehensive feature vector of ;

[0061] S23, calculated based on ladder-class activation mapping method Contribution to classification w k ;

[0062] S24, process each video segment in sequence according to steps S22~S23 to obtain P1~P K The corresponding contribution w1~w K , and w1~w K The maximum value is marked as w max ;

[0063] S25, preset the current maximum contribution Max;

[0064] If max ≥Max, w max The corresponding video segment is marked as a key video segment, otherwise the current window length is reduced by 1, and the process goes to step S26;

[0065] S26, if the current sliding window length L ≥ L min , repeat S21~S25, if L<L min , then L=L min Time max The corresponding video clips are marked as key video clips;

[0066] S3, select key frames from key video clips, if P k For a key video segment, extracting key frames includes steps S31-S32;

[0067] S31, calculate P k The optical flow field and difference between two adjacent frames of images, where , The optical flow field is , the difference is ;

[0068] S32, preset difference threshold θ, if , then Mark as keyframe;

[0069] S4, forming a key frame set from all key frames in the key video segment;

[0070] S5, generating a corresponding adversarial sample for each key frame in the key frame set;

[0071] S6, replacing the corresponding key frames in the video frame sequence X with the adversarial sample to obtain an adversarial video sequence of the video frame sequence X.

[0072] In this embodiment, in S22, P k The first image in , extract its channel average feature vector The method is: Extracted by convolutional neural network Get a feature map of dimension C×W×H , where C, W, and H are The number of channels, width and height, and then the channel average feature vector is obtained according to the following formula ;

[0073] ,

[0074] In the formula, for The value of the cth channel, 1≤c≤C;

[0075] In S22, .

[0076] In S23, , where y is the category label of the target in the image, is the L2 norm.

[0077] In this embodiment, the optical flow field , Difference According to the following formula:

[0078] ,

[0079] ,

[0080] In the formula, is the FlowNet2 optical flow algorithm, N is the total number of pixels in the image, x and y are the horizontal and vertical coordinates of the pixels respectively, is the L2 norm.

[0081] In S5, for any key frame in the key frame set Generating Adversarial Examples The steps include:

[0082] S51, based on random optical flow, simulates two different motion trajectories respectively. Generate two optical flow warped images , ;exist Add random perturbations to generate random perturbation samples ;

[0083] S52, Add initial adversarial perturbations to generate initial adversarial samples ;

[0084] S53, set time loss function , spatial loss function ;

[0085] S54, first minimize the time loss function update , and then minimize the spatial loss function to update ,get .

[0086] Example 2: See Figures 1 to 3 ,A method for generating cross-modal adversarial samples from image to video based on key frame recognition, comprises the following steps;

[0087] S1, get the video frame sequence , A=32, preset maximum sliding window length L max =16, minimum value L min =5, and stride = 1, let the current sliding window length L = L max ;

[0088] S2, obtaining key video clips, is the same as step S2 in embodiment 1. A more specific operation method is given below:

[0089] Round 1:

[0090] Execute step S21, traverse X according to L=16 to perform sliding window sampling, a total of K=17 sampling times, and obtain K=17 video segments P1~P 17 , each video clip contains 16 images, since ,so , and so on for other video clips;

[0091] Execute steps S22 to S24 to calculate P1 to P 17 The comprehensive feature vector of ~ ,calculate ~ Contribution to classification w1~w 17 , and w1~w 17 The maximum value is marked as w max ;

[0092] Execute step S25, preset the current maximum contribution Max=0.8, assume that w max =0.75, it means that the key video clip cannot be obtained under this window length, and the current sliding window length L needs to be reduced by 1 before the second round is performed. Therefore, the current sliding window length L in the second round is 15.

[0093] Round 2:

[0094] Execute step S21, perform sliding window sampling according to L=15, and obtain K=18 video clips in total;

[0095] Execute steps S22 to S24 to obtain 18 contributions and find the maximum value w max =0.75,;

[0096] Execute step S25, assuming that w max =0.85>Max, then w max The corresponding video segments are marked as key frame segments, otherwise the third and fourth rounds are performed until L=5, and L=5 is taken as the last round.

[0097] Last round: When L=5, if w max It is still less than Max. At this time, the next round of repetition is not performed. Instead, w in the round L=5 is set to max The corresponding video segments are marked as key frame segments.

[0098] S3, selecting key frames from key video clips;

[0099] S4, forming a key frame set from all key frames in the key video segment;

[0100] S5, generate a corresponding adversarial sample for each key frame in the key frame set. In S51, the method for generating two optical flow distortion maps for a key frame can be found in the paper: This method can be found in the paper Wei Z, Chen J, Wu Z, et al. Cross-modal transferable adversarial attacks from images to videos[C] / / Proceedings of the IEEE / CVF conference on computer vision and pattern recognition. 2022: 15064-15073.

[0101] S6, replacing the corresponding key frames in the video frame sequence X with the adversarial sample to obtain an adversarial video sequence of the video frame sequence X.

[0102] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for generating cross-modal adversarial samples from images to videos based on key frame recognition, characterized by: The steps include: S1, get the video frame sequence , A is the total number of images in the video frame sequence, X a For the a-th frame image in X, the maximum sliding window length L is preset max , minimum value L min And stride, let the current sliding window length L=L max ; S2, obtaining key video clips, including steps S21 to S26; S21, traverse X according to the current sliding window length L to perform sliding window sampling. If there are K samplings in total, the kth sampling obtains the video segment P k , P k It contains L images, of which the lth image is marked as ,but , 1≤k≤K, 1≤l≤L; S22, for P k For each image in the image, extract its channel average feature vector, and then average all channel average feature vectors to obtain P k The comprehensive feature vector of ; S23, calculated based on ladder-class activation mapping method Contribution to classification w k ; S24, process each video segment in sequence according to steps S22~S23 to obtain P1~P K The corresponding contribution w1~w K , and w1~w K The maximum value is marked as w max ; S25, preset the current maximum contribution Max; If max ≥Max, w max The corresponding video segment is marked as a key video segment, otherwise the current window length is reduced by 1, and the process goes to step S26; S26, if the current sliding window length L ≥ L min , repeat S21~S25, if L<L min , then L=L min Time max The corresponding video clips are marked as key video clips; S3, select key frames from key video clips, if P k For a key video segment, extracting key frames includes steps S31-S32; S31, calculate P k The optical flow field and difference between two adjacent frames of images, where , The optical flow field is , the difference is ; S32, preset difference threshold θ, if , then Mark as keyframe; S4, forming a key frame set from all key frames in the key video segment; S5, generating a corresponding adversarial sample for each key frame in the key frame set; S6, replacing the corresponding key frames in the video frame sequence X with the adversarial sample to obtain an adversarial video sequence of the video frame sequence X.

2. The method for generating cross-modal adversarial samples from images to videos based on key frame recognition according to claim 1, characterized in that: In S22, k The first image in , extract its channel average feature vector The method is; Extracted by convolutional neural network Get a feature map of dimension C×W×H , where C, W, and H are The number of channels, width and height, and then the channel average feature vector is obtained according to the following formula ; ， In the formula, for The value of the cth channel, 1≤c≤C; In S22, .

3. The method for generating cross-modal adversarial samples from images to videos based on key frame recognition according to claim 1, characterized in that: In S23, , where y is the category label of the target in the image, is the L2 norm.

4. The method for generating cross-modal adversarial samples from images to videos based on key frame recognition according to claim 1, characterized in that: Optical flow field , Difference According to the following formula: ， ， In the formula, is the FlowNet2 optical flow algorithm, N is the total number of pixels in the image, x and y are the horizontal and vertical coordinates of the pixels respectively, is the L2 norm.

5. The method for generating cross-modal adversarial samples from images to videos based on key frame recognition according to claim 1, characterized in that: In S5, for any key frame in the key frame set Generating Adversarial Examples The steps include: S51, based on random optical flow, simulates two different motion trajectories. Generate two optical flow warped images , ;exist Add random perturbations to generate random perturbation samples ; S52, Add initial adversarial perturbations to generate initial adversarial samples ; S53, set time loss function , spatial loss function ; S54, first minimize the time loss function update , and then minimize the spatial loss function to update ,get .

Images