A defense method and system against adversarial attacks on video classification

Abstract

This invention discloses a defense method and system against adversarial attacks on video classification. First, for each frame in the video to be defended, a corresponding composite frame is generated. The consistency metric between the composite frame and the composite frame is used to determine whether the frame is under adversarial attack, and a set of frames from the video to be defended that are not under adversarial attack is extracted. Then, the defense network is pre-trained using these un-attacked frames to initiate the first stage of video reconstruction, obtaining semi-finished video samples. Finally, a second stage of video reconstruction is performed based on these semi-finished samples, using an optimal result selection strategy to obtain the final reconstructed video. The reconstructed video can remove perturbations in the video subjected to adversarial attacks, thereby achieving the defense objective. This invention achieves superior defense performance against adversarial attacks on video classification while minimizing the impact on the classification network's performance on un-attacked samples, thus possessing higher practical value.

Description

Technical Field

[0001] This invention belongs to the field of deep learning technology, specifically relating to a defense method and system for dealing with adversarial attacks on video classification. Background Technology

[0002] In recent years, deep neural networks have demonstrated remarkable performance in many computer vision tasks, such as image recognition, video classification, object detection, and image denoising. Visual tasks are closely related to daily life, and applying deep learning technology to real-world scenarios will greatly facilitate life while saving manpower and resources. However, with the rapid development of neural networks, their weaknesses have also been exposed. Research shows that neural networks can be rendered inoperable by adversarial attacks. Adversarial examples involve attackers adding carefully designed micro-perturbations to the original data. Because the perturbation is so small that the human eye cannot perceive the change, it still causes the neural network to output incorrect results. This type of attack is called an adversarial attack. The emergence of adversarial attacks undoubtedly poses serious security challenges to the application scenarios of deep learning.

[0003] Of course, the existence of attacks on neural networks also implies the existence of corresponding defense methods. In the early years of deep learning, most applications of neural networks were in the image domain, so research on attacks and defenses against neural networks largely focused on tasks such as image classification. Currently, defense methods for image domains have achieved excellent results, and the security of neural networks applied to image tasks is no longer a major concern. However, with the rapid growth of video data on the internet, some video domain tasks have become increasingly important, and the application scenarios of deep learning have gradually shifted to the video domain. Recent studies have shown that, similar to the image domain, neural networks applied to video classification are also susceptible to adversarial attacks and cannot function properly. To date, only a few defense methods for adversarial attacks on video classification have been studied, and these methods have relatively low defense accuracy and the quality of reconstructed images. The defense performance for video classification still needs further improvement. Summary of the Invention

[0004] The purpose of this invention is to overcome the shortcomings of the prior art and provide a defense method and system for video classification adversarial attacks, so as to solve the problem that neural networks applied to video classification are attacked by adversarial examples and cannot work properly, and that there is a lack of defense methods against video classification adversarial attacks.

[0005] To achieve the above objectives, the present invention employs the following technical solution:

[0006] A defense method against adversarial attacks on video classification includes:

[0007] Each frame in the video to be defended is extracted as a video frame, and a composite frame corresponding to each video frame is generated. The video type of the video frame and the video type of the composite frame corresponding to the video frame are determined by a classification network. Combined with a consistency function, it is determined whether the video frame is subjected to adversarial attack. The frames in the video to be defended that are not subjected to adversarial attack are extracted to form a frame set.

[0008] A frame is randomly selected from the set of frames that have never been subjected to adversarial attacks as the initial frame, and a pre-training video is built based on the initial frame; a random noise video is built, the random noise video and the pre-training video have the same number of frames, and both have the same number of frames as the video to be defended; the random noise video is input into the defense network, and the pre-training video and the video to be defended are used as targets in turn. Through the iteration of the defense network, the random noise video is trained into a semi-finished video; during the generation of the semi-finished video, the main mode video with the highest metric is selected as the semi-finished video.

[0009] The semi-finished video is input into the defense network. The video to be defended is the target. Through the iteration of the defense network, the video with the highest metric is selected as the final reconstructed video.

[0010] The final reconstructed video is classified using a classification network.

[0011] A further improvement of the present invention is that:

[0012] Preferably, the method for obtaining the composite frame is as follows: each video frame is set as the current frame, the optical flow information between the current frame and the previous frame is calculated using the Gunnar Farneback algorithm, Gaussian noise is added to the optical flow information, and the optical flow information with added Gaussian noise is added to the previous frame to obtain the composite frame of the current frame.

[0013] Preferably, the consensus function is:

[0014]

[0015] Among them, Y t (m) and E represents the probability that a video frame and a composite frame are predicted to be of class m; if E < 0.5, the video frame is considered not to have been attacked; otherwise, the video frame is considered to have been attacked.

[0016] Preferably, during the generation of semi-finished videos, the iteration of the defense network is divided into two stages: the first stage uses pre-trained videos as training targets, and the second stage uses videos to be defended as training targets.

[0017] The loss function for the first-stage defense network is:

[0018]

[0019] Where L represents the L1 distance, and h(.;θ) is the defense network. main With h(.;θ) minor These represent the main mode video and the secondary mode video output by the classification network, respectively.

[0020] The loss function for the second-stage defense network is:

[0021]

[0022] Where ⊙ represents the dot product operation, L represents the L1 distance, and P t For defense video, O main Main mode video and O minor For secondary mode video, C t This is a confidence plot.

[0023] Preferably, the confidence graph for each frame is:

[0024]

[0025] For pixel x, if P in the i-th iteration t (x) compared to the secondary mode Closer to the main mode Set the value at the corresponding position on the confidence plot to 1, otherwise set it to 0.

[0026] Preferably, in each iteration of the defense network, a master mode video is generated, and the classification label of the master mode video is obtained through a classification network. output A histogram is generated using the category tags of all main mode videos. The true tags are then obtained by combining the histogram with the following formula:

[0027]

[0028] Here, l1 and l2 are two classification labels, and p1 is the frequency of the label corresponding to l1. The metric is obtained based on the real label.

[0029] Preferably, the metric is the sum of the confidence level of the label and the structural similarity, where the structural similarity is the structural similarity between the output video and the original video.

[0030] Preferably, during the video reconstruction process, the loss function of the defense network is:

[0031]

[0032] Where L represents the L1 distance, P t For the video to be defended, SSIM is the structural similarity function, O * This is a semi-finished video that has been reconstructed.

[0033] Preferably, the defense network is a fully convolutional network, U-Net.

[0034] A defense system against video classification adversarial attacks, characterized in that it includes:

[0035] The judgment unit is used to extract each frame in the video to be defended as a video frame, generate a composite frame corresponding to each video frame; determine the video type of the video frame and the video type of the composite frame corresponding to the video frame through a classification network, and determine whether the video frame is subjected to adversarial attack by combining a consistency function, and extract the frames in the video to be defended that are not subjected to adversarial attack to form a frame set.

[0036] A semi-finished video generation unit is used to randomly select a frame as an initial frame from the set of frames that have never been subjected to adversarial attacks, and to build a pre-trained video based on the initial frame; to build a random noise video, wherein the random noise video and the pre-trained video have the same number of frames, and both have the same number of frames as the video to be defended; the random noise video is input into the defense network, and the random noise video is trained into a semi-finished video by iterating through the defense network, using the pre-trained video and the video to be defended as targets in turn; during the generation of the semi-finished video, the main mode video with the highest metric is selected as the semi-finished video.

[0037] The video reconstruction generation unit is used to input the semi-finished video into the defense network. Taking the video to be defended as the target, the video with the highest metric is selected as the final reconstructed video through the iteration of the defense network.

[0038] The classification unit is used to classify the final reconstructed video through a classification network.

[0039] Compared with the prior art, the present invention has the following advantages:

[0040] This invention discloses a defense method against adversarial attacks in video classification. First, the video to be defended is acquired. Then, a fully convolutional network, U-Net, is constructed as the defense network and randomly initialized. Next, for each frame in the video (except the first frame), a corresponding synthetic frame is generated. The consistency metric between the original frame and its synthetic frame determines whether the frame has been attacked, and a set of frames from the video that have not been attacked is extracted. Subsequently, the defense network is pre-trained using these un-attacked frames to initiate the first stage of video reconstruction, obtaining semi-finished video samples. Finally, a second stage of video reconstruction is performed based on these semi-finished samples, using an optimal result selection strategy to obtain the final reconstructed video. The reconstructed video can clean up the perturbations in the video that has been attacked, thus achieving the defense objective. This invention achieves superior defense performance against adversarial attacks in video classification while minimizing the impact on the classification network's performance on un-attacked samples, making it more practical. Compared to existing methods, this method has higher defense accuracy and better reconstructed image quality, making it more practical in real-world video classification scenarios.

[0041] This invention also discloses a defense system against adversarial attacks on video classification, comprising a judgment unit, a semi-finished video generation unit, a reconstructed video generation unit, and a classification unit. Through the cooperation between the modular units, the system enables the final reconstructed video to remove perturbations from the video subjected to the adversarial attack. This system exhibits superior defense against adversarial attacks on video classification, with improved classification accuracy, classification confidence of the reconstructed video, and image quality compared to existing methods. The defense algorithm of this invention has minimal impact on the classification accuracy of clean, unattacked videos, making it more practical in real-world application scenarios where the nature of the input sample's attack is unknown. Attached Figure Description

[0042] Figure 1 This is a flowchart of the method of the present invention;

[0043] Figure 2 This is a flowchart for determining whether a video frame to be defended is subject to adversarial attacks in this invention. Detailed Implementation

[0044] The present invention will now be described in further detail with reference to the accompanying drawings:

[0045] In the description of this invention, it should be noted that the terms "center," "upper," "lower," "left," "right," "vertical," "horizontal," "inner," and "outer," etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are used only for the convenience of describing the invention and for simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on the invention. The terms "first," "second," and "third" are used for descriptive purposes only and should not be construed as indicating or implying relative importance. Furthermore, unless otherwise explicitly specified and limited, the terms "installed," "connected," and "linked" should be interpreted broadly. For example, they can refer to a fixed connection or a detachable connection; they can refer to a direct connection or an indirect connection through an intermediate medium; they can refer to the internal connection of two elements. Those skilled in the art can understand the specific meaning of the above terms in this invention based on the specific circumstances.

[0046] This invention discloses a defense method against adversarial attacks on video classification. The entire defense process involves reconstructing the input video using a defense network before it is fed into the video classification network, and then feeding the reconstructed result into the classification network. This achieves the defense effect (if the input video is under attack, the correct category label will be output after reconstruction and purification). The specific defense method includes the following steps:

[0047] Step 1: Obtain the video to be defended, construct a fully convolutional network U-Net as the defense network, and randomly initialize the network parameters without explicit regularization;

[0048] Step 2: For each frame (except the first frame) in the video to be defended, generate its corresponding composite frame. Determine whether the frame has been subjected to adversarial attacks based on the consistency metric between the frame and its composite frame, and extract the set of frames in the video to be defended that have not been subjected to adversarial attacks. For example... Figure 2 The flowchart shown is for determining whether a video frame to be defended is subject to adversarial attacks. The specific process is as follows:

[0049] Step 2.1: For each frame (except the first frame) in the video to be defended, acquire the image of the previous frame and calculate the optical flow between the two frames using the Gunnar Farneback algorithm; if the video to be defended has N frames, then N-1 optical flows can be obtained; assuming the current frame is X... t The previous frame was X. s X s With X t The optical flow between them is defined as:

[0050] O F =(Δu,Δv) (1)

[0051] Δu and Δv describe the pixel displacement in two directions, respectively.

[0052] Step 2.2, in optical flow information O F Add Gaussian noise α ~ N(0, 0.002) to the top. 2 ), and then through X s With O F Generate composite frames The optical flow information with Gaussian noise to be added is superimposed on the previous frame to obtain a composite frame, resulting in a total of N-1 composite frames; the synthesis formula is as follows:

[0053] (u,v)=(i+Δu(i,j),j+Δv(i,j))(2)

[0054] Where (u,v) represents The pixel (i,j) represents X. s The pixel values ​​are determined by the fact that (Δu, Δv) can be fractions, meaning (u, v) might not lie on integer coordinates. Therefore, bilinear sampling is needed to sample the pixel intensity, as shown in the following formula:

[0055]

[0056] Where N(u,v) represents the index of the 4 nearest neighbor at (u,v), and X(i,j) represents the pixel value at (i,j).

[0057] Step 2.3: Set the current frame X t Its corresponding composite frame Simultaneously, the data is fed into a classification network, which classifies the video. The classification network outputs the probability that the video belongs to a certain class, thus initially determining its category. Both the current frame and the synthesized frame are processed by the classification network to obtain their respective classification results. The rationale here is that if the target frame has not been attacked, the two probability distributions will be very close, maintaining consistency. Therefore, a consistency function can be used as a criterion. Thus, the classification results of the current frame and the synthesized frame are input into the consistency function to calculate their consistency metric E. The result of this consistency metric is used to determine whether the current frame has been subjected to an adversarial attack. The definition of the consistency function is as follows:

[0058]

[0059] Where Y t (m) and E represents the probability that the current frame and its corresponding synthetic frame are predicted to be of class m. If E < 0.5, it indicates that the probability distribution has strong consistency, and the detected frame X can be considered to be of class m. t It is not under attack; otherwise, it is considered to have been attacked.

[0060] Step 2.4: Use the above steps to determine whether all frames (except the first frame) have been subjected to adversarial attacks, and extract the set of frames that have not been subjected to adversarial attacks.

[0061] Step 3: Define the loss function of the defense network, using frames that have not been subjected to adversarial attacks (generated video). The defense network is pre-trained using frames from the video (in the image), and the first stage of video reconstruction is initiated to obtain semi-finished video samples after reconstruction. Step 3 is detailed as follows:

[0062] Step 3.1: Set the number of output channels of the defense network to 6 (because an RGB video frame has 3 color channels, so it is set to 6 here), so that it outputs two video samples at the same time. The video composed of the first three channels is called the main mode video O. main The video composed of the last three channels is called sub-mode video O. minor ;

[0063] Step 3.2: Generate a random noise video with the same size and frame number as the video to be defended, as input to the defense network, assuming it is I; randomly select a frame from the frame set in Step 2.4 as the initial frame. If the frame set in Step 2.4 is empty, it means that all frames of the original video have been attacked, so randomly select a frame from the original video as the initial frame. Preferably, considering that the last frame of the video is likely to be less affected by the attack, the last frame is selected as the initial frame; generate a video with the same number of frames as the input video using the initial frame as content. This serves as a pre-training video. Specifically, the initial frame is considered an unattacked frame, and it is copied N times to obtain a video in which all frames have identical content.

[0064] Step 3.3: Define the loss function of the defense network, using both network parameters and the input noisy video as optimization parameters, and start iterative optimization for a set number of rounds. Preferably, the set number of rounds here is 1000. During the iteration process, the main mode video output by the defense network is saved every 10 rounds, and it is simultaneously fed into the classification network to obtain the corresponding classification result and saved.

[0065] For the first 6 iterations, the loss function is set as follows:

[0066]

[0067] Where L represents the L1 distance, and h(.;θ) is the defense network U-Net, h(.;θ) main With h(.;θ) minor These represent the main mode video and the secondary mode video output by the network, respectively. Through the first six iterations, the random noise video I is continuously transformed into video... As the network gets closer, its parameters are continuously optimized. This process also creates a difference between the primary and secondary output modes. The primary mode output will be more biased towards clean, unattacked videos. Through these six iterations, the primary mode output will increasingly fall within the decision domain of clean videos in subsequent iterations.

[0068] For the remaining iterations, first, based on the main mode video O... main and sub-mode video O minor Calculate the confidence map C for each frame. t The formula is as follows:

[0069]

[0070] Where L represents the distance L1, P t Let x be the video to be defended against, and x be a pixel in the image. For pixel x, if P... t (x) compared to the secondary mode Closer to the main mode Then set the value at the corresponding position on the confidence plot to 1, otherwise set it to 0.

[0071] In the (i+1)th iteration, the confidence graph is used to update the training loss function:

[0072]

[0073] Where ⊙ represents dot product. During the iteration process, the main mode video output by the defense network is saved every 10 rounds, and it is fed into the classification network to obtain the corresponding classification results and save them. This allows us to obtain the classification of each main mode video, and after the update, we get the classification results of 100 main video modes.

[0074] The principle behind the above iteration is as follows: if the number of iterations is large enough, the output of the main mode will eventually be exactly the same as the target video (because a loss function is defined). If the target video is the attacked video, then the final output of the main mode is also an attacked video. Therefore, before iterating overfitting, a video is selected that is sufficiently similar to the target video but does not contain the noise in the target video, in order to achieve the purpose of defense (restoring the video while ensuring that the restored video can be correctly classified).

[0075] Step 3.4: Generate a frequency histogram of the classification results. Based on the distribution of the histogram, infer the correct classification label of the input video. Filter out all videos in the main mode output video that are classified with that label. Use the sum of the confidence of the output result with respect to the label and the structural similarity (SSIM) between the output result and the original video as the metric. Select the video with the highest metric in the output results as the semi-finished video for reconstruction.

[0076] Assume the real tag of the video to be defended is l true The actual label output by the network is l output Until this step, it is still difficult to determine whether the input video has been attacked, therefore... output These may not necessarily be the true labels of the aforementioned 100 videos; therefore, it is necessary to consider the histogram results and the output labels. output To infer the true label true What is it? The two labels with the highest frequencies in the histogram are l1 and l2, and the frequency corresponding to label l1 is p1. The true label l is selected. true The formula is as follows:

[0077]

[0078] Filter all videos with the category tag l from the iteratively saved master mode video set. true The video is classified using the confidence score of the label in the output video classification results and the structural similarity between the output video and the original video as a metric. The video with the highest metric in the output results is selected as the semi-finished reconstruction video O. * .

[0079] In the above process, if the input video is an attacked video, then during the iteration process, it is highly likely that both "clean labels" and "attacked labels" will account for the majority of all predicted labels (i.e., the histogram frequency). Conversely, if the input video is a clean video, then during the iteration process, "clean labels" will likely account for the vast majority (over 70%). Therefore, we can use this phenomenon to classify and discuss to determine the true label, resulting in the following two scenarios:

[0080] (1) p1>0.7 means that there is a predicted label that accounts for more than 70% of all labels, which is considered to be a "clean label" (i.e. the input video is considered to be clean).

[0081] (2) If p1 < 0.7, it means that the proportion of p1 did not achieve an overwhelming result, so the video is considered to be attacked. Therefore, l output It's an incorrect label. Based on the above process, the correct label is one of the top two labels in terms of percentage. If l output If the percentage is the highest, then L2 is considered the correct label. If L... output If its percentage isn't the highest, then it's most likely the second highest, so we assume L1 is the true label. (Of course, there's a small probability that L1 might exist.) output This applies to the third and subsequent rankings. In this case, it's difficult to directly determine which label is correct. However, the L1 label, which has the highest percentage, still has a greater probability of being the correct label. Therefore, this situation is also combined with the one mentioned above.

[0082] Step 4: Based on the semi-finished video samples, perform the second stage of video reconstruction. Use the optimal result selection strategy to obtain the best reconstructed sample. The reconstructed result can purify the perturbations in the video that has been subjected to adversarial attacks, thereby achieving the purpose of defense.

[0083] The specific steps of step 4 are as follows:

[0084] Step 4.1: Set the number of output channels of the defense network to 3, so that it outputs only one video sample, and at the same time randomly initialize the network parameters;

[0085] Step 4.2: Take the semi-finished video O obtained in Step 3. * As input to the defense network;

[0086] Step 4.3: Define the loss function of the defense network as follows:

[0087]

[0088] Here, SSIM is the structural similarity function. The network parameters are used as optimization parameters, and a total of 500 iterations of optimization are performed. During the iteration process, the output of the defense network is saved every 10 rounds, and simultaneously fed into the classification network to obtain the corresponding classification result and save it; where L represents the L1 distance, and h(.;θ) is the defense network U-Net, h(.;θ) main The main mode video represents the network output, and SSIM is the structural similarity function.

[0089] Step 4.4: Filter out all videos in the output results that are classified as the original label. Use the confidence of the output results for that label and the sum of the structural similarity (SSIM) between the output results and the original video as the metric, and select the video with the highest metric in the output results as the final reconstructed video.

[0090] The reason the attacked video can be successfully attacked in the method of this invention is because a perturbation is added to the clean video, causing the classifier to misclassify it. The reconstructed video is very similar to the attacked video. The reason this method can correctly classify the video is that it destroys the original perturbation structure, preventing it from perturbing the classifier and thus enabling correct classification.

[0091] This invention also discloses a defense system against adversarial attacks on video classification, comprising:

[0092] The judgment unit is used to extract each frame in the video to be defended; generate a composite frame corresponding to each frame; determine the video type of the video frame and its corresponding composite frame through a classification network, and determine whether the video frame is subjected to adversarial attack by combining a consistency function; and extract the frames in the video to be defended that are not subjected to adversarial attack to form a frame set.

[0093] A semi-finished video generation unit is used to randomly select a frame from the frame set as an initial frame, and build a pre-trained video based on the initial frame; to build a random noise video, wherein the number of frames in the random noise video and the pre-trained video are equal, and both are equal to the number of frames in the video to be defended; the random noise video is input into the defense network, and the random noise video is trained into a semi-finished video by iterating through the defense network, using the pre-trained video and the video to be defended as targets in turn; during the semi-finished video generation process, the main mode video with the highest metric is selected as the semi-finished video.

[0094] The video reconstruction generation unit is used to input the semi-finished video into the defense network. Taking the video to be defended as the target, the video with the highest metric is selected as the final reconstructed video through the iteration of the defense network.

[0095] The classification unit is used to classify the final reconstructed video through a classification network.

[0096] The method of the present invention will be further described below with reference to the accompanying drawings and specific embodiments.

[0097] Example

[0098] Taking a video sample as an example, the process of performing defense on the video sample includes:

[0099] Construct a video classification network model, which can be a combination of CNN and LSTM. Construct a defense network model, U-Net, and initialize it randomly.

[0100] The video samples are fed into a classification network to obtain the classification label "BabyCrawling". The video has 40 frames. Frames 2 to 40 are selected as target frames. The optical flow between the target frame and its preceding frame is calculated. The composite frames from frame 2 to 40 are obtained using the algorithm from step 2. The target frames (frames 2 to 40) and their corresponding composite frames are then fed into the classification model for classification. Based on the consistency function, frames 3, 9, 23, and 35 are identified as clean frames. Frame 3 is extracted, and a 40-frame video is generated using this frame as content.

[0101] Generate a random noise video I with the same size and number of frames (40 frames) as the video sample to be defended, and start 1000 iterations according to the algorithm in step 3. The first 6 rounds select The target videos were used for pre-training. The output of the main mode was saved every 10 rounds during the iteration process, ultimately yielding 100 main mode videos. These 100 videos were fed into a classification network to obtain classification labels. Histogram analysis of the labels revealed that the label "Archery" had a frequency of 52%, ranking first, while the label "BabyCrawling" had a frequency of 41%, ranking second. Given the significant proportions of these two labels, it was determined that the original video samples had been attacked, and their original classification label "BabyCrawling" was identified as incorrect, while "Archery" was deemed the correct classification label. Forty-one video samples classified as "Archery" were extracted from all the main mode videos. Using the confidence score of the "Archery" label in the main mode video classification results and the sum of the structural similarity (SSIM) between the main mode video and the original video samples as metrics, the video with the highest SSIM in the output results was selected as the semi-finished reconstruction video.

[0102] The iterative process in step 4 is performed on the semi-finished video for a total of 500 rounds, with 50 output video samples saved every 10 rounds. These 50 output videos are fed into a classification network, and the confidence score of the "Archery" label in the classification results of the output videos, along with the sum of the structural similarity (SSIM) between the output videos and the original videos, is used as the metric. The video with the highest SSIM score in the output results is selected as the final reconstructed video. Finally, verification confirms that the original output video is correctly classified as "Archery," which is the same as the classification result of the reconstructed video. Therefore, the algorithm successfully achieves the defense effect against adversarial attacks.

[0103] Table 1 shows a comparison of the defense performance of the algorithm in this invention and existing algorithms on video datasets subjected to adversarial attacks. Ours represents the defense algorithm in this invention, while Adaptive, OTD, and Adaptive+OTD are existing defense algorithms (Wei, X.; Zhu, J.; and Su, H. 2019. Sparse adversarial perturbations for videos. In The Thirty-Third AAAI Conference on Artificial Intelligence (AAAI-19), 8973–8980.). Three evaluation metrics are used for the results: CA represents the classification accuracy after defense, TLP represents the mean confidence score of the reconstructed sample for the correct category, and SSIM represents the mean structural similarity between the reconstructed sample and the original video. Higher values ​​for these three metrics indicate better defense performance. The present invention outperforms existing methods in defending against attacks on video datasets.

[0104] Table 1 Comparison of the defensive performance of the present invention and existing algorithms on video datasets subjected to adversarial attacks.

[0105] Adaptive OTD Adaptive+OTD Ours CA 0.7197 0.4848 0.8182 0.8788 TLP 0.6843 0.3993 0.7047 0.8251 SSIM 0.9066 0.8050 0.8246 0.9181

[0106] Table 2 shows a comparison of the classification results of the algorithm in this invention and existing algorithms when applied to unattacked video datasets. The algorithm of this invention also outperforms existing methods when processing unattacked videos, and is more practical in real-world application scenarios where it is unknown whether the input samples have been attacked.

[0107] Table 2 Comparison of classification results of the present invention and existing algorithms when applied to unattacked video datasets.

[0108] Adaptive OTD Adaptive+OTD Ours CA 0.9389 0.7023 0.9237 0.9466 TLP 0.8557 0.5991 0.8014 0.8570 SSIM 0.9089 0.8011 0.8252 0.9215

[0109] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A defensive method against adversarial attacks on video classification, characterized in that, include: Extract each frame from the video to be defended as a video frame, and generate a composite frame corresponding to each video frame; The video type of the video frame is determined by the classification network, and the video type of the corresponding composite frame is determined by the consistency function. Then, it is determined whether the video frame is subjected to adversarial attack, and the frames that are not subjected to adversarial attack in the video to be defended are extracted to form a frame set. Randomly select a frame from the set of frames that have never been subjected to adversarial attacks as the initial frame, and build a pre-trained video based on the initial frame; A random noise video is created, wherein the number of frames in the random noise video and the pre-training video are equal, and the number of frames in both is equal to that in the video to be defended; the random noise video is input into the defense network, and the pre-training video and the video to be defended are used as targets in turn. Through the iteration of the defense network, the random noise video is trained into a semi-finished video; during the generation of the semi-finished video, the main mode video with the highest metric index is selected as the semi-finished video. During the generation of semi-finished videos, the iteration of the defense network is divided into two stages: the first stage uses pre-trained videos as training targets, and the second stage uses videos to be defended as training targets. The loss function for the first-stage defense network is: （5） in L represent L 1 distance, To defend the network, and These represent the main mode video and the secondary mode video output by the classification network, respectively. The loss function for the second-stage defense network is: (7) in, Represents dot product operation. L represent L 1 distance, For defense video, O main Main mode video and O minor For secondary mode video, C t This is a confidence graph; in each iteration of the defense network, a master mode video is generated, and the classification label of the master mode video is obtained through a classification network. l output A histogram is generated using the category tags of all main mode videos. The true tags are then obtained by combining the histogram with the following formula: （8） in, l 1 and l 2 represents two category tags. for l 1. Corresponding label frequency; 2. Metrics obtained based on real labels; The semi-finished video is input into the defense network. Taking the video to be defended as the target, the video with the highest metric is selected as the final reconstructed video through the iteration of the defense network. The defense network is a fully convolutional network U-Net. The final reconstructed video is classified using a classification network.

2. The defense method against video classification adversarial attacks according to claim 1, characterized in that, The method for obtaining the composite frame is as follows: each video frame is set as the current frame, the optical flow information between the current frame and the previous frame is calculated using the Gunnar Farneback algorithm, Gaussian noise is added to the optical flow information, and the optical flow information with added Gaussian noise is added to the previous frame to obtain the composite frame of the current frame.

3. The defense method against video classification adversarial attacks according to claim 1, characterized in that, The consensus function is: （4） in, and The representative video frame and the composite frame were predicted as The probability of a class; if If a video frame is deemed not to have been attacked, it is considered not to have been attacked; otherwise, it is considered to have been attacked.

4. The defense method against video classification adversarial attacks according to claim 1, characterized in that, The confidence graph for each frame is as follows: （6） For pixels x If in the i-th iteration Compared to secondary mode Closer to the main mode Set the value at the corresponding position on the confidence plot to 1, otherwise set it to 0.

5. A defense method against video classification adversarial attacks according to claim 1, characterized in that, The metric is the sum of the label confidence score and the structural similarity score, where the structural similarity score is the structural similarity between the output video and the original video.

6. The defense method against video classification adversarial attacks according to claim 1, characterized in that, During the video reconstruction process, the loss function of the defense network is: （9） in, L represent L 1 distance, For the video to be defended, SSIM is the structural similarity function. This is a semi-finished video that has been reconstructed.

7. A defense system for implementing the defense method of claim 1 to counter video classification adversarial attacks, characterized in that, include: The judgment unit is used to extract each frame in the video to be defended as a video frame and generate a composite frame corresponding to each video frame. The video type of the video frame is determined by the classification network, and the video type of the corresponding composite frame is determined by the consistency function. Then, it is determined whether the video frame is subjected to adversarial attack, and the frames that are not subjected to adversarial attack in the video to be defended are extracted to form a frame set. The semi-finished video generation unit is used to randomly select a frame from the set of frames that have never been subjected to adversarial attacks as the initial frame, and to build a pre-trained video based on the initial frame. A random noise video is created, wherein the number of frames in the random noise video and the pre-training video are equal, and the number of frames in both is equal to that in the video to be defended; the random noise video is input into the defense network, and the pre-training video and the video to be defended are used as targets in turn. Through the iteration of the defense network, the random noise video is trained into a semi-finished video. During the process of generating the semi-finished video, the main mode video with the highest metric index is selected as the semi-finished video. The video reconstruction generation unit is used to input the semi-finished video into the defense network. Taking the video to be defended as the target, the video with the highest metric is selected as the final reconstructed video through the iteration of the defense network. The classification unit is used to classify the final reconstructed video through a classification network.

Images