A Bit-Plane-Based Auxiliary Single Target Tracking Countermeasure Defense Reconstruction Method

Abstract

The present invention discloses an auxiliary single target tracking adversarial defense reconstruction method based on a bit plane, comprising the steps of: obtaining a video frame sequence composed of clean samples; generating an adversarial sample sequence; generating a quantized sample sequence by a bit plane layering method; constructing a defense network that outputs a defense sample sequence; obtaining a tracker; training the defense network and the tracker together to update the network parameters of the defense network to obtain a defense reconstruction model; generating a defense sample sequence of a video frame sequence to be identified based on the defense reconstruction model for tracking by the tracker. The present invention enhances the tracker's defense capability against unknown disturbances without reducing the accuracy of clean samples, can improve the accuracy of target tracking, and solve the problems of high training cost and high computational cost of existing defense methods. And its defense effectiveness can be transferred to other trackers based on CNN or a combination of CNN and Transformer in a plug-and-play manner, with good portability.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to an auxiliary single target tracking adversarial defense reconstruction method based on a bit plane. Background Art

[0002] Video single target tracking is one of the important tasks in computer vision. It refers to using a single target tracker to extract the target visual features in a continuous video frame sequence based on a target area specified in the initial frame, and predict the location of the target in the subsequent video frame sequence. Existing single target trackers are mainly divided into three categories: CNN-based trackers, CNN and Transformer-based trackers, and Transformer trackers.

[0003] Adversarial attacks mainly add subtle perturbations that are difficult for people to detect to clean samples, causing the normally trained deep learning model to output highly confident erroneous predictions. Adversarial attack technology in target tracking tasks is to use a carefully designed objective function to guide the network model or iterative method to generate imperceptible adversarial perturbations and add them to the video frame, so that advanced trackers lose the tracking target during normal tracking. Such as CSA attack, IoU attack, DFA attack, etc. Adversarial defense aims to eliminate the threat brought by adversarial samples. Currently, there are the following methods:

[0004] Sravanti et al. used bit plane-based quantization technology to achieve adversarial defense in the field of image classification. They obtained high-bit plane images through quantization technology and sent them to the neural network for training together with normal clean images. By designing the loss function, the decision consistency of the neural network on high-bit plane images and normal clean images (i.e., the regularization factor Bit Plane Feature Consistency, BPFC) was strengthened, so that the neural network made more use of high-bit plane information to form a rough prediction and only used low-bit plane information to refine the prediction. This method does not require adversarial samples for training, and the computational cost is lower than that of adversarial training, which can improve the robustness of the network. However, this method loses detailed information, and its defense effect is still different from that of adversarial training and other methods. At present, this bit plane-based defense method is mainly used in image classification tasks, and the defense effect is not good when it is directly transferred to target tracking tasks. There are two reasons. First, there is a difference between the network structure of target tracking and the network structure of the classifier. Second, the input of the target tracking task is a continuous image sequence, which contains temporal information that is not present in the image classification task.

[0005] Achyut et al. constructed a representative support dataset of ten categories for the MNIST dataset. When defending against adversarial attacks, the high plane of the adversarial sample is retained and the low plane of the adversarial sample is replaced with the low plane in the support dataset of the same category to achieve the purpose of defense. This method has poor transferability and practicality, and its effectiveness in applying it to different datasets is questionable.

[0006] Liu et al. used different bit plane combinations to input bit plane classifiers for convolution, and then integrated them with the target DNN model for prediction to defend against adversarial attacks. However, this method has high computational and training costs. Summary of the invention

[0007] The purpose of the present invention is to provide a method for solving the above-mentioned problems, reducing the accuracy of clean samples while improving the ability to defend against attacks with low training cost, and an auxiliary single target tracking adversarial defense reconstruction method based on a bit plane.

[0008] In order to achieve the above object, the technical solution adopted by the present invention is as follows: an auxiliary single target tracking countermeasure defense reconstruction method based on a bit plane comprises the following steps;

[0009] S1, obtain a video frame sequence consisting of clean samples , where the clean sample of the i-th frame is I i , its marking , , I i The classification label and regression label of n is the total number of clean samples;

[0010] S2, in I i Add disturbance Get I i Corresponding adversarial examples , forming an adversarial sample sequence ;

[0011] S3, through the bit plane layering method Quantify and get I i Corresponding quantized samples , forming a quantized sample sequence ;

[0012] S4, construct a defense network M rec , used for input , output defense sample sequence ;

[0013] So the defense network M rec Based on the Unet network, when i=1, Output defense samples via Unet network , when i ≥ 2, Input Unet network and fuse the quantized samples of the previous frame Post-output defense sample ;

[0014] S5, obtain a tracker M based on the twin network F , the M F The feature extraction network has J layers, and each layer outputs a feature map;

[0015] The M F For input and , and , M F Output The corresponding feature map of the j-th layer feature extraction network , classification labels and regression labels ,right , M F Output The corresponding feature map of the j-th layer feature extraction network , predict the classification label and predict regression labels , 1≤j≤J;

[0016] S6, M rec and M F Form the total network and construct the total loss function L Tri , including feature loss L fea , defense sample classification loss L cls and defense sample regression loss L reg ;

[0017] S7, preset iteration rounds, use and Train the total network and freeze M during training F The network parameters are set to minimize L Tri Update M rec The network parameters of M are adjusted until the iteration round is reached. rec As a defensive reconstruction model ;

[0018] S8, obtain the video frame sequence to be identified, generate the corresponding quantized sample sequence according to steps S2 and S3, and then reconstruct the model through defense Generate defense sample sequences for tracking by the tracker.

[0019] As a preference: S3 Quantize and obtain quantized samples Specifically:

[0020] Through the bit plane layering method It is divided into the first bit plane b1 to the eighth bit plane b8; the quantization factor k is preset according to the formula Generate quantized samples , where 1≤k<8, j is the jth bit plane The index of .

[0021] Preferably, k=5.

[0022] Preferably, step S4 includes steps S41-S42;

[0023] S41, obtain a Unet network, including a contraction path and an expansion path, to quantize the samples , the contraction path generates the first down-sampling feature ~ the third down-sampling feature in turn ~ , the expansion path generates the third up-sampled feature ~ the first up-sampled feature in turn ~ ,and and , and , and , skip connections;

[0024] S42, yes When i=1, the defense sample is directly output through the Unet network ;

[0025] When i=2~n, the jump connection method of the Unet network is improved, and , and , and Skip connection to generate defense samples ,in, , , are the clean samples I of the i-1th frame respectively. i-1 Corresponding quantized samples The first down-sampling feature, the second down-sampling feature, and the third down-sampling feature.

[0026] As a preference: in S6, the feature loss L fea , total loss function L Tri According to the following formula:

[0027] ,

[0028] ,

[0029] In the formula, α, β, and γ are L fea , L cls and L reg The hyperparameter balancing factor.

[0030] Preferably: α=0.4, β=0.3, γ=0.3.

[0031] As a preference: the tracker M F Including SiamRPN, SiamRPN++, TransT, and STARK.

[0032] As a preference: In S6, the defense sample classification loss L cls Using the cross entropy loss function, the defensive sample regression loss L reg Smooth L1 loss is used.

[0033] Preferably, the training method for one iteration in S7 includes:

[0034] S71, will Enter M rec generate ;

[0035] S72, will , Enter M F Calculate the total loss function L Tri ;

[0036] S73, using ADAM optimizer through L Tri Update M rec network parameters.

[0037] About bit plane slicing: Bit Plane Slicing in English. For an image, the pixel value of each point can be converted into its binary representation. Since the minimum and maximum values ​​of image pixels are between 0 and 255, a total of 8 bits are required to represent the binary representation of pixel values. Therefore, the image can be divided into 8 bit planes, and the bit plane is a matrix with element values ​​of 0 or 1, where the first bit plane b1 contains the lowest order bits of all pixels in the image, and the eighth bit plane b8 contains the highest order bits of all pixels in the image. In adversarial attacks on target tracking, the added imperceptible disturbances are usually located in the low bit planes. Therefore, the present invention removes the first k low bit planes, retains the high bit planes, and forms quantized samples. From the formula, we can see that the higher the bit plane is, the greater the weight is, and the more is retained in the quantized sample. Through this formula, more main visual information of the high bit plane can be retained and most of the disturbance can be removed.

[0038] The trackers described in the present invention include SiamRPN, SiamRPN++, and TransT, all of which are commonly used trackers in visual target tracking. For example, SiamRPN is a tracker that adds RPN (Region Proposal Network) on the basis of SiamFC for single target tracking. SiamRPN++ is an improved tracker based on SiamRPN. TransT (Transformer Tracking) is a feature fusion model based on Transformer, which significantly improves the accuracy of the algorithm by establishing nonlinear semantic fusion and mining long-distance feature associations to effectively aggregate the global information of the target and the search area. STARK mainly uses the encoder in Transformer to model the global spatiotemporal feature correlation between the target object and the search area, and uses the decoder in Transformer to learn query embedding to predict the spatial position of the target object.

[0039] Compared with the prior art, the advantages of the present invention are:

[0040] (1) The present invention designs a defense reconstruction network based on the unique characteristics of video, making full use of the dual information in the space-time dimension for feature fusion and interaction, while enhancing the tracker's defense capability against unknown disturbances without reducing the accuracy of clean samples, thereby improving the accuracy of target tracking.

[0041] Among them, the processing of spatial dimension information is based on the bit plane layering method, and the present invention adopts the bit plane layering method to remove most of the disturbances. This is because the adversarial attack on target tracking causes the tracker to fail to track by adding imperceptible disturbances, and this imperceptible disturbance is usually located in the low bit plane. Therefore, the present invention combines the bit plane layering method to delete the visual information of the low bit plane to remove most of the disturbances, and only retains the main visual information and a small amount of disturbances in the high bit plane. And this method can enhance the robustness of the tracker by visually analyzing the distribution of disturbances in the bit plane without training adversarial samples, which can solve the problems of high training cost and high computational cost of existing defense methods.

[0042] The time dimension information processing is based on the defense network designed by the present invention. Since there is still a small amount of disturbance in the high bit plane after the bit plane layering method is adopted, some detail information will be lost after the low bit plane is deleted, which reduces the accuracy of the clean sample and limits the ability to defend against attacks. Therefore, the present invention designs a defense network M rec It is used to make up for the loss of low-bit planes and restore detail information. In addition, the network can strengthen the fusion of feature information, which is beneficial to improving the performance of defense.

[0043] (2) The defense network designed based on the present invention can maintain good generalization against various black-box and white-box adversarial attack methods. Compared with adversarial training, the present invention maintains better accuracy on clean samples.

[0044] (3) The defense network trained by the present invention can transfer its defense performance to other trackers based on CNN or CNN and Transformer in a plug-and-play manner without adjusting the parameters of the tracker, and has good transferability. When the defense network is used in combination with other trackers, it will not introduce too much computational overhead and the speed is considerable. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 This is a schematic diagram of the present invention;

[0046] Figure 2 This is the Unet network structure diagram;

[0047] Figure 3 This is a flowchart of the defense network workflow of the present invention. DETAILED DESCRIPTION

[0048] The present invention will be further described below in conjunction with embodiments.

[0049] Example 1: See Figures 1 to 3 ,A bit-plane-based auxiliary single target tracking adversarial defense reconstruction method includes the following steps;

[0050] S1, obtain a video frame sequence consisting of clean samples , where the clean sample of the i-th frame is I i , its marking , , I i The classification label and regression label of n is the total number of clean samples;

[0051] S2, in I i Add disturbance Get I i Corresponding adversarial examples , forming an adversarial sample sequence ;

[0052] S3, through the bit plane layering method Quantify and get I i Corresponding quantized samples , forming a quantized sample sequence ;

[0053] S4, construct a defense network M rec , used for input , output defense sample sequence ;

[0054] So the defense network M rec Based on the Unet network, when i=1, Output defense samples via Unet network , when i ≥ 2, Input Unet network and fuse the quantized samples of the previous frame Post-output defense sample ;

[0055] S5, obtain a tracker M based on the twin network F , the M F The feature extraction network has J layers, and each layer outputs a feature map;

[0056] The M F For input and , and , M F Output The corresponding feature map of the j-th layer feature extraction network , classification labels and regression labels ,right , M F Output The corresponding feature map of the j-th layer feature extraction network , predict the classification label and predict regression labels , 1≤j≤J;

[0057] S6, M rec and M F Form the total network and construct the total loss function L Tri , including feature loss L fea , defense sample classification loss L cls and defense sample regression loss L reg ;

[0058] S7, preset iteration rounds, use and Train the total network and freeze M during training F The network parameters are set to minimize L Tri Update M rec The network parameters of M are adjusted until the iteration round is reached. rec As a defensive reconstruction model ;

[0059] S8, obtain the video frame sequence to be identified, generate the corresponding quantized sample sequence according to steps S2 and S3, and then reconstruct the model through defense Generate defense sample sequences for tracking by the tracker.

[0060] In this embodiment: Quantize and obtain quantized samples Specifically:

[0061] Through the bit plane layering method It is divided into the first bit plane b1 to the eighth bit plane b8; the quantization factor k is preset according to the formula Generate quantized samples , where 1≤k<8, j is the jth bit plane For example, when k=5, .

[0062] Step S4 includes steps S41-S42;

[0063] S41, obtain a Unet network, including a contraction path and an expansion path, to quantize the samples , the contraction path generates the first down-sampling feature ~ the third down-sampling feature in turn ~ , the expansion path generates the third up-sampled feature ~ the first up-sampled feature in turn ~ ,and and , and , and , skip connections;

[0064] S42, yes When i=1, the defense sample is directly output through the Unet network ;

[0065] When i=2~n, the jump connection method of the Unet network is improved, and , and , and Skip connection to generate defense samples ,in, , , are the clean samples I of the i-1th frame respectively. i-1 Corresponding quantized samples The first down-sampling feature, the second down-sampling feature, and the third down-sampling feature.

[0066] In step S6, the feature loss L fea , total loss function LTri According to the following formula:

[0067] ,

[0068] ,

[0069] In the formula, α, β, and γ are L fea , L cls and L reg The hyperparameter balancing factor is: In this embodiment, α=0.4, β=0.3, and γ=0.3.

[0070] The tracker M F Including SiamRPN, SiamRPN++, TransT, and STARK.

[0071] In step S6, the defense sample classification loss L cls Using the cross entropy loss function, the defensive sample regression loss L reg Smooth L1 loss is used.

[0072] The training method for one iteration in step S7 includes:

[0073] S71, will Input M rec generate ;

[0074] S72, will , Enter M F Calculate the total loss function L Tri ;

[0075] S73, using ADAM optimizer through L Tri Update M rec network parameters.

[0076] For more information about quantizing samples to generate defense samples, see Figure 2 and Figure 3 . Figure 2 The Unet network structure diagram in the prior art shows the contraction path on the left side of the network and the quantization samples. Output in sequence ~ , the right side is the extended path, output in sequence ~ , and in and , and , and Establish a jump connection between Figure 2In the figure, Conv is a convolution operation, Up-Conv is a deconvolution upsampling operation, 3×3, 2×2, and 1×1 are the convolution kernel sizes corresponding to the convolution operation, and ReLU is the activation function. The defense samples are obtained through this method, which also corresponds to Figure 3 The first line handles the flow. Figure 3 This is the working diagram of the defense network of the present invention, which is based on Figure 2 Based on , only the skip connection method is changed. For example, , no longer according to the prior art and , and , and Instead of skip connections, and , and , and Skip connection to generate defense samples .

[0077] The twin network-based tracker described in the present invention has a workflow mainly divided into three parts: feature extraction, feature fusion and prediction. The twin network described in the present invention refers to the feature extraction part using two completely identical feature extraction networks.

[0078] Regarding CNN trackers, taking SiamRPN tracker as an example, SiamRPN includes a twin network and an RPN network. The twin network is two CNN networks with exactly the same structural parameters. The template frame and the search area frame are input into the twin network, and the features are extracted by a CNN network respectively, and then input into the RPN network containing classification branches and regression branches to generate a proposal frame and predict the location and confidence of the target. The CNN network extracts features, and the RPN network performs feature fusion and predicts the location and confidence of the target.

[0079] Regarding the CNN tracker, take the SiamRPN++ tracker as an example. In the SiamRPN++ tracker, the feature extraction still uses the twin network, but two ResNet50s with the same structural parameters are selected to replace the CNN network in SiamRPN. ResNet50 performs multi-layer feature extraction, and each layer corresponds to a feature map. For example, the feature map of the j-th layer feature extraction network described in the present invention is .

[0080] Regarding the tracker that combines CNN and Transformer, taking the TransT tracker as an example, it includes three modules: a twin network, a Transformer-based feature fusion network, and a prediction head. The twin network uses two ResNet50s as feature extraction networks. After extracting the features, these features are reshaped using a 1×1 convolutional layer and input into the Transformer-based feature fusion network. Each layer in the feature fusion network has a self-context enhancement module (ECA) and a cross-feature enhancement (CFA) module to enhance self-attention and cross-attention. Finally, the fused features are input into the prediction head, which uses simple classification and regression branches to locate the target and find the prediction box position. In the CNN backbone network, ResNet50 performs multi-layer feature extraction, and each layer corresponds to a feature map. For example, the feature map of the j-th layer feature extraction network .

[0081] Regarding the tracker that combines CNN and Transformer, taking the STARK tracker as an example, feature extraction is still based on ResNet for multi-layer feature extraction.

[0082] Figure 1 The tracker in the example uses the TransT tracker structure, which includes a twin network, a feature fusion network, and a prediction head. The twin network is composed of two ResNet50s as feature extraction networks, and each ResNet50 performs J layers of feature extraction.

[0083] Defense reconstruction model trained by the method of the present invention , can be plug-and-play applied to other trackers, with good migration. For example, to identify a video frame sequence, it is only necessary to generate the corresponding quantized sample sequence according to steps S2 and S3, and then reconstruct the model through defense. A defense sample sequence is generated, and then the defense sample sequence is input into a tracker, and the tracker can track the target normally according to the prior art.

[0084] Example 2: In order to test the defense method proposed in the present invention, we conducted tests on three benchmark datasets, namely OTB100, VOT2018, and LaSOT. For the hyperparameter settings of the experiment, we set the quantization factor k=5, α=0.4, β=0.3, and γ=0.3, and used the classic white-box attack CSA method (template and search area attack) and the black-box attack IoU method (search area attack) to test the defense performance of the present invention. The experimental results are shown in Table 1:

[0085] Table 1: Table of defense results of the present invention

[0086] ,

[0087] Among the indicators in Table 1: Success is the success rate, Precision is the precision rate, Accuracy is the accuracy rate, Robustness is the robustness, Norm.Precision is the normalized precision, and EAO (Expected Average Overlap) is the expected average coverage.

[0088] Experiments have proved that the BPDN defense network proposed in the present invention can significantly improve the robustness of the tracker, maintain good generalization against various black-box and white-box adversarial attack methods, and perform well on clean samples. In addition, the defense network not only has low training cost and high efficiency, but can also be plug-and-play applied to other trackers and has good portability.

[0089] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A bit-plane-based auxiliary single-target tracking adversarial defense reconstruction method, characterized by: The steps include: S1, obtain a video frame sequence consisting of clean samples , where the clean sample of the i-th frame is I i , its marking , , I i The classification label and regression label of n is the total number of clean samples; S2, in I i Add disturbance Get I i Corresponding adversarial examples , forming an adversarial sample sequence ; S3, through the bit plane layering method Quantify and get I i Corresponding quantized samples , forming a quantized sample sequence ; S4, construct a defense network M rec , used for input , output defense sample sequence ; So the defense network M rec Based on the Unet network, when i=1, Output defense samples via Unet network , when i ≥ 2, Input Unet network and fuse the quantized samples of the previous frame Post-output defense sample ; S5, obtain a tracker M based on the twin network F , the M F The feature extraction network has J layers, and each layer outputs a feature map; The M F For input and , and , M F Output The corresponding feature map of the j-th layer feature extraction network , classification labels and regression labels ,right , M F Output The corresponding feature map of the j-th layer feature extraction network , predict the classification label and predict regression labels , 1≤j≤J; S6, M rec and M F Form the total network and construct the total loss function L Tri , including feature loss L fea , defense sample classification loss L cls and defense sample regression loss L reg ; S7, preset iteration rounds, use and Train the total network and freeze M during training F The network parameters are set to minimize L Tri Update M rec The network parameters of M are adjusted until the iteration round is reached. rec As a defensive reconstruction model ; S8, obtain the video frame sequence to be identified, generate the corresponding quantized sample sequence according to steps S2 and S3, and then reconstruct the model through defense Generate defense sample sequences for tracking by the tracker; S3 Middle Quantize and obtain quantized samples Specifically: Through the bit plane layering method It is divided into the first bit plane b1 to the eighth bit plane b8; the quantization factor k is preset according to the formula Generate quantized samples , where 1≤k<8, j is the jth bit plane The index of In S6, the feature loss L fea , total loss function L Tri According to the following formula: ， ， In the formula, α, β, and γ are L fea , L cls and L reg The hyperparameter balancing factor.

2. The bit-plane-based auxiliary single target tracking countermeasure defense reconstruction method according to claim 1, characterized in that: k=5。 3. The auxiliary single target tracking countermeasure defense reconstruction method based on bit plane according to claim 1, characterized in that: Step S4 includes steps S41-S42; S41, obtain a Unet network, including a contraction path and an expansion path, to quantize the samples , the contraction path generates the first down-sampling feature ~ the third down-sampling feature in turn ~ , the expansion path generates the third up-sampled feature ~ the first up-sampled feature in turn ~ ,and and , and , and , skip connections; S42, yes When i=1, the defense sample is directly output through the Unet network ; When i=2~n, the jump connection method of the Unet network is improved, and , and , and Skip connection to generate defense samples ,in, , , are the clean samples I of the i-1th frame respectively. i-1 Corresponding quantized samples The first down-sampling feature, the second down-sampling feature, and the third down-sampling feature.

4. The auxiliary single target tracking countermeasure defense reconstruction method based on bit plane according to claim 1, characterized in that: α=0.4, β=0.3, γ=0.

3.

5. The auxiliary single target tracking countermeasure defense reconstruction method based on bit plane according to claim 1, characterized in that: The tracker M F Including SiamRPN, SiamRPN++, TransT, and STARK.

6. The auxiliary single target tracking countermeasure defense reconstruction method based on bit plane according to claim 1, characterized in that: In S6, the defense sample classification loss L cls Using the cross entropy loss function, the defensive sample regression loss L reg Smooth L1 loss is used.

7. The auxiliary single target tracking countermeasure defense reconstruction method based on bit plane according to claim 1, characterized in that: The training methods for one iteration in S7 include: S71, will Enter M rec generate ; S72, will , Enter M F Calculate the total loss function L Tri ; S73, using ADAM optimizer through L Tri Update M rec network parameters.

Images