A general robust adversarial attack method for target tracking based on a twin network

Abstract

The application discloses a kind of target tracking general robust countermeasure attack methods based on twin network, including the following steps: using GIOU guided general robust target tracking countermeasure attack method, by being filled in to input video frame flip transformation produces abundant sample, with the GIOU score between video frame real label frame and target tracker prediction frame as guide produces general countermeasure disturbance, simultaneously to the disturbance application multiple transformation operations to find the best general disturbance, so that disturbance can resist various challenges in target tracking.The application can directly migrate the disturbance trained to any unknown image, the generated countermeasure sample image is not only not easy to be found, and does not need additional computing resources, realizes the efficiency and robustness of target tracking attack.

Description

Technical Field

[0001] This invention relates to the field of deep learning, and in particular to a general robust adversarial attack method for target tracking based on Siamese networks. Background Technology

[0002] With the development of computing power in computer hardware, deep learning technology, with its excellent performance and strong fitting ability, has been widely applied in fields such as computer vision, natural language processing, speech recognition, and industrial control. Visual target tracking is one of the key tasks in computer vision, aiming to predict the location and size of a target in subsequent frames given a target in the first frame. It plays a very important role in the field of public safety, such as video surveillance, human-computer interaction, intelligent transportation, autonomous driving, drone tracking, image target segmentation, and target behavior recognition.

[0003] In recent years, thanks to the rapid development of deep learning (DL) technology, target tracking algorithms have made significant breakthroughs. In particular, target tracking algorithms such as Siamese networks have achieved excellent performance in both accuracy and speed, reaching 91% accuracy on the OTB video tracking dataset, and their speed far exceeds real-time.

[0004] Current research on adversarial examples largely focuses on sample-dependent adversarial perturbation methods. These methods require generating specific adversarial perturbations for each sample, making the attack dependent on the sample and unable to be directly transferred to unknown images, resulting in low efficiency. Furthermore, most real-world intelligent video surveillance operates in black-box scenarios. To achieve secure gameplay in real-world environments, target tracking adversarial attack methods require a certain degree of universality and transferability. Existing target tracking adversarial attack methods all require generating separate adversarial perturbations for each image, making them unsuitable for direct application to any unknown real-world images. This lack of universality and transferability, coupled with the consumption of additional computational resources, undermines their effectiveness. Summary of the Invention

[0005] Purpose of the invention: The purpose of this invention is to provide a general robust adversarial attack method for target tracking based on Siamese networks, which directly transfers and superimposes trained perturbations onto any unknown image, thereby achieving high efficiency and robustness in target tracking attacks.

[0006] Technical solution: The present invention provides a general robust adversarial attack method for target tracking based on twin networks. The specific implementation process includes the following steps:

[0007] S1, Sample clean templates and clean search regions X from the dataset videos. i Obtain the training sample set;

[0008] S2, obtain clean templates and Z clean search regions X.i Using the Numpy library, each clean search region X is processed separately. i After T c Transformation generates rich data samples T for the search region c (X i );

[0009] S3, randomly initialize an initial perturbation image U using PyTorch functions. This initial perturbation image U has the same size as the clean search region; add the initial perturbation image U to the rich data samples T in the search region. c (X i In this context, an adversarial search region T is formed. c (X i )′;

[0010] S4 will enrich the data samples T in the search area. c (X i ), Adversarial search region T c (X i )′、The clean template is fed into the SiamFC++ tracker to obtain the original score sorting list Rank of the bounding box and the score sorting list Rank′ after adding perturbation;

[0011] S5, calculate the GIOU score of the ground truth bounding box and each predicted bounding box of the SiamFC++ tracker, sort them from smallest to largest, and denote them as GR;

[0012] S6, calculate the index of the box corresponding to the highest score in the original score sorting list Rank using the sorting function, calculate the index of the box corresponding to the box with the smallest GIOU score, and perform a flip attack.

[0013] S7, calculate the scale {d} corresponding to the boxes with GIOU > 0.7 in the search region. i |d1, d2, d3, d4, ..., d n}, execute a scale attack; where d i The scale plot represents the width and height values ​​of the predicted bounding box;

[0014] S8, Introducing perceptual loss L perp Ensure the effectiveness of the search area T c (X i )′ and the rich data sample T of the search region c (X i The absolute value of the pixel difference does not exceed the maximum value of the disturbance;

[0015] S9, jointly updated general adversarial perturbations, introduces T via the NumPy library. r Transformation;

[0016] S10: When the optimization objective converges, the current perturbation image is used as the best general perturbation image and output.

[0017] Furthermore, in step S3, the value of the initial perturbation image U after random initialization is between (-10, 10); the perturbation constraint condition is l. ∞ ≤10.

[0018] Furthermore, the implementation process of step S4 is as follows:

[0019] S4-1, For the j-th iteration, linearly add the perturbation image optimized in the (j-1)-th iteration to the N-th search region image in the j-th video to obtain the adversarial search region of the j-th video. Let the perturbation image optimized in the (j-1)-th iteration be denoted as U. j-1 ;

[0020] Where 1≤j≤M, 1≤N≤P, M is the number of videos in the training sample set; P is the number of search regions in the j-th video; the perturbation image after the 0th optimization is a randomly initialized perturbation image;

[0021] S4-2, based on the search region of the j-th video training sample and the confidence score map of the SiamFC++ tracker, obtain the original score ranking list Rank and the perturbation score ranking list Rank' of the bounding boxes of P search regions.

[0022] Furthermore, in step S5, the GIOU predicted by the SiamFC++ tracker is set to the minimum value.

[0023] Furthermore, in step S6, the specific calculation formula for performing the flip is as follows:

[0024]

[0025] Where L flip Let P be the loss function for the flip attack, where P is the number of search regions in the j-th video.

[0026] Rank′[gindex] represents the confidence score of the box with the smallest GIOU value in the sorted list.

[0027] Rank′[index] represents the confidence score corresponding to the highest-scoring box after the perturbation was added.

[0028] Furthermore, in step S7, the specific formula for performing the scale attack is as follows:

[0029]

[0030] Where L scaleLet n be the scale attack loss function, where n represents the number of bounding boxes with GIOU > 0.7 in the j-th video, ||·|| p L represents p Norm; d i The scale plot represents the width and height values ​​of the predicted bounding box; Δd i The width and height are predefined values, and the values ​​are random numbers between (0, 1).

[0031] Furthermore, step S8 introduces a perceptual loss L. perp The specific formula is as follows:

[0032]

[0033] Where L perp This is the perceptual loss, where P is the total number of search regions in the j-th video; ||·|| 2 For MSE Loss; T c (X i ) represents a rich data sample for the search area; T c (X i )′ represents the adversarial search region.

[0034] Furthermore, the detailed implementation steps of step S9 are as follows:

[0035] S9-1 uses flip attack loss, scale attack loss, and perp perception loss as optimization objectives to iteratively optimize the perturbation image after the (j-1)th optimization, resulting in the perturbation image after the j-th optimization. The total loss is specifically described as follows:

[0036] L total =L flip +L scale +L perp

[0037] Where L total L represents the total loss function. flip L represents the loss function for flip attack. scale L represents the loss function for scale-based attacks. perp Indicates perceived loss;

[0038] S9-2 employs momentum stochastic gradient descent, targeting the total loss L... total Perform backpropagation iterative optimization on the (j-1)th optimized perturbation image to update the perturbation image;

[0039] Let U be the perturbation image updated during the i-th iteration. i The update process is specifically represented as follows:

[0040]

[0041] δ i =δ i-1 +lr·sign(g i )

[0042] δ i =min(max(-∈,δ) i ), ∈)

[0043] Where g i is the momentum of the i-th iteration, μ is the momentum parameter, set to 0.5; sign(·) is the sign function, which returns 1 when (·) is greater than 0, 0 when (·) is equal to 0, and -1 when (·) is less than 0; lr is the learning rate. It is the gradient, δ i It is the perturbation image after the i-th perturbation; ∈ represents the maximum perturbation value; ∈ represents the maximum perturbation value, min() represents taking the smaller of the two numbers, and max() represents taking the larger of the two numbers;

[0044] S9-3, introducing T to general counter-disturbance r Transform the perturbation to obtain a new round of iterations, and repeat steps S4-S8 to continue iterative optimization.

[0045] Furthermore, in step S9-2, during the j-th iteration of optimization, it is determined whether the pixel value of the perturbation image after the (j-1)-th optimization is greater than the maximum constraint;

[0046] If the pixel value of the perturbation image after the (j-1)th optimization is greater than the maximum constraint value, the perturbation is considered to be in a saturated state, and δ is rescaled to half of its original value, and iterative optimization continues. If the pixel value of the perturbation image after the (j-1)th optimization is less than or equal to the maximum constraint value, then there is no need to scale the current perturbation image δ, and it is kept as is, and iterative optimization continues.

[0047] Compared with the prior art, the significant advantages of this invention are as follows:

[0048] 1. This invention targets visual target tracking tasks, using the SiamFC++ target tracker as the research object. Based on the two-stage task (classification and regression) of visual target tracking, it determines the optimal target finding adversarial perturbation. It has a small number of parameters, is fast, and can guide the target tracker to predict the trajectory of the moving target along the direction farthest from the target's location. It can be directly transferred to any unknown image without requiring additional computing resources.

[0049] 2. The perturbation generation method mentioned in this invention can withstand various challenges in target tracking, improve the robustness of perturbations, is suitable for targets of various sizes in multiple scenarios, and can generate perturbations efficiently.

[0050] 3. Utilizing the GIOU-guided general robust adversarial attack method for target tracking, starting from the two-stage tasks of classification and regression in visual target tracking, the method feeds a clean template, a clean search region, and an adversarial search region into the SiamFC++ tracker, respectively, and performs flip attacks and scaling attacks to find the optimal general perturbation. This allows the trained perturbation to be directly transferred and superimposed onto any unknown image. The generated adversarial example images are not only difficult to detect, but also do not require additional computational resources, achieving both high efficiency and robustness in target tracking attacks. Attached Figure Description

[0051] Figure 1 This is a flowchart of the present invention;

[0052] Figure 2 This is a framework diagram of the present invention;

[0053] Figure 3(a) shows the original search area.

[0054] Figure 3(b) shows the optimal perturbation image obtained through training.

[0055] Figure 3(c) shows the image of the adversarial search region. Detailed Implementation

[0056] The present invention will now be described in further detail with reference to the accompanying drawings and specific embodiments.

[0057] like Figure 1 and Figure 2 As shown, this invention proposes a general robust adversarial attack method for target tracking based on Siamese networks, which specifically includes the following steps:

[0058] S1, Sample clean templates and clean search regions X from the dataset videos. i Obtain the training sample set: The training sample set includes multiple video segments, each with different durations and resolutions. Clean templates and clean search regions X are sampled from the videos. i .

[0059] In this embodiment, a training set was selected from the GOT-10k dataset (over 500 classes, with more than 10,000 video sequences). For each video sequence, the target template was cropped in the first frame. In subsequent frames, samples were taken uniformly every 5 frames, and the search region was cropped. The template region size was cropped to 127*127 pixels, and the search region size was cropped to 303*303 pixels. The OTB100 test set (98 videos), VOT2018 test set (60 videos), and LaSOT test set (280 videos) were used to evaluate the ability of general robust adversarial images to attack arbitrary unknown videos. The training and test sample sets had no overlap in videos.

[0060] S2, obtain clean templates and Z clean search regions X. i Using the Numpy library, each clean search region X i After T c Transformations (lighting simulation, rotation, flipping, filling, etc.) generate rich data samples T for the search region. c (X i This transformation is a random transformation, meaning that for any search region image, a transformation method is randomly selected; T is introduced. c The purpose of the transformation is to maintain the richness of the data, increase the amount of data in the dataset, and improve the general perturbation-resistant transferability.

[0061] S3, using PyTorch functions, randomly initialize an initial perturbation image U with the same size as the clean search region, i.e., 303*303 pixels, where the perturbation constraint is l. ∞ ≤1. Add the initial perturbation image U to the search region to enrich the data sample T. c (X i In this context, an adversarial search region T is formed. c (X i )′.

[0062] The perturbation image U is initialized using the np.random.uniform random initialization function, which makes it more likely that the algorithm will find the global optimum. The initial value of the perturbation image U is between (-10, 10).

[0063] S4, clean search area X i , and the counter-search region T c (X i A clean template is fed into the SiamFC++ tracker to obtain the original bounding box score ranking list Rank and the score ranking list Rank′ after perturbation. The SiamFC++ tracker is a model pre-trained using the GoogLeNet backbone network. The SiamFC++ target tracking model consists of two stages: a classification stage and a regression stage. The classification stage is used to roughly locate the location of the target, and mainly has two branches: a classification branch and a quality assessment branch. The classification branch outputs a 17*17 confidence score map, and the quality assessment branch outputs a 17*17 quality assessment map. The regression stage is used to regress the precise location and size of the bounding box, outputting a 17*17*4 regression map. The specific operation of step S4 is as follows:

[0064] S4-1, For the j-th iteration, linearly add the perturbation image optimized in the (j-1)-th iteration to the N-th search region image in the j-th video to obtain the adversarial search region of the j-th video. Let U be the perturbation image optimized in the (j-1)-th iteration. j-1Where 1≤j≤M, 1≤N≤P, M is the number of videos in the training sample set; P is the number of search regions in the j-th video; the perturbation image after the 0th optimization is a randomly initialized perturbation image.

[0065] S4-2, based on the search region of the j-th video training sample and the confidence score map of the SiamFC++ tracker, obtain the original score ranking list Rank and the perturbation score ranking list Rank' of the P search regions: by comparing the clean template and the clean search region X... i The data is fed into the SiamFC++ tracker to obtain the original bounding box score sorted list Rank; the clean template and the adversarial search region T are then processed. c (X i The data is fed into the SiamFC++ tracker to obtain the perturbation score, which is then sorted by Rank.

[0066] S5, calculate the GIOU score of the ground truth bounding box and each predicted bounding box of the SiamFC++ tracker, sort them from smallest to largest, and denote them as GR;

[0067] For each search region, the SiamFC++ tracker generates 289 predicted bounding boxes, resulting in 289 GIOU scores. The GIOU score is calculated based on the ground truth bounding box and the tracker's predicted bounding boxes (one ground truth bounding box and multiple predicted bounding boxes per image, totaling 289 predicted boxes). The specific formula is as follows:

[0068]

[0069] Where R represents the true bounding box of each search region in the video, the true bounding box being the actual location of the target in the image, and its coordinates are provided in the dataset; P i ∪ represents the i-th predicted bounding box of the SiamFC++ tracker in the current search region; ∪ represents the union. For the SiamFC++ tracker, given the input template and search region, two outputs are obtained: the output of the classification stage and the output of the regression stage. The confidence score map is obtained from the output of the classification stage; the coordinates P of the target tracker's predicted bounding box can be calculated from the output of the regression stage. i C indicates that it includes R and P. i Minimum bounding box (containing R and P) i The minimum convex closed bounding box (IOU(R, P)). i The Interchange-of-Use (IOU) score refers to the score between the ground truth bounding box and each predicted bounding box of the object tracker. It is calculated using the following formula:

[0070]

[0071] Where ∩ represents the intersection and ∪ represents the union.

[0072] Step S5 involves calculating GIOU scores for each image separately, generating multiple corresponding GIOU scores. GIOU scores measure the distance between two borders; the greater the distance between borders, the smaller the GIOU value, and the closer the borders, the larger the GIOU value.

[0073] S6: Calculate the index of the highest-scoring bounding box in the original score ranking list (Rank) using a ranking function. Calculate the index (gindex) of the bounding box with the lowest GIOU score. Perform a flip attack to make Rank′[gindex] similar to Rank′[index]. This causes the confidence score of the original predicted bounding box to decrease, while the confidence score of the bounding box with the lowest GIOU score increases. This misleads the tracker from treating the original predicted bounding box as the final predicted bounding box, instead mistaking the bounding box with the lowest GIOU score as the prediction result. The bounding box with the lowest GIOU score is far from the true target bounding box, thus achieving the attack objective. The specific steps are as follows:

[0074] Calculate the index of the box corresponding to the highest score:

[0075] Step S2 can extract Z clean search regions X. i For each clean search region, the object tracker generates 289 predicted bounding boxes, each with its own index number and corresponding confidence score. The index of the box corresponding to the highest score in the original score ranking list (Rank) is obtained using the PyTorch function `torch.sort()`, which sorts the scores from highest to lowest. The function returns two values: the confidence score for each box and the original index value of each box in its unsorted state. Through this process, the index value of the box corresponding to the highest score in the original score ranking list (Rank) is obtained.

[0076] Calculate the index gindex corresponding to the box with the lowest GIOU score:

[0077] The GIOU scores of the 289 predicted boxes calculated in step S5 are sorted from smallest to largest using the torch.sort() function in PyTorch. The result returns two values: one is the GIOU score of each box, and the other is the index value of each box when it is not sorted. Through the above process, the index value gindex corresponding to the box with the smallest GIOU score is obtained.

[0078] Perform a flip operation to make Rank′[gindex] similar to Rank′[index]. That is, when the value of Rank′[gindex] - Rank′[index] approaches 0, the two are similar. The specific calculation formula is as follows;

[0079]

[0080] Where L flip The loss function for the flip attack is P, where P is the total number of search regions in the j-th video; Rank′[gindex] represents the confidence score of the box with the smallest GIOU value in the sorted list, and Rank′[index] represents the confidence score of the box with the highest score after adding perturbation.

[0081] The operating principle of step S6 will be explained as follows:

[0082] A smaller GIOU value indicates that the predicted bounding box is farther from the true target bounding box. This operation step makes the difference between Rank′[gindex] and Rank′[index] approach 0, so that the confidence score of the box finally predicted by the original tracker is as similar as possible to the score corresponding to the box with the smallest GIOU value. Since the tracker predicts the final target bounding box based on the original bounding box score sorting list Rank, this step has the effect of decreasing the confidence score of the original predicted box and increasing the confidence score of the bounding box with the lowest GIOU score. In this way, the tracker will not regard the original predicted box as the final predicted box, but will regard the box with the lowest GIOU score as the final predicted box. Since the box with the lowest GIOU score is the farthest from the true target box, the attack effect is achieved. The attack is to make the tracker unable to predict the correct target bounding box, but to deviate from the target position as much as possible. This attack method is called the flip attack in this invention.

[0083] S7: Calculate the scale map {d} corresponding to the boxes with GIOU>0.7 in the search region. i |d1, d2, d3, d4, ..., d n}, perform a scale attack, adjusting the scale of boxes with GIOU scores greater than a certain threshold to a predefined scale Δd. i Similarly, bounding boxes with a GIOU value greater than 0.7 are selected for a scaling attack, causing the predicted bounding box's width and height to shrink rapidly, reducing the overlap with the ground truth bounding box. Specifically:

[0084]

[0085] Where L scaleLet n be the scale attack loss function, where n represents the number of bounding boxes with GIOU > 0.7 in the j-th video, ||·|| p L represents p Norm; d i The scale plot represents the width and height values ​​of the predicted bounding box; Δd i The width and height are predefined values, and the values ​​are random numbers between (0, 1).

[0086] By calculating d i -Δd i The goal is to make the width and height of the predicted bounding box as small as possible. The smaller the width and height, the smaller the bounding box, and the smaller the overlap with the original real target box, thus achieving the effect of a scale attack.

[0087] Among them, the scale map {d i |d1, d2, d3, d4, ..., d n The result is obtained by inputting the template and search area into the SiamFC++ tracker, and it is calculated from the corrected coordinate values ​​corresponding to the 289 prediction boxes.

[0088] The SiamFC++ tracker outputs two things: a confidence score map and a regression map. These correspond to the two-stage task of Siamese network object tracking—classification and regression. In step S6, the flip attack designed in the classification stage only interferes with the bounding box position, without further interfering with the size of the bounding box. Although the SiamFC++ tracker selects the predicted box based on the highest confidence score, it only yields a rough, not precise, location. The other output of the SiamFC++ tracker (the regression map) is designed to address the accuracy issue; correcting the coordinate values ​​of the boxes allows for more accurate positioning of the final predicted object bounding box. Therefore, the regression map in the regression stage also needs to be perturbed. Based on the regression map, a scale map, i.e., the size of the predicted bounding box, can be derived. If the target tracker can still predict the correct target bounding box after the flip attack, a scale attack can be used to correct this, allowing the bounding box to shrink rapidly.

[0089] To achieve more accurate target bounding box localization, step S7 performs a scaling attack by scaling the bounding boxes with GIOU scores greater than a certain threshold to a scale d. i With a predefined scale Δd i resemblance.

[0090] A higher GIOU value indicates that the bounding box is closer to the true target box, and therefore more likely to be the target location. In this invention, the GIOU value ranges from -1 to 1. This invention selects boxes with a GIOU > 0.7 for calculation to achieve a trade-off. Since the goal of step S7 is to find predicted boxes close to the true target box and interfere with their scale, if GIOU > 0.9 or GIOU > 0.8, fewer boxes are calculated, resulting in a relatively poor effect. If GIOU > 0.7, more boxes are calculated, resulting in stronger interference and greater targeting, specifically those boxes very close to the true target box. If the GIOU score is below 0.7, the targeting is weak, easily leading to resource waste and consumption.

[0091] Therefore, according to the calculation process in step S7, for each image, borders with a GIOU value > 0.7 are selected. Let the number of borders be n. By calling the .size() function, the scale attack starts from the regression task (that is, the precise position of the corresponding border) and selects borders with a GIOU > 0.7 for attack.

[0092] S8: To make the generated image closer to the original image and prevent image distortion, a perceptual loss L is introduced. perp Let the adversarial search area T c (X i )′ and the original search region T c (X i The images should be as similar as possible, meaning the absolute value of the pixel difference between them should not exceed the maximum value of the perturbation. This helps train subtle, universal adversarial perturbations invisible to the human eye, preventing distortion of adversarial example images. Specifically:

[0093]

[0094] Where L perp This is the perceptual loss, where P is the total number of search regions in the j-th video; ||·|| 2 For MSE Loss; T c (X i ) represents a rich data sample for the search area; T c (X i )′ represents the adversarial search region.

[0095] S9: Based on the training sample set obtained in step S1 and the two-stage task of the SiamFC++ tracker, iteratively optimize the initial perturbation image U to obtain the optimal general adversarial perturbation image: update the adversarial perturbation image, and jointly update the general adversarial perturbation using the flip attack loss (flip), scale attack loss (scale), and perceptual loss (perp). T is then introduced through the NumPy library. rTransformations (translation, resizing, etc.) are applied to the general adversarial perturbation to obtain a new perturbation after the next iteration. Steps S4-S8 are repeated to continue iterative optimization and improve the robustness of the perturbation. The specific operation process is as follows:

[0096] S9-1: Update the adversarial perturbation image. Using flip attack loss, scale attack loss, and perp perception loss as optimization objectives, iteratively optimize the perturbation image after the (j-1)th optimization to obtain the perturbation image after the j-th optimization. The total loss is specifically described as follows:

[0097] L total =L flip +L scale +L perp (6)

[0098] Where L total L represents the total loss function, i.e., the objective function; flip L represents the loss function for flip attack. scale L represents the loss function for scale-based attacks. prep Indicates perceived loss;

[0099] S9-2: Employing momentum stochastic gradient descent, targeting the total loss L... total The perturbation image after the (j-1)th optimization is iteratively optimized through backpropagation to update the perturbation image. Stochastic gradient descent with momentum can accelerate convergence and reduce oscillations during the convergence process, thereby improving accuracy.

[0100] The main parameters of the momentum stochastic gradient descent optimizer include:

[0101] δ: Represents the learnable parameters in the model that need to be updated. In this embodiment, δ is the general perturbation image.

[0102] `lr` represents the learning rate. The learning rate is a hyperparameter that guides how the network weights are adjusted using the gradient of the loss function. A lower learning rate results in a slower rate of change in the loss function. A higher learning rate means more frequent weight updates. In this embodiment, `lr` is set to 0.02.

[0103] g represents the momentum factor. Momentum is updated using the gradient at the current point in each iteration, and then used as the increment for the next iteration. In this embodiment, the momentum factor g = 1.

[0104] The perturbation image updated during the i-th iteration is specifically represented as follows:

[0105]

[0106] δ i =δ i-1+lr·sign(g i (8)

[0107] δ i =min(max(-∈,δ) i ),∈) (9)

[0108] Where g i is the momentum of the i-th iteration, μ is the momentum parameter, set to 0.5; sign(·) is the sign function, which returns 1 when (·) is greater than 0, 0 when (·) is equal to 0, and -1 when (·) is less than 0; lr is the learning rate. It is the gradient, δ i It is the perturbation image after the i-th perturbation; ∈ represents the maximum perturbation value; ∈ represents the maximum perturbation value, min() represents taking the smaller of the two numbers, and max() represents taking the larger of the two numbers.

[0109] For the j-th iteration of optimization, it is determined whether the pixel value of the perturbation image after the (j-1)-th optimization is greater than the maximum constraint. In this embodiment, the maximum constraint value is 0.04.

[0110] If the pixel value of the perturbation image after the (j-1)th optimization is greater than the maximum constraint value, the perturbation is considered to be in a saturated state. This may render the update invalid after δ reaches its maximum value. Therefore, in this case, δ is scaled back to half of its original value, and iterative optimization continues. If the pixel value of the perturbation image after the (j-1)th optimization is less than or equal to the maximum constraint value, there is no need to scale the current perturbation image δ. It is kept as is, and iterative optimization continues.

[0111] S9-3: To enable the perturbation to withstand various challenges of target tracking, a T-type perturbation is introduced into the general countermeasures perturbation. r Transformations (translation, multi-scale resizing, etc.) are used to obtain the perturbation after a new round of iterations. Steps S4-S8 are repeated to continue iterative optimization and improve the robustness of the perturbation.

[0112] S10: When the optimization target converges, the current perturbation image is used as the best general perturbation image and output. The final general adversarial perturbation image can be superimposed on any training video search area to form adversarial examples, which can guide the visual target tracking model to move in the direction with the greatest distance from the target.

[0113] The optimization objective is the total loss L mentioned in step S9. total Optimization objective convergence refers to the state in which the gradient change of the model tends to be gradual during the training process; when the loss changes very little and tends to be gradual as the number of iterations increases, the optimization objective converges.

[0114] The optimal perturbation image is one that is the same size as the search area. Figure 3(a) , 3(b) Figure 3(c) shows the effect of adding a general perturbation image to the original search region image. The best perturbation image trained (Figure 3(b)) is superimposed on the original search region (Figure 3(a)), and the superimposed image is the adversarial search region image (Figure 3(c)). Figure 3(a) , 3(b) As can be seen in 3(c), the optimal perturbation image focuses on the location where the target exists in the image, enabling a targeted attack on the target in the image, reducing ineffective perturbations, and enhancing the effectiveness of the perturbation. Furthermore, although the original image is superimposed with the general adversarial perturbation of this invention, and the pixel values ​​of the original search region image are changed to become the adversarial search region image, the human eye cannot distinguish it from the original search region image because a corresponding loss function is specifically designed to prevent distortion of the adversarial sample image, thus achieving the concealment of the perturbation.

[0115] In this embodiment, the general robust adversarial perturbation obtained from training is tested on three test datasets: OTB100, VOT2018, and LaSOT, to evaluate the tracking performance of the target tracking model SiamFC++.

[0116] The optimal size for the general adversarial perturbation image output is 303*303*3, where 303, 303, and 3 represent the length, width, and number of RGB channels of the perturbation image, respectively.

[0117] The following is a verification of the attack performance of the optimal general perturbation obtained by the general robust adversarial attack method based on Siam networks in the SiamFC++ target tracking model:

[0118] The generated best general perturbation is linearly added to the search region of the test set video to evaluate tracking performance. For the OTB100 and LaSOT test datasets, tracking performance is measured by Success, which represents the percentage of predicted states obtained by the tracking algorithm that have an overlap rate greater than 0.5 with the original target. For the VOT2018 test dataset, Accuracy and Robustness are used, and tracking performance is ranked by EAO (mean overlap expectation). ↑ indicates higher performance and ↓ indicates lower performance. Dataset represents the test dataset, Metric represents the evaluation metric, Original represents the original tracking results of SiamFC++, After Attack represents the tracking performance after using the general adversarial perturbation attack, and Drop represents the performance degradation.

[0119] As shown in Table 1, the target tracking adversarial attack method based on twin networks of the present invention has achieved good attack results and successfully confused the target tracker.

[0120] Table 1

[0121]

[0122] Taking the OTB100 dataset as an example, Table 1 shows that the original SiamFC++ tracker's tracking success rate was 0.683. After the attack using the general robust adversarial perturbation method of this invention, the tracking success rate dropped to 0.120, a decrease of 0.563, indicating that the attack method of this invention significantly reduced the tracking performance of the SiamFC++ tracker. Taking VOT2018 as an example, higher accuracy and lower robustness indicate better tracking performance. After the attack, accuracy decreased significantly, while robustness increased significantly. The average overlap expectation, which combines accuracy and robustness, can be used as the final evaluation index for VOT2018, decreasing from the original 0.426 to 0.065, a decrease of 0.361, verifying the effectiveness of the attack method of this invention.

Claims

1. A general robust adversarial attack method for target tracking based on Siamese networks, comprising the following steps: S1, Sample clean templates and clean search regions from the dataset videos. Obtain the training sample set; S2, Obtain the clean template and Z clean search regions. Using the NumPy library, each clean search region is processed separately. go through Transformation generates rich data samples for the search region ( ); the The transformation is a random transformation, including lighting simulation, rotation, flipping, and filling; S3, Initialize an initial perturbation image U randomly using PyTorch functions. This initial perturbation image U has the same size as the clean search region; add the initial perturbation image U to the search region to enrich the data samples. ( In this context, a counter-search zone is formed. ; S4 will enrich the data samples in the search area. ( ) Counter-search area A clean template is fed into the SiamFC++ tracker to obtain the original score sorting list Rank of the bounding boxes and the score sorting list after perturbation. ; S5, calculate the GIOU score of the ground truth bounding box and each predicted bounding box of the SiamFC++ tracker, and sort them from smallest to largest; S6, calculate the index of the box corresponding to the highest score in the original score sorting list Rank using the sorting function, calculate the index of the box corresponding to the box with the smallest GIOU score, and perform a flip attack. S7, calculate the scale corresponding to the boxes with GIOU > 0.7 in the search region { }, execute a scale attack; in The scale plot represents the width and height values ​​of the predicted bounding box; S8, introducing perceptual loss Ensure the search area is countered With rich data samples in the search area The absolute value of the pixel difference does not exceed the maximum value of the disturbance; S9, jointly updated with general adversarial perturbations, introduced via the NumPy library. Transformation; the Transformations include translation and multi-scale resizing; Step S9 specifically includes: S9-1 uses flip attack loss, scale attack loss, and perp perception loss as optimization objectives to iteratively optimize the perturbation image after the (j-1)th optimization, resulting in the perturbation image after the j-th optimization. The total loss is specifically described as follows: ； in Represents the total loss function. This represents the loss function for the flip attack. Represents the loss function for scale attacks. Indicates perceived loss; S9-2 employs momentum stochastic gradient descent, targeting the total loss. Perform backpropagation iterative optimization on the (j-1)th optimized perturbation image to update the perturbation image; Let the perturbation image updated during the i-th iteration be denoted as . The update process is specifically represented as follows: ； ； ； in It is the momentum in the i-th iteration. This is a parameter for momentum, set to 0.5; It is a sign function, when When it is greater than 0, then Return 1; when When it equals 0, then Returns 0; when If less than 0, then Returns -1; lr is the learning rate. It is the gradient. It is the perturbation image after the i-th perturbation; This represents the maximum perturbation value; min() represents taking the smaller of the two numbers, and max() represents taking the larger of the two numbers; S9-3, Introduction of General Countermeasures Disturbance Transform to obtain the perturbation after a new round of iterations, and repeat steps S4-S8 to continue iterative optimization; S10, When the optimization objective converges, the current perturbation image is used as the best general perturbation image and output. The convergence of the optimization objective refers to the state in which the gradient change of the model tends to be gradual during the training process. When the loss of the optimization objective changes very little and tends to be gradual as the number of iterations increases, the optimization objective converges.

2. The general robust adversarial attack method for target tracking based on Siamese networks as described in claim 1, characterized in that, In step S3, the value of the initial perturbation image U after random initialization is between (-10, 10); the perturbation constraint condition is... ≤10.

3. The general robust adversarial attack method for target tracking based on Siamese networks as described in claim 1, characterized in that, The implementation process of step S4 is as follows: S4-1, For the j-th iteration, linearly add the perturbation image optimized in the (j-1)-th iteration to the N-th search region rich data sample in the j-th video to obtain the adversarial search region of the j-th video. The perturbation image optimized in the (j-1)-th iteration is denoted as... ; Where 1≤j≤M, 1≤N≤P, M is the number of videos in the training sample set; P is the total number of search regions in the j-th video; the perturbation image after the 0th optimization is a randomly initialized perturbation image; S4-2, based on the search region of the j-th video training sample and the confidence score map of the SiamFC++ tracker, obtain the original score ranking list Rank and the perturbation score ranking list Rank' of the bounding boxes of P search regions.

4. The general robust adversarial attack method for target tracking based on Siamese networks as described in claim 1, characterized in that, In step S5, the GIOU predicted by the SiamFC++ tracker is taken as the minimum value.

5. The general robust adversarial attack method for target tracking based on Siamese networks as described in claim 1, characterized in that, In step S6, the specific calculation formula for performing the flip operation is as follows: ； in Let P be the loss function for the flip attack, where P is the total number of search regions in the j-th video. This indicates the confidence score corresponding to the box with the smallest GIOU value in the sorted list. This represents the confidence score corresponding to the box with the highest score after the perturbation was added.

6. The general robust adversarial attack method for target tracking based on Siamese networks as described in claim 1, characterized in that, In step S7, the specific formula for performing the scale attack is as follows: ； in Let n be the scale attack loss function, where n represents the number of bounding boxes with GIOU > 0.7 in the j-th video. represent Norm; The scale plot represents the width and height values ​​of the predicted bounding box; The width and height are predefined values, and the values ​​are random numbers between (0, 1).

7. The general robust adversarial attack method for target tracking based on Siamese networks as described in claim 1, characterized in that, Step S8 introduces perceptual loss. The specific formula is as follows: ； in It is the perceptual loss, where P is the total number of search regions in the j-th video; For MSE Loss; A rich sample of data representing the search area; This represents the area to be searched.

8. The general robust adversarial attack method for target tracking based on Siamese networks as described in claim 1, characterized in that, In step S9-2, during the j-th iteration of optimization, it is determined whether the pixel value of the perturbation image after the (j-1)-th optimization is greater than the maximum constraint. If the pixel value of the perturbation image after the (j-1)th optimization is greater than the maximum constraint value, the perturbation is considered to be in a saturated state, and δ is rescaled to half of its original value, and iterative optimization continues. If the pixel value of the perturbation image after the (j-1)th optimization is less than or equal to the maximum constraint value, then there is no need to scale the current perturbation image δ, and it can be kept as is, and the iteration optimization can continue.

Images