Attack resisting method for multi-target tracking

Abstract

The invention discloses a multi-target tracking-oriented anti-attack method, which relates to the technical field of deep learning model security, and comprises the following steps: S1, extracting a to-be-attacked sample from an image sequence of an MOT17, MOT20 or BDD100K data set; s2, obtaining a target detection result by adopting YOLOX or the Internet; s3, matching the historical trajectory through intersection-to-union ratio and a Hungary algorithm, and determining a trajectory to be attacked; s4, an interference false alarm is generated in the reverse direction of target movement by using a false alarm generation module in combination with Kalman filtering; s5, calculating image disturbance based on a PGD algorithm and a targeted loss function, and generating an adversarial sample; s6, inputting the adversarial sample into trackers such as SORT, ByteTrack and the like to evaluate an attack effect; and S7, optimizing a disturbance generation parameter according to an evaluation result. According to the method, adaptive false alarm generation is realized through the TGM module, only one association operation is executed for each frame, attack focuses on an MOT association stage, verification is valid on multiple data sets and trackers, the attack efficiency, the concealment and the success rate can be improved, and the method is suitable for security test and vulnerability verification of the MOT model.

Description

Technical Field

[0001] This invention relates to the field of deep learning model security technology, specifically to an adversarial attack method for multi-target tracking. Background Technology

[0002] In the field of adversarial attacks against deep learning models, a pioneering work first investigated the vulnerability of deep learning models to adversarial examples; since then, various methods have been proposed, such as FGSM, BIM, JSMA and PGD; in the field of computer vision, a large amount of research has focused on adversarial attacks against various visual perception tasks, such as object detection, object tracking and semantic segmentation.

[0003] Among these studies, the most relevant to multi-target tracking (MOT) attacks are single-target tracking (SOT) attack methods and detection attack methods. One study proposed a novel one-off adversarial attack method for model-free single-target tracking; in the first frame, adding only a small perturbation to the target region can cause state-of-the-art trackers to lose the target in subsequent frames. Another study proposed a decision-based black-box attack method that weakens the accuracy of temporally continuous bounding boxes by reducing the IoU score. Yet another method proposed an online incremental attack technique with spatial awareness, capable of executing spatiotemporally sparse perturbations online, making the attack more difficult to detect; this method achieves basic spatial awareness attacks by adapting to existing attack methods. However, due to task differences and the trajectory recovery mechanism of MOT itself, these methods have very limited effectiveness in attacking multi-target trackers.

[0004] Recently, some studies have proposed an attack method to deceive trackers by misleading Kalman filters. To improve the attack effect, this method requires multiple correlation steps, resulting in high time costs for generating adversarial examples. Other studies have further simplified the optimization process and proposed an attack paradigm that combines missed detections and false alarms. However, this method manually sets a large number of hyperparameters during the target design process, and its target generation strategy does not consider the unique correlation mechanism in MOT, resulting in randomness in the generated target boxes. Our proposed target generation module can explicitly reduce the IoU between the trajectory and the real detection, thereby inducing the trajectory to correlate with false alarms.

[0005] In general, existing adversarial attack techniques for multi-target tracking can be divided into two categories: optimization attack methods and interference attack methods. Optimization attack methods inject false alarms into the detection results to guide the target trajectory away from its true position, thereby misleading the tracker. They utilize a one-to-one mapping optimization strategy to generate perturbations for each detection individually. During the optimization process, association steps are repeatedly executed for each target to generate adversarial examples. Interference attack methods combine false alarm injection and missed detection creation to interfere with the detection results and trigger ID switching, thereby causing trajectory errors. During the attack process, redundant detection boxes and missed detection areas are generated simultaneously, with the false alarm positions being fixed areas preset by the user.

[0006] However, the one-to-one optimization strategy is inefficient and has poor scalability in multi-target scenarios; repeated execution of correlation steps leads to a serious time consumption in the process of generating false alarms; and its attack effect on mainstream high-performance trackers such as ByteTrack is not good. The target generation strategy proposed in this patent can effectively improve the speed of generating false alarms and realize real-time adversarial attacks.

[0007] The number of false alarms is redundant, which conflicts with the NMS mechanism commonly used in target detection; the number of attacked frames is significantly abnormal and easily detected; the design of false alarm locations lacks flexibility and the attack lacks interpretability; after the attack ends, the original trajectory may be re-correlated, leading to attack failure; the counter-attack method proposed in this patent can effectively improve the concealment of the attack and increase the success rate of the attack. Summary of the Invention

[0008] To address the shortcomings of existing technologies, this invention provides an adversarial attack method for multi-target tracking, which solves the problems mentioned in the background section.

[0009] To achieve the above objectives, the present invention provides the following technical solution: an adversarial attack method for multi-target tracking, comprising the following specific steps:

[0010] S1. Extract the sample to be attacked from the image sequence;

[0011] S2. Use target detection methods to obtain the target detection results of the sample to be attacked;

[0012] S3. Determine the attack trajectory based on the similarity matching between historical trajectories and target detection results;

[0013] S4. Combine the target trajectory information to generate interference false alarms for the target detection results;

[0014] S5. Calculate image perturbation based on the location of the false alarm, and add the perturbation to the sample to be attacked to obtain the adversarial sample;

[0015] S6. Input the adversarial examples into existing multi-target tracking methods and evaluate the attack effect;

[0016] S7. Based on the attack effect evaluation results, adjust the generation parameters of the image perturbation (including the number of iterations of the PGD algorithm, loss function, etc.) to optimize the adversarial example generation effect.

[0017] Optionally, in step S1, the image sequence is derived from the MOT17 dataset, the MOT20 dataset, or the BDD100K dataset, and the specific parameters of each dataset are as follows:

[0018] MOT17 dataset: contains various common scene videos, provides detection results from multiple detectors, the training set contains 7 sequences (a total of 8500 frames), and the test set contains 7 sequences (a total of 9000 frames).

[0019] MOT20 dataset: Focuses on extremely dense crowd scenes (high occlusion, dense targets), the training set contains 4 sequences (a total of 11284 frames), and the test set contains 4 sequences (a total of more than 10000 frames).

[0020] BDD100K dataset: A large-scale autonomous driving video dataset covering weather conditions such as sunny, rainy, and foggy, as well as time conditions such as day and night. The training set contains more than 70,000 frames, and the test set contains more than 10,000 frames.

[0021] Optionally, in step S2, the target detection method is YOLOX or Centernet; the target detection result is represented as a set D, where each element in set D is a target bounding box, and the target bounding box is represented by... It means that, among them The coordinates of the bounding box center are This represents the width and height of the bounding box.

[0022] Optionally, in step S3, the historical trajectory is the target bounding box of the previous frame saved by the multi-target tracking method, denoted as set T; the similarity is achieved by calculating the intersection-union ratio of the bounding boxes, and the calculation formula is as follows: ;

[0023] Where A is any bounding box in set D, and B is any bounding box in set T. The area of ​​the intersection of the two frames. The area of ​​the union of the two frames;

[0024] Calculated The value is presented in the form of a confusion matrix (i.e., a cost matrix). The cost matrix is ​​input into the Hungarian algorithm to obtain the matching result with the minimum cost, and the successfully matched trajectory is determined. The successfully matched trajectory and the target detection result are used together as the trajectory to be attacked.

[0025] Optionally, in step S4, the interference false alarm is generated by the false alarm generation module (TGM); the false alarm generation module combines historical trajectory information and iteratively executes the "prediction-update" process through Kalman filtering:

[0026] Prediction phase: Based on the position and velocity parameters of the trajectory in the previous frame, predict the prior motion state of the trajectory in the current frame;

[0027] Update phase: Combine the target detection results of the current frame to correct the prior state and determine the precise movement direction and speed of the trajectory to be attacked;

[0028] Finally, in the opposite direction of the movement direction, the false alarm position and the size of the false alarm bounding box are calculated according to the movement speed to generate interference false alarms.

[0029] Optionally, in step S5, the image perturbation is calculated using the PGD (Projected Gradient Descent) algorithm combined with a specially designed loss function, as follows:

[0030] When the object detection method is YOLOX, the loss function is: in, Indicates the target classification loss. This represents the regression loss of the target bounding box. Hyperparameters (value range 0.1-0.5);

[0031] When the object detection method is CenterNet, the loss function is: ;

[0032] in, The regression loss represents the target bounding box, including the boundary size and center position offset. This represents the loss from predicting the heatmap, and... The design is the same as that in the previous example, and is represented as follows: Where Y is the target true value heatmap, N is the number of targets, and hyperparameters are... and .

[0033] Optionally, in step S6, the multi-target tracking method includes SORT, ByteTrack, CenterTrack, and FairMOT; the attack effect evaluation indicators include ID switching rate (IDSW(%)), ID switching rate (IDSW_im(%)), ID accuracy (IDF1), number of target IDs (IDs), false negative rate (FN(%)), and false alarm rate (FP(%)).

[0034] Optionally, in step S7, if the evaluation result satisfies "IDSW (%) increases by ≥1.5% and IDF1 decreases by ≥5%", then the current generated parameters are retained; if not, then the number of PGD algorithm iterations (±2 times) or loss function hyperparameters (±0.1) are adjusted, and steps S5-S6 are re-executed until the evaluation result meets the standard.

[0035] This invention provides an adversarial attack method for multi-target tracking, which has the following beneficial effects:

[0036] This adversarial attack method for multi-target tracking supports the generation of adaptive false alarms for multiple targets in the same frame, improving attack efficiency and applicability. This patent automatically calculates the position and size of false alarms by designing a TGM (TargetsGeneratingModule), eliminating the need for manual adjustment.

[0037] In addition, this patent performs the association operation only once per frame, keeping the attack process simple and efficient;

[0038] Unlike previous technical solutions, this patent has unique advantages, including:

[0039] (1) The TGM module proposed in this patent can be automatically adjusted according to the target position, motion state and size, adapting to various scenarios and trackers;

[0040] (2) The attack strategy designed in this patent focuses on the key weakness of MOT - the association phase, and has theoretical basis and transparency;

[0041] (3) This patent has demonstrated its good performance on several mainstream trackers (such as SORT, ByteTrack, CenterTrack, FairMOT) and datasets (BDD100K, MOT17, MOT20). Attached Figure Description

[0042] Figure 1 This is a schematic diagram of the invention process. Detailed Implementation

[0043] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments.

[0044] In the description of this invention, unless otherwise stated, "a plurality of" means two or more; the terms "upper," "lower," "left," "right," "inner," "outer," "front end," "rear end," "head," "tail," etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings, and are only for the convenience of describing the invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of the invention. Furthermore, the terms "first," "second," "third," etc., are used for descriptive purposes only and should not be construed as indicating or implying relative importance.

[0045] In the description of this invention, it should be noted that, unless otherwise explicitly specified and limited, the terms "connected" and "linked" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium. Those skilled in the art can understand the specific meaning of the above terms in this invention based on the specific circumstances.

[0046] Please see Figure 1 This invention provides a technical solution: an adversarial attack method for multi-target tracking, comprising the following specific steps:

[0047] S1. Extract the samples to be attacked from the image sequences. The image sequences are from the MOT17 dataset, MOT20 dataset, or BDD100K dataset. The specific parameters of each dataset are as follows:

[0048] MOT17 dataset: contains various common scene videos, provides detection results from multiple detectors, the training set contains 7 sequences (a total of 8500 frames), and the test set contains 7 sequences (a total of 9000 frames).

[0049] MOT20 dataset: Focuses on extremely dense crowd scenes (high occlusion, dense targets), the training set contains 4 sequences (a total of 11284 frames), and the test set contains 4 sequences (a total of more than 10000 frames).

[0050] BDD100K dataset: A large-scale autonomous driving video dataset covering weather conditions such as sunny, rainy, and foggy, as well as time conditions such as day and night. The training set contains more than 70,000 frames of sequences, and the test set contains more than 10,000 frames of sequences.

[0051] S2. Obtain the target detection results of the sample to be attacked using a target detection method, such as YOLOX or Centernet; the target detection results are represented as a set D, where each element of set D is a target bounding box, and the target bounding box is represented by... It means that, among them The coordinates of the bounding box center are This represents the width and height of the bounding box;

[0052] S3. Determine the attack trajectory based on the similarity matching between historical trajectories and target detection results. The historical trajectory is the target bounding box of the previous frame saved by the multi-target tracking method, represented as set T. The similarity is achieved by calculating the intersection-union ratio of the bounding boxes, and the calculation formula is as follows: Where A is any bounding box in set D, and B is any bounding box in set T. The area of ​​the intersection of the two frames. The area of ​​the union of the two frames;

[0053] Calculated The value is presented in the form of a confusion matrix (i.e., a cost matrix). The cost matrix is ​​input into the Hungarian algorithm to obtain the matching result with the minimum cost and determine the successfully matched trajectory. The successfully matched trajectory and the target detection result are used together as the trajectory to be attacked.

[0054] S4. Combining the target trajectory information, generate interference false alarms for the target detection results. The interference false alarms are generated by the False Alarm Generation Module (TGM). The False Alarm Generation Module combines historical trajectory information and iteratively executes the "prediction-update" process through Kalman filtering.

[0055] Prediction phase: Based on the position and velocity parameters of the trajectory in the previous frame, predict the prior motion state of the trajectory in the current frame;

[0056] Update phase: Combine the target detection results of the current frame to correct the prior state and determine the precise movement direction and speed of the trajectory to be attacked;

[0057] Finally, in the opposite direction of the movement direction, the false alarm position and the size of the false alarm bounding box are calculated according to the movement speed to generate interference false alarms;

[0058] S5. Calculate image perturbation based on the location of false alarms and add the perturbation to the sample to be attacked to obtain adversarial samples. The image perturbation is then calculated using the PGD (Projected Gradient Descent) algorithm combined with a specially designed loss function, as detailed below:

[0059] When the object detection method is YOLOX, the loss function is: ,in, Indicates the target classification loss. This represents the regression loss of the target bounding box. This is a hyperparameter (value range 0.1-0.5).

[0060] When the object detection method is CenterNet, the loss function is: ,in, The regression loss represents the target bounding box, including the boundary size and center position offset. This represents the loss from predicting the heatmap, and... The design is the same as that in the previous example, and is represented as follows: Where Y is the target true value heatmap, N is the number of targets, and hyperparameters are... and ;

[0061] S6. Input adversarial examples into existing multi-target tracking methods and evaluate the attack effect. Multi-target tracking methods include SORT, ByteTrack, CenterTrack, and FairMOT. Attack effect evaluation indicators include ID switching rate (IDSW (%)), ID switching rate (IDSW_im (%)), ID accuracy (IDF1), number of target IDs (IDs), false negative rate (FN (%)), and false alarm rate (FP (%)).

[0062] S7. Based on the attack effect evaluation results, adjust the generation parameters of the image perturbation (including the number of PGD algorithm iterations and loss function hyperparameters) to optimize the adversarial example generation effect. If the evaluation results meet the requirement of "IDSW (%) increase ≥1.5% and IDF1 decrease ≥5%", then retain the current generation parameters; if not, adjust the number of PGD algorithm iterations (±2 times) or loss function hyperparameters (±0.1), and re-execute steps S5-S6 until the evaluation results meet the requirements.

[0063] Dataset tracker Attack methods IDSW_im(%)↑ IDF1↓ IDSW (%)↑ IDs↑ FN(%)↑ FP(%)↑ BDD100K ByteTrack none - 56 1.49 9890 38.7 2.3 BDD100K ByteTrack Hijacking 51.8 48.2 2.73 18141 59.32 9.57 BDD100K ByteTrack F&Fattack 68.92 49.7 2.85 18980 52.77 11.31 BDD100K ByteTrack TAAttack 74.87 46.9 3.34 22217 58.15 10.6 BDD100K SORT clean - 53.42 1.92 12770 39.2 7.15 BDD100K SORT Hijacking 50.89 45.62 2.31 15364 55.32 10.23 BDD100K SORT F&Fattack 62.35 48.12 2.41 16029 51.79 14.32 BDD100K SORT TAAttack 71.45 44.39 2.79 17557 57.32 10.02 MOT17 ByteTrack none - 87.58 0.36 1755 17.92 3.88 MOT17 ByteTrack Hijacking 68.17 77.28 2.14 2218 19.02 3.94 MOT17 ByteTrack F&Fattack 85 73.76 3.13 3105 17.39 5.05 MOT17 ByteTrack TAAttack 81.7 74.16 3.02 2996 18.92 4.09 MOT17 SORT clean - 86.44 0.84 2242 16.15 6.21 MOT17 SORT Hijacking 68.03 76.04 2.98 3077 16.92 6.17 MOT17 SORT F&Fattack 78.29 73.89 3.81 3686 16.24 6.58 MOT17 SORT TAAttack 79.95 71.59 4.01 3879 17.54 6.24 MOT20 ByteTrack none - 92.77 0.11 20106 10.67 2.32 MOT20 ByteTrack Hijacking 57.97 66.87 2.02 22975 11.63 2.62 MOT20 ByteTrack F&Fattack 88.56 78.25 3.09 37256 10.14 3.26 MOT20 ByteTrack TAAttack 85.23 75.18 3.58 41240 11.89 2.54 MOT20 SORT clean - 93.14 0.21 22022 9.58 2.88 MOT20 SORT Hijacking 58.92 83.27 2.23 28950 10.19 3.07 MOT20 SORT F&Fattack 87.59 77.76 3.47 40376 9.58 3.14 MOT20 SORT TAAttack 91.24 75.34 3.78 44890 9.97 3.05

[0064] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. A method of adversarial attack oriented to multi-target tracking, characterized in that, The specific steps include the following: S1. Extract the sample to be attacked from the image sequence; S2. Use target detection methods to obtain the target detection results of the sample to be attacked; S3. Determine the attack trajectory based on the similarity matching between historical trajectories and target detection results; S4. Combine the target trajectory information to generate interference false alarms for the target detection results; S5. Calculate image perturbation based on the location of the false alarm, and add the perturbation to the sample to be attacked to obtain the adversarial sample; S6. Input the adversarial examples into existing multi-target tracking methods and evaluate the attack effect; S7. Based on the attack effect evaluation results, adjust the generation parameters of the image perturbation to optimize the adversarial sample generation effect.

2. The method of claim 1, wherein, In step S1, the image sequence is derived from the MOT17 dataset, the MOT20 dataset, or the BDD100K dataset. The specific parameters of each dataset are as follows: MOT17 dataset: contains videos of various common scenes, provides detection results from multiple detectors, the training set contains 7 sequences, and the test set contains 7 sequences; MOT20 dataset: Focuses on extremely dense crowd scenes, with 4 sequences in the training set and 4 sequences in the test set; BDD100K dataset: A large-scale autonomous driving video dataset covering weather conditions such as sunny, rainy, and foggy, as well as time conditions such as day and night. The training set contains more than 70,000 frames, and the test set contains more than 10,000 frames.

3. The method of claim 1, wherein, In step S2, the target detection method is YOLOX or Centernet; the target detection result is represented as a set D, where each element in set D is a target bounding box, and the target bounding box is represented by... It means that, among them The coordinates of the bounding box center are This represents the width and height of the bounding box.

4. The multi-target tracking oriented adversarial attack method according to claim 1, characterized in that, In step S3, the historical trajectory is the target bounding box of the previous frame saved by the multi-target tracking method, denoted as set T; the similarity is achieved by calculating the intersection-union ratio of the bounding boxes, and the calculation formula is as follows: Where A is any bounding box in set D, and B is any bounding box in set T. The area of ​​the intersection of the two frames. The area of ​​the union of the two frames; Calculated The value is presented in the form of a confusion matrix. The cost matrix is ​​input into the Hungarian algorithm to obtain the matching result with the minimum cost, and the successfully matched trajectory is determined. The successfully matched trajectory and the target detection result are used together as the trajectory to be attacked.

5. The multi-target tracking oriented adversarial attack method according to claim 1, characterized in that, In step S4, the interference false alarm is generated by the false alarm generation module; the false alarm generation module combines historical trajectory information and iteratively executes the "prediction-update" process through Kalman filtering: Prediction phase: Based on the position and velocity parameters of the trajectory in the previous frame, predict the prior motion state of the trajectory in the current frame; Update phase: Combine the target detection results of the current frame to correct the prior state and determine the precise movement direction and speed of the trajectory to be attacked; Finally, in the opposite direction of the movement direction, the false alarm position and the size of the false alarm bounding box are calculated according to the movement speed to generate interference false alarms.

6. The multi-target tracking oriented adversarial attack method according to claim 1, characterized in that, In step S5, the image perturbation is calculated using the PGD algorithm combined with a specially designed loss function, as follows: When the object detection method is YOLOX, the loss function is: ,in, Indicates the target classification loss. This represents the regression loss of the target bounding box. This is a hyperparameter (value range 0.1-0.5). When the object detection method is CenterNet, the loss function is: ,in, The regression loss represents the target bounding box, including the boundary size and center position offset. This represents the loss from predicting the heatmap, and... The design is the same as that in the previous example, and is represented as follows: , Where Y is the target true value heatmap, N is the number of targets, and hyperparameters are... and .

7. The multi-target tracking oriented adversarial attack method according to claim 1, characterized in that, In step S6, the multi-target tracking methods include SORT, ByteTrack, CenterTrack, and FairMOT; the attack effectiveness evaluation indicators include ID switching rate, ID switching rate, ID accuracy, number of target IDs, false alarm rate, and false alarm rate. In step S7, if the evaluation result satisfies "IDSW (%) increases by ≥1.5% and IDF1 decreases by ≥5%", then the current generated parameters are retained; if not, then the number of iterations of the PGD algorithm or the hyperparameters of the loss function are adjusted, and steps S5-S6 are re-executed until the evaluation result meets the standard.

Images