A target tracking method based on two-stage selection

Abstract

The application relates to a target tracking method based on two-stage selection, comprising the following steps: acquiring an input video stream sequence, for a current frame, extracting image features and generating at least one candidate mask and a confidence score corresponding to each candidate mask based on memory bank information; predicting the motion state of the target in the current frame based on a motion prediction model, and calculating a fitting evaluation score between each candidate mask and the predicted motion state; selecting a first candidate result from the candidate masks based on the confidence score and the fitting evaluation score, and judging whether the fitting evaluation score of the first candidate result meets a preset stability condition; if yes, the first candidate result is determined as the final segmentation result of the current frame; if not, a similarity evaluation score between each candidate mask and a historical segmentation result stored in a historical state bank is calculated, and a second candidate result is selected from the candidate masks as the final segmentation result of the current frame based on the similarity evaluation score.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of target tracking, and particularly relates to a target tracking method based on two-stage selection. BACKGROUND

[0002] Target tracking is one of the core tasks in the field of computer vision, aiming to continuously locate and frame or segment the target of interest from video sequences. Anti-UAV target tracking is a typical target tracking task. In the field of anti-UAV target tracking, current algorithms can be divided into two types: special-purpose and general-purpose.

[0003] Special-purpose algorithms refer to methods specially designed for anti-UAV target tracking tasks and obtained after training on anti-UAV target tracking datasets. MJCT is one of the best algorithms currently performing on the AntiUAV series, which is a representative dataset in the field of anti-UAV target tracking. The algorithm includes three key modules: a robust motion constraint module, a flexible spatial remapping module, and an adaptive template update strategy. The robust motion constraint module can constrain the position and size changes of the tracking results based on the motion consistency of the target, reducing the problem of tracking drift. The flexible spatial remapping module solves the problem of large displacement of the target between frames based on spatial consistency by identifying the corresponding relationship between adjacent frames. The adaptive template update strategy dynamically selects a substitute template to maintain feature consistency in the face of dramatic changes in target appearance, ensuring stable tracking when the target pose changes dramatically. However, this technology has the following shortcomings: ① It lacks prediction of the future state of the target, which may lead to tracking failure due to the rapid changes in the state of the UAV during rapid motion. ② Since this method has only been trained on a small number of infrared modal anti-UAV datasets, it cannot directly adapt to video data of other modalities such as visible light. ③ All special-purpose methods, including this algorithm, have the common defect that the limited training data limit their generalizability, and their performance often drops significantly when faced with UAV models or background environments that have not been seen before.

[0004] The general method refers to a general tracking or video segmentation method. Such a method is often trained based on a large amount of general task data set, and is suitable not only for anti-UAV scenes, but also for various other tracking or segmentation tasks. SAMURAI is a representative general video segmentation and tracking method, which is improved based on a general segmentation model SAM2, uses a Kalman filter to predict object motion, and improves the tracking effect of moving objects by optimizing the selection logic of three candidate boxes. Compared with the special algorithm, the basic model SAM2 behind SAMURAI has seen a large amount of video data in the training process, and has segmentation capability for various objects, so it can be directly adapted to videos of various modalities such as infrared and visible light, and can process various types of drones in various scenes, and has obvious advantages in generalization. However, SAMURAI still has the following defects: ① Since the Kalman filter is based on the assumption that the object moves linearly when modeling motion information, it is difficult to handle complex nonlinear motion, which is not uncommon in UAV flight scenes. ② During the process of capturing the rapid motion of the UAV, the camera often needs to change the angle quickly. Due to the frame rate limitation of the camera, the UAV may have discontinuous instantaneous displacement in the picture. After SAMURAI performs motion prediction, it directly calculates the motion credibility based on the degree of coincidence of the candidate box position and the predicted value. This calculation method is sensitive to the position of the object in the picture, and it is difficult to handle the case where the UAV has instantaneous displacement in the picture. ③ When evaluating the quality of the candidate box, SAMURAI only considers the matching degree with the motion prediction result, but does not consider the degree of consistency with the target historical state, so it is difficult to achieve consistent tracking in some complex scenes, and it is easy to miss the target.

[0005] Based on the deficiencies of the above two representative existing technologies in the anti-UAV target tracking task, the present application designs a new target tracking algorithm, which has strong robustness on one hand, can combine the motion prediction result and the historical state of the target to select the optimal result, and can handle complex nonlinear motion, sudden changes in camera angle and other extreme scenes; on the other hand, it has strong generalization and can freely adapt to various modalities, scenes and models.

[0006] It should be noted that the information disclosed in the above background section is only for understanding the background of the present application, and therefore can include information that does not constitute prior art known to those of ordinary skill in the art. SUMMARY

[0007] The top technical problem to be solved by the present application is: how to significantly improve the robustness and accuracy of a visual target tracking system in a complex dynamic environment while maintaining high generalization, specifically, how to provide a target tracking method that can handle complex nonlinear motion and sudden changes in camera angle while maintaining good generalization performance, overcoming the problems of incomplete motion modeling and poor adaptability to extreme scenarios in the prior art. To this end, the present application provides a target tracking method based on two-stage selection.

[0008] The technical solution adopted by the present application to solve the above technical problems is as follows.

[0009] The first aspect of the present application provides a target tracking method based on two-stage selection, comprising the following steps:

[0010] S1, obtaining an input video stream sequence, for a current frame, extracting image features and generating at least one candidate mask and a confidence score corresponding to each candidate mask based on memory bank information;

[0011] S2, performing a first mask selection stage: predicting the motion state of the target in the current frame based on a motion prediction model, and calculating the fit evaluation score between each candidate mask and the predicted motion state;

[0012] S3, based on the confidence score and the fit evaluation score, selecting a first candidate result from the candidate masks, and determining whether the fit evaluation score of the first candidate result meets a preset stability condition;

[0013] S4, if it is met, the first candidate result is determined as the final segmentation result of the current frame, if it is not met, a second mask selection stage is performed: calculating the similarity evaluation score between each candidate mask and the historical segmentation result stored in the historical state bank, and based on the similarity evaluation score, selecting a second candidate result from the candidate masks as the final segmentation result of the current frame.

[0014] In some embodiments, the fit evaluation score of step S2 includes a motion score and a geometric score, the motion score is calculated based on the degree of overlap between the candidate mask and the motion prediction position, and the geometric score is calculated based on the similarity degree of the candidate mask and the motion prediction position in geometric features, the geometric features including at least one of aspect ratio and area.

[0015] In some embodiments, step S3 further comprises the following steps: weighted sum of the confidence score, the motion score and the geometric score; selecting the candidate mask with the highest weighted score as the first candidate result.

[0016] In some embodiments, the similarity evaluation score of step S4 comprises a reflection score based on appearance features and a reflection score based on geometric shapes; the reflection score based on appearance features is obtained by calculating the similarity of image features of the candidate mask and the image features of the historical segmentation result; the reflection score based on geometric shapes is obtained by calculating the similarity of geometric shapes of the candidate mask and the historical segmentation result.

[0017] In some embodiments, step S4 further comprises the following steps: weighted summing the reflection score based on appearance features and the reflection score based on geometric shapes; selecting the candidate mask with the highest weighted score as the second candidate result.

[0018] In some embodiments, the method further comprises the following steps: updating the final segmentation result to the historical state library and the memory library; determining whether to add the feature information of the current frame to the memory library based on the confidence score and / or the motion score of the final segmentation result of the current frame; only when the confidence score and / or the motion score is higher than a preset quality threshold, the feature information of the current frame is added to the memory library.

[0019] In some embodiments, the motion prediction model of step S2 is a neural network model trained for predicting target motion, and the neural network model comprises one of a long short-term memory network (LSTM), a recurrent neural network (RNN) and a multi-layer perceptron (MLP).

[0020] In some embodiments, the method is applied to the field of anti-UAV, and the video stream sequence is from a monitoring device, and the target is a UAV.

[0021] The second aspect of the present application provides a computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to implement the target tracking method based on two-stage selection provided by the first aspect of the present application.

[0022] The third aspect of the present application provides a video target tracking device, comprising: a memory for storing a computer program; a processor coupled to the memory for executing the computer program to implement the target tracking method based on two-stage selection provided by the first aspect of the present application.

[0023] The fourth aspect of the present application provides an anti-UAV system, comprising: an imaging unit for acquiring a video stream containing a UAV; a processing unit connected to the imaging unit for performing the target tracking method based on two-stage selection provided by the first aspect of the present application on the video stream to track the UAV; and a response unit connected to the processing unit for taking measures against the UAV based on the output of the processing unit.

[0024] The fifth aspect of the present application provides an intelligent monitoring system, comprising: an image acquisition device; an analysis server, which is communicatively connected to the image acquisition device and is configured to execute the target tracking method based on two-stage selection provided in the first aspect of the present application to track a specific target in a video stream.

[0025] The sixth aspect of the present application provides a computer-readable storage medium, which stores a computer program, and the computer program can generate bit stream data for target tracking and implement the target tracking method based on two-stage selection provided in the first aspect of the present application when executed by a processor.

[0026] The present application has the following beneficial effects:

[0027] The present application realizes high-precision and high-stability real-time tracking of a target in a wide range of application scenarios and modalities, and specifically has the following effects: the present application introduces a two-stage mask selection mechanism, first uses motion state prediction to make a proactive and future-oriented state selection, and quickly responds to target changes when the prediction is reliable; when the prediction is unreliable, a conservative reflection selection based on a historical state library is enabled to ensure the consistency of tracking, thereby realizing robust and accurate target tracking in complex nonlinear motion, sudden changes in camera perspective, target occlusion and appearance changes and other extremely challenging scenarios on the basis of high generalization. Specifically, the motion state prediction and fitting evaluation of the first stage can effectively handle the continuous motion of the target, and the similarity evaluation of the second stage provides a reliable backup mechanism when the motion prediction fails (such as instantaneous displacement and reappearance after occlusion), and the logical progression and collaborative work of the two mechanisms constitute a tracking system with strong adaptability and high fault tolerance, ultimately achieving the top-level effect of improving the robustness and accuracy of the overall tracking.

[0028] Further, the present application refines the fitting evaluation score into a motion score and a geometric score, so that the evaluation of the first stage not only focuses on the position coincidence degree, but also focuses on the shape similarity, thereby more effectively handling the case where the target appearance position jumps due to sudden changes in camera perspective but the shape remains basically unchanged, and improving the accuracy of the first-stage selection. In addition, the present application integrates the confidence score, the motion score and the geometric score by weighted summation, which can flexibly balance the information from the target generation quality, the motion fitting degree and the shape fitting degree, so that the selection of the first candidate result is more comprehensive and reasonable.

[0029] Further, the application refines the similarity evaluation score into a reflection score based on appearance features and a reflection score based on geometric shapes, so that the reflection selection in the second stage can comprehensively evaluate the consistency of the candidate target with the historical state from two dimensions of appearance and shape, thereby accurately identifying the real target when the target rotates, is partially occluded, or the illumination changes, and enhancing the discrimination ability of the algorithm in complex scenes. In addition, the application synthesizes the reflection scores based on appearance features and geometric shapes by weighted summation, which can adjust the weights of the two reflection mechanisms according to the actual scene to adapt to the importance of appearance or shape stability in different situations, so that the selection of the second candidate result is more adaptive.

[0030] Further, the application introduces a memory selection mechanism based on confidence scores and motion scores, which ensures that only high-quality and high-credibility tracking results are added to the memory library for subsequent frame processing, effectively preventing low-quality or incorrect memory information from polluting the feature extraction and candidate generation process, and improving the stability of subsequent tracking from the source.

[0031] Other benefits of the application will be further described below. BRIEF DESCRIPTION OF DRAWINGS

[0032] The above and / or additional aspects and advantages of the present application will become apparent and more readily appreciated from the following description, taken in conjunction with the following drawings in which:

[0033] Figure 1 A flow mechanism related to the present application;

[0034] Figure 2 A two-stage judgment flowchart related to the present application;

[0035] Figure 3 A memory selection mechanism related to the present application;

[0036] Figure 4 A result diagram for the comparative test of various general-purpose algorithms using the AntiUAV-300 dataset of visible light modalities in Case 1.

[0037] Figure 5 A result diagram for the comparative test of representative general-purpose algorithms and special-purpose algorithms using the AntiUAV-300 and AntiUAV-410 datasets of infrared modalities in Case 2.

[0038] Figure 6 A result diagram for the comparative test of representative general-purpose algorithms and special-purpose algorithms using the AntiUAV-600 dataset of infrared modalities in Case 2.

[0039] Figure 7Target tracking diagram corresponding to sample 1 in case 3;

[0040] Figure 8 Target tracking diagram corresponding to sample 2 in case 3;

[0041] Figure 9 Target tracking diagram corresponding to sample 3 in case 3;

[0042] Figure 10 Target tracking diagram corresponding to sample 4 in case 3;

[0043] Figure 11 Target tracking diagram corresponding to sample 5 in case 3;

[0044] Figure 12 Test result diagram corresponding to removing the motion predictor based on neural network or the reflection module respectively in deformation scheme 1 in case 4;

[0045] Figure 13 Test result diagram corresponding to changing the neural network structure based on which the motion predictor is based on in deformation scheme 2 in case 4;

[0046] Figure 14 Test result diagram corresponding to respectively trying to remove 1-2 of the three score indicators in deformation scheme 3 in case 4 when selecting the mask based on the motion prediction;

[0047] Figure 15 Test result diagram corresponding to respectively removing the three similarity indicators in the reflection module in deformation scheme 4 in case 4;

[0048] Figure 16 Test result diagram corresponding to changing the weight of the three similarity indicators in the weighted score in the reflection module in deformation scheme 4 in case 4;

[0049] Figure 17 Test result diagram corresponding to changing the motion score S m The set screening threshold corresponding to the test result diagram. DETAILED DESCRIPTION

[0050] The following detailed description of the embodiments of the present application. It should be emphasized that the following description is merely exemplary, and is not intended to limit the scope of the present application and its applications.

[0051] Moreover, the terms "first", "second", etc. are used herein only to describe different instances, and do not imply or suggest relative importance or a number of the technical features indicated. Therefore, the features defined as "first" and "second" can explicitly or implicitly include one or more of the features. In the description of the embodiments of the present application, the meaning of "a plurality" is two or more, unless otherwise specifically limited.

[0052] The core of the present application is to realize that single motion prediction or history matching can fail in complex scenes, therefore a two-stage, from aggressive prediction to conservative backtracking dynamic mask selection framework is proposed, supplemented by strict memory quality control mechanism, thereby ensuring generalization while greatly improving the robustness of tracking. Specifically, through the two-stage mask selection mechanism. For each frame of the input video stream, first generate multiple candidate masks and their confidence scores. The first stage evaluates the candidate masks based on the motion prediction results, and if the evaluation result is reliable, it is directly output. If it is not reliable, it enters the second stage, which evaluates and selects based on the similarity of the candidate masks and the historical state. Finally, the memory bank and the historical state bank are updated through the strict memory selection mechanism.

[0053] In some embodiments, a flow mechanism diagram of a target tracking method based on two-stage selection of the present application is shown in FIG. 1, which is described with reference to Figure 1 wherein the image encoder, memory attention, mask decoder, memory encoder and memory bank are all original parts of SAM2 and SAMURAI. The present application introduces a neural network-based motion predictor, reflection module, historical state bank and memory selection mechanism on the basis of them.

[0054] For a segment of input video stream, the algorithm will process the pictures contained therein in sequence frame by frame. Each new picture input will be encoded by the image encoder of SAM2, and the historical memory will be fused through the memory attention mechanism of SAM2, and then input into the mask decoder, which will decode and output 3 candidate masks and the corresponding 3 confidence scores S IoU For the 3 masks, the present application will perform two-stage mask selection through the neural network-based motion predictor and the reflection module. After selecting the final result, it will be added to the historical state bank and the memory bank. These two banks are used to store the latest segmentation results, and the historical state bank will be used for mask selection, while the memory bank will be used for memory attention mechanism after selection. It can be understood that the confidence score SIoU is a value output by the mask decoder at the same time as the mask. Its intuitive explanation is: the model thinks that the fitting degree between the result it predicts and the ground truth.

[0055] Reference is made to Figure 2The mask selection mechanism of the two stages and the process of memory selection are introduced in detail below.

[0056] In some embodiments, the core idea of the first-stage mask selection is "aggressive target state prediction based on motion prediction". The present application trains a neural network-based motion predictor based on a large number of UAV motion trajectories, which is based on a 4-layer LSTM with a hidden layer dimension of 64, an input dimension of 40, and an output dimension of 8. Compared with Kalman filter, the neural network-based motion predictor can better handle complex nonlinear motion conditions. The present application extracts 161895 pieces of trajectory data from the training set annotations of the Anti-UAV300 dataset for training the neural network-based trajectory predictor. The specific extraction method is as follows: read the complete coordinate annotation sequence contained in each video in the dataset, obtain the bounding box annotation corresponding to each frame, and form an effective trajectory data (5 frames as input + 1 frame as output) by combining the data corresponding to every 6 consecutive frames. During the training process, the optimizer is Adam, the initial learning rate is 1 e-3 , the number of training rounds is 50, and the loss function is defined as follows:

[0057]

[0058] where L MSE and L CIoU are the commonly used MSE loss function and CIoU loss function in the field of computer vision, , are the weights determined according to the performance tuning of the verification set, which are set to 1 and 100 respectively.

[0059] The present application models the object position state as an 8-dimensional vector, and the meanings of each dimension are as follows: the horizontal coordinate x of the center point of the bounding box, the vertical coordinate y of the center point of the bounding box, the width w of the bounding box, the height h of the bounding box, and the first-order difference of the four with the state of the previous frame 、 、 、 (which is used to represent the rate of change of each dimension).

[0060] During the tracking process, the present application maintains a history state library with a capacity of 5 and updated in a FIFO mechanism, which is used to store:

[0061] (1) the output mask result corresponding to each frame;

[0062] (2) the bounding box and 8-dimensional state vector obtained by converting the mask;

[0063] (3) The image features output by the last layer after the frame image is encoded by the image encoder of the SAM2. The motion predictor predicts the target position of the next frame based on the history state library. A 40-dimensional vector composed of 5 history states is input into the motion predictor, which will predict an 8-dimensional target state vector, and then the first 4 dimensions are taken to obtain the predicted bounding box position (i.e. the horizontal and vertical coordinates of the center point of the bounding box and the width and height of the bounding box).

[0064] By calculating the intersection-over-union of the predicted bounding box and the bounding box corresponding to the candidate mask, a motion score S reflecting the degree of coincidence of the two is obtained m . The calculation method of the intersection-over-union is the ratio of the intersection of the two to the union of the two in area. In addition, in order to deal with the situation of target instantaneous movement caused by sudden change of camera angle, the application also calculates a geometric score S g based on the similarity of the aspect ratio and area of the predicted position and the candidate mask, and the detailed calculation method is as follows.

[0065] Let B P represent the bounding box predicted by the motion predictor. For each candidate mask M, the corresponding bounding box B M is directly extracted. Define the function R(B) as the aspect ratio (w / h) of the bounding box B, define the function A(B) as the area (w·h) of the bounding box B, and define the function sim(x, y) = min(x, y) / max(x, y), then the calculation formula of S g is:

[0066]

[0067] wherein , is the weight determined according to the performance tuning of the verification set, which is set to 0.6 and 0.4 respectively. This score is based on the geometric features of the target to judge the similarity, which can ignore the deviation in position and thus deal with the situation of target instantaneous movement.

[0068] Finally, the mask selection in the first stage will be based on the weighted score calculated by S IoU , S m and S g , so as to well balance each measurement index. The calculation formula of the weighted score is as follows:

[0069]

[0070] wherein , , The weights determined according to the verification set performance tuning are respectively set to 0.7, 0.1, and 0.2. In this stage, the algorithm attempts to predict the new state of the target based on its past state, and selects the mask that best matches the expected state. However, in some extreme scenarios, prediction can be difficult, and in such cases, the focus needs to be turned back to the past to find the target state that best matches the past state, thus the second stage of the reflection module.

[0071] In some embodiments, the core idea of the second stage mask selection is "reflecting on the conservative target state retention based on historical state", which together with the first stage forms a strategy from aggressive to conservative, and well handles cases of different difficulties. If any one of the motion score S m and the geometric score Sg is lower than the corresponding threshold requirement and , the application will calculate the reflection scores S sr and S gr based on the reflection mechanisms based on appearance features and geometric shapes respectively. The detailed calculation process is as follows.

[0072] Based on the historical state library maintained by the motion predictor, the application reads the output mask M i corresponding to each frame i and the features F i output by the last layer after the corresponding frame image is encoded by the image encoder of SAM2. For each frame image, the Mask-Gated Average Pooling (MGAP) method is used to calculate the feature vector of the frame as the average representation of the mask region image features. Define as an indicator function, then The calculation process is defined as follows:

[0073]

[0074] Take the average of all 5 feature vectors calculated based on the historical state library to obtain . And for the current frame, the feature vector is calculated for the current frame image feature F and each candidate mask M using the same method. Define the function CosSim() as the cosine similarity, then the calculation formula of S sr is:

[0075]

[0076] Based on the historical state database maintained by the motion predictor, this invention averages the five bounding boxes contained therein to obtain the average bounding box BA. For each candidate mask M, its corresponding bounding box BM is directly extracted. The function R(B) is defined as the aspect ratio (w / h) of bounding box B, the function A(B) is defined as the area (w·h) of bounding box B, and the function sim(x,y) = min(x,y) / max(x,y) is defined. Then S... gr The calculation formula is:

[0077]

[0078] in , The weights, determined based on performance tuning on the validation set, were set to 0.4 and 0.6 respectively. This geometric score corresponds to S in the first stage. g The calculation methods are similar, and both measure geometric similarity, but the difference lies in: S in the first stage g The first measure is the similarity between the candidate mask and the predicted value, reflecting whether the shape of the candidate mask matches the predicted trend; while the second stage, S... gr It measures the similarity between the candidate mask and historical values, reflecting whether the shape of the candidate mask matches the historical state of the target.

[0079] Finally, the mask with the highest weighted score is selected based on their weighted sum. The calculation formula is as follows:

[0080]

[0081] in , The weights, determined based on the performance tuning of the validation set, are set to 0.05 and 0.95 respectively.

[0082] The weighted score above incorporates two reflection mechanisms. The appearance-based reflection mechanism compares the similarity between the image features of the candidate mask region and the image features of the historical result region after average pooling, thus identifying the target most similar in appearance to the historical result. The geometry-based reflection mechanism compares the geometry of the candidate mask region with the historical result, identifying the target most similar in geometry. In this stage, the algorithm abandons predictions about the future and instead evaluates the consistency of the target solely based on its historical state. This strategy enables robust target tracking in complex scenes.

[0083] refer to Figure 3 Regarding the memory selection mechanism, this invention introduces a confidence score-based mechanism on top of the existing SAM2 memory bank with a capacity of 7 and updated using a FIFO mechanism. IoUand motion score S IoU The mechanism of memory screening. Whenever the image encoder finishes encoding a new frame of image, before the memory attention step is needed, the invention will select 7 frames of memory according to the following steps for memory attention.

[0084] Firstly, the invention will directly select the initial frame given the target location hint to the first position of the 7-frame memory bank, and directly select the latest frame of memory to the last position. Then, for the remaining 5 positions, fill them according to the following logic: starting from the latest memory frame, traverse the history memory forward, screen out S IoU and S m memory frames that reach the corresponding threshold (the screening threshold σ IoU and σ m are 0.5 and 0.7 respectively) into the candidate pool. When all the history memories are traversed, or the capacity of the candidate pool reaches 30 frames, the traversal stops. Then, for each memory frame in the candidate pool, the invention will calculate its weighted score of S IoU and S m , select the top 5 frames with the highest weighted score, and fill them into the remaining 5 positions of the memory bank in the order of their time sequence. The formula for calculating the weighted score is as follows:

[0085]

[0086] wherein , are the weights determined according to the performance tuning of the validation set, set to 0.5, 0.5 respectively.

[0087] This selection mechanism can ensure that memory frames with stable target motion states are screened out, thereby strictly controlling the quality of memory frames and effectively preventing low-quality memories from polluting image features.

[0088] It should be noted that for all the hyperparameters mentioned above, the invention uses the validation set in the AntiUAV-300 dataset to determine them through grid search, and the optimization process aims to maximize the accuracy (Acc). The final results after optimization are as shown above. For the scores S IoU , S m , S g , S sr , S gr mentioned above, their values are ensured to be in the interval [0, 1] when designed, thus ensuring the fairness of weighted summation and eliminating the need for additional normalization operations.

[0089] It should be noted that the above technical solutions can adapt to most scenarios, but in some extreme scenarios, such as when the target moves smoothly but the appearance suddenly changes greatly (for example, a drone suddenly turns on a strong light), S m and S g may still be high and will not trigger the reflection module. At this time, the selection of the first stage may select the wrong target (such as tracking another similar object that is not lighted) because it does not have appearance similarity checks because it relies too much on movement and history. Therefore, as a preferred embodiment, a third judgment condition can also be introduced: when the confidence score S IoU of the best candidate mask is lower than the threshold τ IoU , the second stage reflection module is also forced to trigger. In this way, the situation of appearance change and motion trajectory coincidence can be dealt with.

[0090] Further, the reflection module only compares history, and if the target appears after being blocked for a long time, its appearance and shape may have changed legally (such as flying out after being blocked by a cloud layer), the reflection module may mistakenly think that this "new" target does not match the history and ignore it, resulting in losing or following the wrong target. Therefore, as a preferred embodiment, a rejection mechanism can also be introduced for the reflection module: if all candidate masks corresponding to the geometric similarity score S gr are lower than the set threshold, or all candidate masks corresponding to the appearance similarity score S sr are lower than the set threshold, it is considered that the appearance or shape of the target has changed legally, and the target consistency assumption based on the reflection module does not hold at this time, and the result given also lacks credibility. Therefore, in this case, the result of the reflection module is abandoned, and the mask with the highest confidence score S IoU is selected as the final result. In this way, the extreme case of motion trajectory not meeting the prediction and appearance or shape also changing greatly can be handled.

[0091] Compared with the prior art, the present application has the following beneficial effects:

[0092] BACKGROUND The Kalman filter adopted by SAMURAI is based on the assumption of linear state space model, whose state transition function and observation model are both linear, so it has poor fitting ability for nonlinear motion patterns. The present application adopts a motion predictor based on neural network. Various types of neural networks have strong nonlinear function approximation ability and can automatically learn the state evolution law from data. The LSTM adopted by the best embodiment of the present application can selectively retain and update the historical state in the time dimension by introducing a gating memory unit (input gate, forget gate and output gate) with nonlinear transformation, so as to learn the complex, nonlinear and long-term dependent motion evolution law from sequence data. Therefore, the motion predictor based on neural network introduced in the present application has stronger expression ability when modeling nonlinear motion in the real world.

[0093] Based on the prediction result of the motion predictor, the present application takes the weighted result of the motion score S m and the geometric score S g as the basis for mask selection. Compared with the background technology SAMURAI which simply uses the motion score S m , the introduction of the geometric score S g can better handle the transient motion problem caused by camera motion. This is because when camera transient motion occurs, the target position in the pictures of the two frames before and after may change instantaneously. At this time, there may be a large deviation in the coordinate position between the moved target and the predicted value, resulting in a small degree of overlap (S m is unreliable), but the geometric shape will not change significantly due to the position movement, and still has predictability (S g is reliable). Including S g in the weighted score can reduce the misjudgment caused by the abnormality of S m when camera transient motion occurs (for example, because S m of the correct mask A is slightly smaller than another incorrect mask B, the incorrect mask B with obvious incorrect geometric shape is directly selected), so the mask that best fits the motion prediction result is selected from the geometric shape dimension, thus alleviating the problem caused by transient motion.

[0094] Unlike the single-stage mask selection strategy adopted by the background technology SAMURAI, the present application also introduces a second-stage reflection module. This stage will reevaluate and select the mask based on the geometric shape and appearance features of the target in the historical state, both of which are beneficial to maintaining the consistency of the target, for the following reasons: the shape of the unmanned aerial vehicle target is often stable during flight, and will not change significantly in shape, size or appearance in a short period of time. When selecting a mask based on the geometric score S gr , a target similar to the historical state of the target in geometric features can be selected, which is beneficial to exclude some objects that are similar in color but different in shape. The appearance score Ssr According to the calculation based on the image features after the image is encoded by the SAM2 image encoder, the similarity of the mask area on the image features can reflect the appearance feature similarity of the candidate mask and the historical target in the high-dimensional semantic space, and thus it is beneficial to exclude some objects that are similar in shape but different in appearance. In summary, the reflection module of the present application ensures the consistency of the target in the tracking process from two aspects of geometric shape and appearance feature.

[0095] Some specific implementation cases are provided below to embody the beneficial effects of the present application.

[0096] Case one: refer to Figure 4 , the AntiUAV-300 dataset of the visible light mode is used to compare and test various general algorithms. It can be seen that the method proposed in the present application is superior to other existing methods in the accuracy (Acc), precision (P) and area under the success rate-IoU threshold curve (AUC) indicators. This proves the effect of the two-stage mask selection mechanism and the memory screening mechanism proposed in the present application on improving the tracking indicators.

[0097] Case two: refer to Figure 5 , 6 , the AntiUAV-300, AntiUAV-410 and AntiUAV-600 datasets of the infrared mode are used to compare and test representative general algorithms and special algorithms. It can be seen that under the infrared mode, the method proposed in the present application is also superior to other existing methods, which also proves the technical effects brought by the improvements proposed in the present application.

[0098] Case three: the visual results will be compared to further prove the effects of the modules of the present application.

[0099] Example 1: refer to Figure 7 , the scene of the UAV target flying in front of a building background with complex texture and similar color is shown, and the target is temporarily occluded. In this result, it can be seen that due to the introduction of the geometric score in the mask selection step of the motion predictor, the prediction of the target shape by the method of the present application is more accurate, while the SAM2.1 and SAMURAI appear to have incomplete mask shapes due to the interference of the background. In addition, in the face of the scene where the target is occluded and then reappears, the method of the present application realizes faster target recovery due to its more perfect memory selection mechanism. The other two methods temporarily lose the target after the target reappears.

[0100] Example 2: refer to Figure 8, which shows the extremely small UAV target flying in the night with low light. Under the interference of the complex light source in the background and the auxiliary line in the picture, the baseline model frequently appears to be wrong with the target and loses the target, while the algorithm of the application realizes robust and stable tracking. This proves that the nonlinear motion modeling mechanism introduced in the neural network-based motion predictor of the application can greatly improve the correctness of the mask selection and output the tracking result that best meets the motion law.

[0101] Example 3: Reference Figure 9 , the camera view is further introduced on the basis of example 2. At the moment of view movement, SAM2.1 without motion modeling mechanism and SAMURAI with only linear motion modeling mechanism both lose the target. The reflection mechanism introduced in the application helps the algorithm to find the target that is most consistent with the historical state, and even if the target is highly similar to the light background, the correct mask result can be obtained.

[0102] Example 4: Reference Figure 10 , in addition to the rapid movement of the camera view, the appearance change caused by the target rotation is also introduced. From frame 797 to frame 868, the UAV target rotates to the right in the picture, causing its appearance to change from two green light points to one red and one green light point. During this process, SAM2.1 and SAMURAI gradually only segment the local part of the target, while the method of the application can always accurately give the complete contour of the target, which proves the robustness of the mask selection mechanism based on geometric shape and the more perfect memory selection mechanism introduced in the application when facing the appearance change of the target.

[0103] Example 5: Reference Figure 11 , which shows the complex situation that the UAV frequently disappears and reappears due to the occlusion of dense obstacles. In this case, the application shows a significant advantage in target recovery ability compared with SAM2.1 and SAMURAI. When the target reappears, the method of the application can always quickly recover, and after being occluded for many times, it always maintains consistent tracking of the same target, which reflects the strong robustness of the memory selection mechanism of the application in complex scenes.

[0104] Case Four: Provide some variant schemes of the application.

[0105] Variant Scheme 1: Reference Figure 12, respectively, the motion predictor based on neural network or the reflection module can be removed, and a deformation scheme with only a single-stage mask selection mechanism can be obtained. Both deformation schemes are slightly inferior to the best scheme in terms of indicators. After removing the reflection module, the Acc, P, and AUC indicators on the visible light modal data of the AntiUAV-300 test set decrease by 0.3%, 0.3%, and 0.2%, respectively, because the absence of the reflection module leads to poor consistency of tracking. Further removing the neural network-based trajectory predictor based on this, the Acc, P, and AUC indicators decrease by 4.5%, 4.7%, and 4.3%, respectively, because the absence of the motion predictor causes the model to ignore the motion trend of the target when selecting the mask.

[0106] Deformation scheme 2: refer to Figure 13 , changing the neural network structure based on the motion predictor, two deformation schemes of MLP and RNN can be obtained, and their performance in various indicators is about 1% lower than that of the best scheme using LSTM. Although both MLP and RNN neural networks have the ability to fit nonlinear functions, the gated memory unit (input gate, forget gate, and output gate) of the nonlinear transformation of LSTM can better learn complex and nonlinear motion rules from sequence data, thus achieving more accurate prediction.

[0107] Deformation scheme 3: refer to Figure 14 When selecting a mask based on motion prediction, 1-2 of the three score indicators are removed, respectively, to obtain several deformation schemes, all of which perform worse than the optimal scheme that uses all three score indicators. Removing S IoU causes P and AUC indicators to decrease significantly, because the selection of the mask ignores the most basic mask confidence factor, resulting in the selection of a poor quality mask. Removing S g causes Acc and AUC indicators to decrease significantly, because the selection of the mask does not discriminate based on the geometric shape, and is easily disturbed by target transient due to camera shaking. Removing S m causes Acc and AUC indicators to decrease significantly, because the selection of the mask does not discriminate based on the position output by the motion predictor, making it vulnerable to interference from similar objects at other positions.

[0108] Deformation scheme 4: refer to Figure 15 , 16 In the reflection module, removing the three similarity indicators or changing their weights in the weighted score can obtain several deformation schemes, all of which perform worse than the optimal scheme. Removing S grAfter removing or reducing the area similarity component or the aspect ratio similarity component in the similarity score, or reducing the proportion of the similarity score, the Acc, P, and AUC indicators all decrease significantly, because the tracking process ignores the consistency of the target geometry and is easily disturbed by appearance similar objects. After removing or reducing the Ssr that measures the feature similarity or reducing the proportion of the Ssr, the three indicators also decrease significantly, because the tracking process ignores the consistency of the target appearance features and is easily affected by similar interference objects in size or shape.

[0109] Variant 5: Refer to Figure 17 In the memory selection mechanism, the motion score S m The screening threshold is set A plurality of variants can be obtained, and their performances are all worse than the optimal variant. Increasing the threshold will result in more stringent screening of memories, and the selected memory frames are either scarce in quantity or too scattered in time span, and thus cannot fully represent the state of the UAV target in a short period, thereby reducing the output mask quality. Reducing the threshold will result in too loose screening of memories, and the selected memory frames are of poorer quality and may contain incorrect tracking results, thereby affecting the output mask quality.

[0110] In summary, the present application introduces a neural network-based motion predictor, which uses a neural network to predict the target position and effectively cope with nonlinear complex motion scenarios. The module receives the past 5 eight-dimensional state vectors in the historical state library as input, predicts the eight-dimensional state vector of the target in the next frame, and thus extracts the target position. Based on this prediction result, the present application introduces a mechanism of motion score and geometric score weighted evaluation, which effectively copes with sudden changes in camera angle, selects the mask that conforms to the prediction from the two aspects of motion trend and geometric shape change trend; the present application introduces a reflection module, which is started when the motion predictor is not reliable, reads the historical geometric shape and historical appearance features of the target from the historical state library, and simultaneously reevaluates all candidate masks based on the two factors to select the most consistent mask, thereby realizing robust tracking of consistency; the present application optimizes the memory selection mechanism by including the motion score in the screening index, based on the confidence score and the motion score, first selects a certain capacity of memory candidate frames, and then selects the best memory from them, strictly controls the memory quality, ensures that the selected memory has high reliability and stable motion state, and effectively improves the quality of the generated mask.

[0111] Those skilled in the art will appreciate that embodiments of the present application can be readily used as a method, a system or a computer program product. Accordingly, the present application can take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present application can take the form of a computer program product on one or more computer readable storage media (including, but not limited to, disk memory, CD-ROMs, optical storage devices, etc.) embodying computer readable program code.

[0112] The present application is described in reference to the flowchart illustrations and / or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general purpose computer, special purpose computer, embedded processing system or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart illustrations and / or block diagrams block or blocks. Figure 1 one or more functions specified in the flowchart illustrations and / or block diagrams block or blocks. Figure 1 one or more functions specified in the flowchart illustrations and / or block diagrams block or blocks.

[0113] These computer program instructions can also be stored in a computer readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer readable memory produce an article of manufacture including instructions which implement the flowchart illustrations and / or block diagrams block or blocks. Figure 1 one or more functions specified in the flowchart illustrations and / or block diagrams block or blocks. Figure 1 one or more functions specified in the flowchart illustrations and / or block diagrams block or blocks.

[0114] These computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the flowchart illustrations and / or block diagrams block or blocks. Figure 1 one or more functions specified in the flowchart illustrations and / or block diagrams block or blocks. Figure 1 one or more functions specified in the flowchart illustrations and / or block diagrams block or blocks.

[0115] The background section of the present application can include information about background issues or context of the present application, and does not necessarily constitute prior art. Therefore, the contents included in the background section are not admissions of prior art by the applicant.

[0116] The above further describes the present application in conjunction with specific / preferred embodiments, and cannot be deemed to limit the specific implementation of the present application to these descriptions. For those skilled in the art to which the present application belongs, without departing from the concept of the present application, they can make several substitutions or variations to the described embodiments, and these substitutions or variations shall be deemed to fall within the protection scope of the present application. In the description of the present application, the description of the terms "an embodiment", "some embodiments", "a preferred embodiment", "an example", "a specific example", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are contained in at least one embodiment or example of the present application. In the present application, the illustrative description of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner. In the case of no mutual contradiction, those skilled in the art can combine and combine the different embodiments or examples described in the present application and the features of the different embodiments or examples. Although the embodiments of the present application and their advantages have been described in detail, it should be understood that various changes, substitutions and modifications can be made herein without departing from the scope of protection of the patent application.

Claims

1. A target tracking method based on two-stage selection, characterized in that, Includes the following steps: S1. Obtain the input video stream sequence. For the current frame, extract image features and generate at least one candidate mask and a confidence score corresponding to each candidate mask based on the memory bank information. S2. Perform the first mask selection stage: predict the motion state of the target in the current frame based on the motion prediction model, and calculate the matching evaluation score between each candidate mask and the predicted motion state. S3. Based on the confidence score and the matching evaluation score, select a first candidate result from the candidate mask, and determine whether the matching evaluation score of the first candidate result meets the preset stability condition. S4. If satisfied, the first candidate result is determined as the final segmentation result of the current frame. If not satisfied, the second mask selection stage is executed: calculate the similarity evaluation score between each candidate mask and the historical segmentation result stored in the historical state library, and select the second candidate result from the candidate masks as the final segmentation result of the current frame based on the similarity evaluation score.

2. The method according to claim 1, characterized in that, The matching evaluation score in step S2 includes a motion score and a geometric score. The motion score is calculated based on the degree of overlap between the candidate mask and the motion prediction position, and the geometric score is calculated based on the degree of similarity between the candidate mask and the motion prediction position in terms of geometric features, including at least one of aspect ratio and area.

3. The method according to claim 2, characterized in that, Step S3 further includes the following steps: weighted summation of confidence score, motion score and geometry score; and selection of the candidate mask with the highest weighted score as the first candidate result.

4. The method according to claim 1, characterized in that, The similarity evaluation score in step S4 includes a reflection score based on appearance features and a reflection score based on geometry; the reflection score based on appearance features is obtained by calculating the similarity between the image features of the candidate mask and the image features of the historical segmentation results; The geometry-based reflection score is obtained by calculating the similarity in geometry between the candidate mask and the historical segmentation result.

5. The method according to claim 4, characterized in that, Step S4 further includes the following steps: performing a weighted summation of the reflection score based on appearance features and the reflection score based on geometry; selecting the candidate mask with the highest weighted score as the second candidate result.

6. The method according to claim 1, characterized in that, The method also includes the following steps: updating the final segmentation result to the historical state database and the memory database; determining whether to add the feature information of the current frame to the memory database based on the confidence score and / or motion score of the final segmentation result of the current frame; and adding the feature information of the current frame to the memory database only when the confidence score and / or motion score is higher than a preset quality threshold.

7. The method according to claim 1, characterized in that, The motion prediction model in step S2 is a neural network model trained to predict the motion of a target. The neural network model includes one of Long Short-Term Memory Network (LSTM), Recurrent Neural Network (RNN), and Multilayer Perceptron (MLP).

8. The method according to claim 1, characterized in that, The method is applied in the field of anti-drone operations, where the video stream sequence comes from a monitoring device and the target is a drone.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the computer program implements the target tracking method based on two-stage selection as described in any one of claims 1 to 8.

10. A video target tracking device, characterized in that, include: Memory, used to store computer programs; A processor, coupled to the memory, is used to execute the computer program to implement the target tracking method based on two-stage selection as described in any one of claims 1 to 8.

11. An anti-drone system, characterized in that, include: Imaging unit, used to acquire video streams containing drones; A processing unit, connected to the imaging unit, is configured to perform a target tracking method based on two-stage selection as described in any one of claims 1 to 8 on the video stream to track the drone; a response unit, connected to the processing unit, is configured to take countermeasures against the drone based on the output of the processing unit.

12. An intelligent monitoring system, characterized in that, include: Image acquisition equipment; An analysis server, communicatively connected to the image acquisition device, is configured to perform a target tracking method based on two-stage selection as described in any one of claims 1 to 8, to track a specific target in a video stream.

13. A computer-readable storage medium, characterized in that, It stores a computer program that, when executed by a processor, can generate bitstream data for target tracking and implement the target tracking method based on two-stage selection as described in any one of claims 1 to 8.

Images