Shielding target stable tracking method and device based on trajectory prediction

Abstract

The invention provides an occlusion target stable tracking method and device based on trajectory prediction. According to the method, feature extraction and target detection are carried out on a single-frame image, then matching of detection frames is carried out according to target detection results of multiple-frame images, and when the detection frames are mismatched, it is determined that a target is shielded; the target shielding time is judged, and if the target shielding time is short shielding, a least square method is used for target trajectory prediction; and if the shielding is long, performing target trajectory prediction by using Kalman filtering. The invention further provides an association strategy, when the detection frames are mismatched, the mismatched detection frames are associated with the high-value detection frames and the prediction frames respectively, and the high-value detection frames can comprise non-deterministic state detection frames with association values, super-life detection frames and the like. By using the method, the problem of multi-target tracking identity switching after occurrence of shielding can be solved, the anti-shielding capability of a multi-target tracking algorithm is improved, and the stable tracking effect after multi-target shielding is ensured.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to a method and device for stably tracking an occluded target based on trajectory prediction. Background Art

[0002] As an important research topic in the field of computer vision, multi-target tracking has very important application value in the fields of intelligent monitoring, anomaly detection, autonomous driving, security, etc. with the continuous improvement of monitoring systems in recent years.

[0003] The key to target tracking technology is to establish the association between the current frame image and the next frame image. Target detection in a single frame image is an important step to provide a stable initial tracking position, but it cannot complete data association. Associating the detection position of each frame image with the position predicted by the motion trajectory to give the target identity is the key to stable multi-target tracking.

[0004] The field of multi-target tracking is mainly divided into two categories, offline tracking algorithms and online tracking algorithms. Offline tracking algorithms cannot be applied to real-time tracking, and online tracking algorithms are the current research trend in the field of multi-target tracking. The current online real-time tracking algorithm is studied on the basis of the SORT (Simple Online and Realtime Tracking) algorithm. The SORT algorithm decomposes the multi-target tracking problem into the target bounding box detection part, which is responsible for predicting and updating the state prediction of trajectory information, and the data association part, which is responsible for matching between targets and trajectories. In the SORT algorithm, the input image is output through the target detection algorithm to output the position and category of the target, and the state of each detected target is predicted and updated through the Kalman filter. Finally, the Hungarian algorithm is used to solve the cost matrix of the IOU matching between the predicted target and the target detected in the current frame. The SORT algorithm has a fast tracking speed but does not solve the frequent identity switching of the algorithm.

[0005] When the target is occluded, ensuring that the target identity does not switch is the core problem that needs to be solved for stable and continuous tracking of multiple targets. To address this problem, the DeepSort algorithm introduces a feature re-identification model (Reid) based on the Sort algorithm, using cosine distance to calculate the similarity of target feature information and Mahalanobis distance to constrain target motion information for data association, effectively reducing the number of identity switches.

[0006] Although the DeepSort algorithm is widely used in multi-target tracking, its tracking effect is not ideal for occluded targets. The Reid model cannot extract enough feature information, which affects the re-identification of targets and the association of target identities. For the multi-target anti-occlusion stable tracking task, it is still necessary to further improve the target trajectory association, reduce the target identity switching frequency, and thus improve the stability of the tracking algorithm. Summary of the invention

[0007] In view of this, the present invention provides a method for stable tracking of occluded targets based on trajectory prediction, which can improve the stable tracking effect of multiple targets after occlusion.

[0008] In order to solve the above technical problems, the present invention is implemented as follows.

[0009] A stable tracking method for an occluded target based on trajectory prediction, comprising:

[0010] Step 1: Perform feature extraction and target detection on a single frame image;

[0011] Step 2: Match the detection frame according to the target detection results of multiple frames of images. When the detection frame mismatch occurs, determine the target occlusion; determine the target occlusion time. If it is a short occlusion, execute step 3; if it is a long occlusion, execute step 4;

[0012] Step 3: In the case of short occlusion, the least square method is used to predict the target trajectory;

[0013] Step 4: In the case of long occlusion, use Kalman filtering to predict the target trajectory.

[0014] Preferably, in step 2, when the detection frame is mismatched, the mismatched detection frame is associated with the high-value detection frame and the prediction frame respectively, and the cosine distance between the mismatched detection frame and the high-value detection frame and the prediction frame is calculated. If the minimum cosine distance is less than a set threshold, the corresponding high-value detection frame or prediction frame is considered to be the most appropriate association item, and the mismatched detection frame is switched to a matching state, and it is considered that no occlusion occurs;

[0015] The high-value detection frame is a detection frame in which a target appears once.

[0016] Preferably, the high-value detection frame is used only once, and after being determined as a related item, the high-value detection frame is deleted from the high-value detection frame set.

[0017] Preferably, the high-value detection frame further includes a detection frame exceeding a maximum lifespan.

[0018] Preferably, in step 1, when extracting features from a single frame image, a dual-path attention feature extraction model is used;

[0019] The dual-path attention feature extraction model includes a self-attention feature extraction channel, a recurrent attention feature extraction channel, and a feature encoding module;

[0020] In the self-attention feature extraction channel, the input image is used as a positive sample and processed by the first convolution module and the first attention module in sequence to obtain the first feature;

[0021] In the cyclic attention feature extraction channel, the input image is cyclically shifted to obtain multiple negative samples, each negative sample is added and fused after being processed by the second convolution module, and then processed by the second attention module and the third convolution module to obtain a second feature; the second feature and the first feature obtained from the self-attention feature extraction channel are added and fused into a third feature;

[0022] After the first feature and the third feature are concatenated, feature encoding is performed to complete feature extraction.

[0023] Preferably, the first attention module and the second attention module have the same structure, both including a first convolutional layer, a second convolutional layer, a third convolutional layer, a first dot product module and a second dot product module; after the input data is processed by the first convolutional layer, the second convolutional layer and the third convolutional layer respectively, data Q, data K and data V are output; after data Q and data K are multiplied in the first dot product module, attention weight information is output; the attention weight information is multiplied with data V in the second dot product module, and feature information is output.

[0024] The present invention also provides a stable tracking device for an occluded target based on trajectory prediction, comprising a single-frame image target detection module, a detection frame matching module, an occlusion judgment module, a short occlusion prediction module, and a long occlusion prediction module;

[0025] The single-frame image target detection module is used to perform feature extraction and target detection on a single-frame image;

[0026] The detection frame matching module is used to match the detection frames according to the target detection results of multiple frames of images, and determine the target occlusion when the detection frames are mismatched;

[0027] The occlusion judgment module is used to judge the target occlusion time, if it is a short occlusion, then notify the short occlusion prediction module; if it is a long occlusion, then notify the long occlusion prediction module;

[0028] The short occlusion prediction module is used to predict the target trajectory using the least squares method under short occlusion conditions;

[0029] The long occlusion prediction module is used to predict the target trajectory using Kalman filtering in the case of long occlusion.

[0030] Preferably, the single-frame image target detection module includes a dual-path attention feature extraction model and a target detection module; the dual-path attention feature extraction model includes a self-attention feature extraction channel, a cyclic attention feature extraction channel and a feature encoding module;

[0031] The self-attention feature extraction channel includes a first convolution module and a first attention module processing; the input image is used as a positive sample, and is processed by the first convolution module and the first attention module in sequence to obtain a first feature;

[0032] The cyclic attention feature extraction channel includes a fusion module, and a cyclic attention module composed of a cyclic shift module, a second convolution module, an addition module, a second attention module, and a third convolution module; the cyclic shift module performs cyclic shift on the input image to construct a negative sample; each negative sample is processed by the second convolution module and then added and fused in the addition module; then, after being processed by the second attention module and the third convolution module, a second feature is obtained; the fusion module adds the first feature and the second feature to obtain a third feature;

[0033] The feature encoding module concatenates the first feature and the third feature, performs feature encoding, and completes feature extraction;

[0034] The target detection module predicts the target position and target identity based on the extracted features output by the feature encoding module, and completes the target detection of a single frame image.

[0035] Preferably, the first attention module and the second attention module have the same structure, both including a first convolutional layer, a second convolutional layer, a third convolutional layer, a first dot product module and a second dot product module; after the input data is processed by the first convolutional layer, the second convolutional layer and the third convolutional layer respectively, data Q, data K and data V are output; after data Q and data K are multiplied in the first dot product module, attention weight information is output; the attention weight information is multiplied with data V in the second dot product module, and feature information is output.

[0036] Preferably, when the detection frame is mismatched, the detection frame matching module further associates the mismatched detection frame with the high-value detection frame and the prediction frame respectively, calculates the cosine distance between the mismatched detection frame and the high-value detection frame and the prediction frame, and if the minimum cosine distance is less than a set threshold, the corresponding high-value detection frame or prediction frame is considered to be the most appropriate associated item, and the mismatched detection frame is switched to a matching state, and it is considered that no occlusion occurs; the high-value detection frame is a detection frame in which a target appears once and a detection frame that exceeds a maximum lifespan; a high-value detection frame is used only once, and when a high-value detection frame is determined to be an associated item, it is deleted from the high-value detection frame set.

[0037] Beneficial effects:

[0038] (1) The present invention adopts different methods to predict the target trajectory according to the length of the occlusion time. Among them, when the tracking target is occluded, the existing DeepSORT and SORT algorithms use Kalman filters to predict the trajectory. When the target is frequently occluded, the real measurement value is small, and the Kalman gain calculation is inaccurate, resulting in a large deviation between the Kalman filter state estimate and the real value, a large deviation in the trajectory prediction, and the failure of the detection frame association after the tracking target is occluded, and the identity is switched. Therefore, the present invention introduces a least squares algorithm to fit the target motion trajectory for frequent occlusions, and the target motion trajectory can be effectively predicted using a small number of video frame information; while for the tracking target with long occlusion, there are many real measurement values, the Kalman gain calculation is more accurate, and it is more appropriate to use a Kalman filter for prediction. The present invention distinguishes between short occlusion and long occlusion situations, can effectively predict the target motion trajectory in various situations, can solve the problem of identity switching in multi-target tracking after occlusion occurs, and improves the anti-occlusion ability of the multi-target tracking algorithm.

[0039] (2) In a preferred embodiment, in order to solve the problem that the target area in the image accounts for a small proportion and the overlapping target feature extraction is difficult, a dual-path self-attention mechanism model is designed, in which the recurrent attention channel and the self-attention module form a dual-channel parallel feature extraction structure; the recurrent attention pays special attention to the periodic patterns in the sequence. It captures the periodic features in the data by introducing an additional mechanism, which may be ignored in the traditional attention mechanism. The use of the recurrent attention channel improves the reliability and robustness of accurate extraction of multi-target features in complex scenes.

[0040] (3) In a preferred embodiment, a multi-target association strategy is proposed to retain some non-deterministic detection frames for target association, thereby improving target stability. Furthermore, detection frames with a lifespan exceeding the maximum can be retained to participate in target association, further improving target stability, solving the problem of multi-target tracking identity switching after occlusion occurs, and improving the anti-occlusion capability of the multi-target tracking algorithm.

[0041] (4) In a preferred embodiment, the high-value detection frame is used only once when associated, to prevent frequent switching caused by different tracking targets in a shorter video frame competing for the high-value detection frame.

[0042] (5) In a preferred embodiment, a preferred attention module structure is provided, which can weight and combine the features at various positions in the image to generate a more recognizable feature representation and highlight the positional relationship of the target in the image. This is particularly effective for detecting small targets in complex backgrounds because it can highlight the target by strengthening the weight of related features. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1The present invention is a flow chart of a stable tracking method for an occluded target based on trajectory prediction.

[0044] Figure 2 This is a schematic diagram of the principle of single-frame image target detection of the present invention.

[0045] Figure 3 This is a structural diagram of the cyclic attention module of the present invention.

[0046] Figure 4 The present invention is a block diagram of the composition of the occluded target stable tracking device based on trajectory prediction. DETAILED DESCRIPTION

[0047] The present invention is described in detail below with reference to the accompanying drawings and embodiments.

[0048] The present invention provides a stable tracking method for an occluded target based on trajectory prediction. Figure 1 As shown, the method comprises the following steps:

[0049] Step 1: Extract features from a single-frame image and complete single-frame image target detection.

[0050] like Figure 2 As shown, this embodiment provides a feature extraction scheme based on a dual attention mechanism, which is applied to single-frame image target detection. Single-frame image target detection is completed through a deep learning-based detection framework, and target position information is output.

[0051] Since the target area in the image accounts for a small proportion and it is difficult to extract features of overlapping targets, the present invention designs a recurrent attention module to form a dual-channel parallel feature extraction structure with the self-attention module. Figure 2 ,The target feature extraction model based on the dual-path attention mechanism includes a self-attention feature extraction channel, a recurrent attention feature extraction channel and a feature encoding module.

[0052] In the self-attention processing channel, the input image is used as a positive sample, and is processed by the first convolution module and the first self-attention module in sequence to obtain the first feature and complete the feature extraction of the target area. The attention mechanism is used to filter out a small amount of important information from a large amount of information, globally associate weights and make a weighted sum of the inputs, reduce dependence on external information, and focus on capturing the internal correlation of data or features.

[0053] In the cyclic attention feature extraction channel, there are two parts, the cyclic attention module and the fusion module. The input image is input into the cyclic attention module, and multiple negative samples are obtained through cyclic shift. Each negative sample is added and fused after convolution processing, and then processed by the second self-attention module and convolution operation is performed again to obtain the second feature. The second feature and the first feature obtained by the self-attention feature extraction channel are added and fused by the fusion module to obtain the third feature output.

[0054] The structure of the recurrent attention module is as follows Figure 3 As shown. The cyclic attention module includes a cyclic shift module, a second convolution module, an addition module, a second self-attention module, and a third convolution module. Among them, the cyclic shift module cyclically shifts the input image to construct a large number of negative samples. Circular shift can be achieved by translating pixels and padding with zeros. Each negative sample has the same upper and lower bounds in each channel. For any pixel, its color components can be considered to be independent and identically distributed. After convolution, each negative sample is added and fused in the addition module; then, after being processed by the second self-attention module and the third convolution module, the second feature is obtained. See also Figure 2 , the second feature output by the cyclic attention module is added to the positive sample feature (first feature) to obtain the third feature. The present invention uses a two-way self-attention mechanism, which is used to filter out a small amount of important information from a large amount of information, globally associate weights and do a weighted sum of the inputs, reduce dependence on external information, and focus on capturing the internal correlation of data or features, which makes the target area prominent, the algorithm is easy to focus on the target, the optimization direction is more accurate, and it can converge to the optimal solution faster.

[0055] Next, the first feature and the third feature both enter the feature encoding module, which concatenates the first feature and the third feature, performs feature encoding, and completes feature extraction.

[0056] In a preferred embodiment, the self-attention modules of the two attention channels adopt the same structure, such as Figure 2 and Figure 3As shown, they all include a first convolutional layer, a second convolutional layer, a third convolutional layer, a first dot product module and a second dot product module; after the input data is processed by the first convolutional layer, the second convolutional layer and the third convolutional layer respectively, data Q, data K and data V are output; after data Q and data K are multiplied in the first dot product module, attention weight information is output; the attention weight information and data V are multiplied in the second dot product module to output feature information. In practice, the self-attention module can also adopt other spatial attention models and channel attention models. The self-attention module structure provided in this embodiment can be weighted and combined according to the feature importance of each position in the image, so as to generate a more recognizable feature representation. This is particularly effective for small target detection in complex backgrounds, because it can highlight the target by strengthening the weight of related features. Moreover, the self-attention mechanism can dynamically adjust the degree of attention to different areas, making the model more flexible when processing images with complex backgrounds or multi-target scenes.

[0057] After image feature extraction is completed based on the dual-channel feature extraction architecture, target position and target identity are further predicted to complete target detection of a single frame image. In this embodiment, regression prediction is used to obtain the target position, and classification prediction is used to complete target identity confirmation.

[0058] Step 2: Match the detection frame according to the target detection results of multiple frames. When the detection frame mismatch occurs, determine the target occlusion; determine the target occlusion time. If it is a short occlusion, execute step 3; if it is a long occlusion, execute step 4.

[0059] In this step, when a mismatch detection frame appears, occlusion occurs, and the occlusion time is recorded. In this embodiment, two thresholds can be set. When the occlusion time is within the first threshold, it is considered that short occlusion occurs, and step 3 is executed; if the occlusion time exceeds the first threshold but is within the second threshold, it is considered that long occlusion occurs. If it exceeds the second threshold, it is considered that the target disappears or tracking fails.

[0060] In a preferred embodiment, a multi-target association strategy is proposed to handle target association in the case of detection frame mismatch.

[0061] In the existing processing, when a target appears in a single image but is not continuous, a single-frame image detection frame is obtained. The single-frame image detection frame may have a certain degree of uncertainty and belongs to an indeterminate detection frame. Common tracking algorithms often directly remove such detection frames, but such detection frames still have a certain probability of accurately predicting the target position. Therefore, the present invention designs a target association strategy to further utilize such detection frames to improve target stability. The discarded indeterminate detection frames still have association value and are retained in the association strategy of the present invention. At the same time, the detection frames that are deleted due to exceeding the maximum lifespan (such as stationary targets) are retained. This part of the detection frame also has association value for long-term target occlusion. Usually, most long-occluded target occlusions are deleted because the detection frame exceeds the maximum lifespan and lacks a suitable matching frame, resulting in association failure; the association strategy of the present invention sets these two parts of the detection frame as high-value detection frames and associates them after the target is occluded.

[0062] When a frame is associated, the mismatched detection frame is associated with the high-value detection frame and the prediction frame respectively. The cosine distance between the mismatched detection frame and the high-value detection frame and the prediction frame is calculated through the feature information extracted by the previous network model, and the minimum cosine distance is found. If the minimum pre-distance is less than the set threshold, the corresponding high-value detection frame or prediction frame is considered to be the most suitable association item, and the mismatched detection frame is switched to a matching state, and it is considered that no occlusion occurs. The high-value detection frame is only used once when it is associated to prevent frequent switching caused by different tracking targets in a shorter video frame snatching the high-value detection frame.

[0063] The formula for calculating cosine distance is:

[0064]

[0065] Among them, v1 represents the two-dimensional position vector of the center of the detection box, v2 represents the two-dimensional position vector of the center of the high-value detection box or prediction box; v1·v2 represents the dot product of the two vectors (i.e., ∑ i v 1i ×v 2i ), ‖v1‖, ‖v2‖ represent the norms of two vectors, that is, i represents the i-th element in the vector. The dot product can measure the similarity of two vectors in the same direction, the norm measures the length or size of the vector, and the cosine distance measures their similarity by calculating the cosine value of the angle between the two vectors. The smaller the cosine value, the higher the similarity. The cosine distance range is [0,2], where 0 means exactly the same, 1 means orthogonal (unrelated), and 2 means completely opposite.

[0066] The prediction box here comes from the result obtained from the previous frame prediction.

[0067] Step 3: In the case of short occlusion, use the least squares method to predict the target trajectory.

[0068] When tracking a target with occlusion, algorithms such as DeepSORT and SORT use Kalman filters to predict the trajectory. When the target is frequently occluded, the actual measurement value is small, and the Kalman gain calculation is inaccurate, resulting in a large deviation between the Kalman filter state estimate and the actual value, a large deviation in trajectory prediction, and failure to associate the detection frame when the tracking target is occluded, and the identity is switched. The present invention introduces a least squares algorithm to fit the target motion trajectory for frequent occlusion, and can effectively predict the target motion trajectory using a small number of video frame information.

[0069] The target motion trajectory can be regarded as linear motion in the short term, and the least squares method finds the best function matching the data by minimizing the square of the error.

[0070] During the target tracking process, the center point coordinates (x i ,y i ), when the target is short-occluded, the number of center point coordinates is the number of motion trajectory information n.

[0071] Construct the least squares error function as shown in formula (1).

[0072]

[0073] Where: S is the fitting error; n is the number of motion trajectory information; x i ,y i is the trajectory information, i.e. the coordinates of the center point of the tracking frame; a, b are the slope and intercept of the trajectory;

[0074]

[0075]

[0076] Formula (2) and formula (4) for S 2 Find the partial derivative so that the partial derivative value is 0, and you can calculate S 2 The minimum value of the extreme point. The trajectory information of the target x i ,y i It is known that the problem is transformed into solving a and b, that is, the trajectory.

[0077] Considering a as a constant, taking the partial derivative of b and applying it to formula (3) yields formula (4).

[0078]

[0079] The known center point coordinates and trajectory information are converted into the solution of the motion trajectory through the least squares algorithm. When the target occlusion ends and the prediction box matches the detection box, they are associated through the target matching strategy in the previous step.

[0080] Step 4: In the case of long occlusion, use Kalman filtering to predict the target trajectory.

[0081] For long-occluded tracking targets, there are many real measurement values, and the Kalman gain calculation is more accurate. It is more appropriate to use the Kalman filter for prediction. Assume that there are multiple random variables in the dynamic system, and the variables conform to the Gaussian distribution. Each variable has a mean μ and a variance σ 2 :

[0082] 1) The covariance matrix measures the correlation between random variables; the values ​​of the matrix are the degree of correlation between the variables.

[0083] 2) At each moment there is an optimal estimate for all variables and the covariance matrix P.

[0084] 3) There is uncertainty in the dynamic system. At each moment there is process noise w and measurement noise v.

[0085]

[0086]

[0087]

[0088] Where: is the estimated value; k is the time unit; K is the Kalman gain; z is the measured value;

[0089] Assume the displacement between frames as a linear uniform velocity model. Get the horizontal and vertical coordinates u and v of the center of the tracking target, s is the area of ​​the bounding box, and r is the aspect ratio of the bounding box; introduce the change information corresponding to the state, that is, Describe the state of motion.

[0090] Through the definition of the estimated value, the estimated value at time k in formula (5) can be converted from the measured value at time k and the estimated value at time k-1, as shown in formula (6). Formula (7) Defined as the Kalman gain K.

[0091] When there are multiple random variables, the solution of K is done using data fusion.

[0092]

[0093] Where: x, y, z are assumed random variables; is the variance of the random variable; σ x σ y ,σ x σ z ,σ y σ zis the covariance between random variables; P is the covariance matrix;

[0094] The random variables at time unit k are correlated. The covariance matrix P is formed by the assumed random variables and the corresponding variances and covariances. The values ​​in the P matrix represent the degree of linkage between the variables.

[0095] When the tracking target is detected, the covariance of x and x is initialized, and the state prediction is performed using formula (9);

[0096]

[0097] Where: X k is the system prediction information at time k; z k is the measurement information at time k; w is the process noise; v is the measurement noise; A, B, H are the parameter matrices of the motion equation;

[0098] The Kalman filter is to find the optimal Kalman gain, that is, the error, under the influence of noise. The minimum trace of the covariance matrix of .

[0099] K=P k-1 H T (HP k-1 H T +R) -1 (10)

[0100]

[0101] P k =P k-1 -KHP k-1 (12)

[0102] Where: R is the noise matrix.

[0103] After calculating the Kalman gain K, the tracking target state information is updated using formula (11) and formula (12). After the target is long occluded, the association strategy of the present invention is also used for association.

[0104] Based on the above method, the present invention also provides a method and device for stable tracking of an occluded target based on trajectory prediction, such as Figure 4 As shown, the device includes a single-frame image target detection module, a detection frame matching module, an occlusion judgment module, a short occlusion prediction module, and a long occlusion prediction module.

[0105] Single-frame image target detection module, used for feature extraction and target detection of single-frame images;

[0106] The detection frame matching module is used to match the detection frames according to the target detection results of multiple frames of images, and determine the target occlusion when the detection frames are mismatched;

[0107] The occlusion judgment module is used to judge the target occlusion time. If it is a short occlusion, the short occlusion prediction module is notified; if it is a long occlusion, the long occlusion prediction module is notified;

[0108] The short occlusion prediction module is used to predict the target trajectory using the least squares method under short occlusion conditions;

[0109] The long occlusion prediction module is used to predict the target trajectory using Kalman filtering in the case of long occlusion.

[0110] Among them, the single-frame image target detection module includes a dual-path attention feature extraction model and a target detection module; the dual-path attention feature extraction model includes a self-attention feature extraction channel, a recurrent attention feature extraction channel and a feature encoding module;

[0111] The self-attention feature extraction channel includes a first convolution module and a first attention module processing; the input image is used as a positive sample, and is processed by the first convolution module and the first attention module in sequence to obtain a first feature;

[0112] The cyclic attention feature extraction channel includes a cyclic shift module, a second convolution module, an addition module, a second attention module, a third convolution module, and a fusion module; the cyclic shift module performs cyclic shift on the input image to construct a negative sample; each negative sample is processed by the second convolution module and then added and fused in the addition module; then, after being processed by the second attention module and the third convolution module, a second feature is obtained; the fusion module adds the first feature and the second feature to obtain a third feature;

[0113] The feature encoding module concatenates the first feature and the third feature, performs feature encoding, and completes feature extraction;

[0114] The target detection module predicts the target position and target identity based on the extracted features output by the feature encoding module, and completes the target detection of a single frame image.

[0115] Among them, Figure 2 and Figure 3 As shown, the first attention module and the second attention module have the same structure, both including a first convolutional layer, a second convolutional layer, a third convolutional layer, a first dot product module and a second dot product module; after the input data is processed by the first convolutional layer, the second convolutional layer and the third convolutional layer respectively, data Q, data K and data V are output; after data Q and data K are multiplied in the first dot product module, attention weight information is output; the attention weight information is multiplied with data V in the second dot product module, and feature information is output.

[0116] In a preferred embodiment, a new association strategy is configured in the detection frame matching module. When the detection frame is mismatched, the mismatched detection frame is further associated with the high-value detection frame and the prediction frame, respectively, and the cosine distance between the mismatched detection frame and the high-value detection frame and the prediction frame is calculated. If the minimum cosine distance is less than a set threshold, the corresponding high-value detection frame or prediction frame is considered to be the most appropriate association item, and the mismatched detection frame is switched to a matching state, and it is considered that no occlusion occurs; the high-value detection frame is a detection frame in which a target appears once and a detection frame that exceeds a maximum lifespan; a high-value detection frame is used only once, and when a high-value detection frame is determined to be an association item, it is deleted from the high-value detection frame set.

[0117] The above specific embodiments only describe the design principle of the present invention. The shapes and names of the components in the description may be different and are not limited. Therefore, those skilled in the art in the field of the present invention may modify or replace the technical solutions recorded in the above embodiments; and these modifications and replacements do not deviate from the creative purpose and technical solutions of the present invention and should all fall within the protection scope of the present invention.

Claims

1. A stable tracking method for occluded targets based on trajectory prediction, characterized in that: include: Step 1: Perform feature extraction and target detection on a single frame image; Step 2: Match the detection frames according to the target detection results of multiple frames of images. When the detection frames are mismatched, determine the target occlusion; Determine the target occlusion time. If it is a short occlusion, execute step 3. If it is a long occlusion, go to step 4; Step 3: In the case of short occlusion, the least square method is used to predict the target trajectory; Step 4: In the case of long occlusion, use Kalman filtering to predict the target trajectory.

2. The method according to claim 1, characterized in that In step 2, when the detection frame is mismatched, the mismatched detection frame is associated with the high-value detection frame and the prediction frame respectively, and the cosine distance between the mismatched detection frame and the high-value detection frame and the prediction frame is calculated. If the minimum cosine distance is less than the set threshold, the corresponding high-value detection frame or prediction frame is considered to be the most appropriate association item, and the mismatched detection frame is switched to a matching state, and it is considered that no occlusion occurs; The high-value detection frame is a detection frame in which a target appears once.

3. The method according to claim 2, characterized in that The high-value detection box is used only once. When the high-value detection box is determined as an associated item, it is deleted from the high-value detection box set.

4. The method according to claim 2 or 3, characterized in that The high-value detection frames further include detection frames that exceed a maximum lifespan.

5. The method according to claim 1, characterized in that In step 1, when extracting features from a single frame image, a dual-path attention feature extraction model is used; The dual-path attention feature extraction model includes a self-attention feature extraction channel, a recurrent attention feature extraction channel, and a feature encoding module; In the self-attention feature extraction channel, the input image is used as a positive sample and processed by the first convolution module and the first attention module in sequence to obtain the first feature; In the cyclic attention feature extraction channel, the input image is cyclically shifted to obtain multiple negative samples, each negative sample is added and fused after being processed by the second convolution module, and then processed by the second attention module and the third convolution module to obtain a second feature; the second feature and the first feature obtained from the self-attention feature extraction channel are added and fused into a third feature; After the first feature and the third feature are concatenated, feature encoding is performed to complete feature extraction.

6. The method according to claim 5, characterized in that The first attention module and the second attention module have the same structure, both including a first convolutional layer, a second convolutional layer, a third convolutional layer, a first dot product module and a second dot product module; after the input data is processed by the first convolutional layer, the second convolutional layer and the third convolutional layer respectively, data Q, data K and data V are output; after data Q and data K are multiplied in the first dot product module, attention weight information is output; the attention weight information is multiplied with data V in the second dot product module to output feature information.

7. A stable tracking device for an occluded target based on trajectory prediction, characterized in that: It includes single-frame image target detection module, detection frame matching module, occlusion judgment module, short occlusion prediction module, and long occlusion prediction module; The single-frame image target detection module is used to perform feature extraction and target detection on a single-frame image; The detection frame matching module is used to match the detection frames according to the target detection results of multiple frames of images, and determine the target occlusion when the detection frames are mismatched; The occlusion judgment module is used to judge the target occlusion time, and if it is a short occlusion, notify the short occlusion prediction module; If it is a long occlusion, notify the long occlusion prediction module; The short occlusion prediction module is used to predict the target trajectory using the least squares method under short occlusion conditions; The long occlusion prediction module is used to predict the target trajectory using Kalman filtering in the case of long occlusion.

8. The device according to claim 7, characterized in that The single-frame image target detection module includes a dual-path attention feature extraction model and a target detection module; the dual-path attention feature extraction model includes a self-attention feature extraction channel, a cyclic attention feature extraction channel and a feature encoding module; The self-attention feature extraction channel includes a first convolution module and a first attention module processing; the input image is used as a positive sample, and is processed by the first convolution module and the first attention module in sequence to obtain a first feature; The cyclic attention feature extraction channel includes a fusion module, and a cyclic attention module composed of a cyclic shift module, a second convolution module, an addition module, a second attention module, and a third convolution module; the cyclic shift module cyclically shifts the input image to construct a negative sample; each negative sample is processed by the second convolution module and then added and fused in the addition module; and then processed by the second attention module and the third convolution module to obtain a second feature; The fusion module adds the first feature and the second feature to obtain a third feature; The feature encoding module concatenates the first feature and the third feature, performs feature encoding, and completes feature extraction; The target detection module predicts the target position and target identity based on the extracted features output by the feature encoding module, and completes the target detection of a single frame image.

9. The device according to claim 8, characterized in that The first attention module and the second attention module have the same structure, both including a first convolutional layer, a second convolutional layer, a third convolutional layer, a first dot product module and a second dot product module; after the input data is processed by the first convolutional layer, the second convolutional layer and the third convolutional layer respectively, data Q, data K and data V are output; after data Q and data K are multiplied in the first dot product module, attention weight information is output; the attention weight information is multiplied with data V in the second dot product module to output feature information.

10. The device according to claim 7, characterized in that When the detection frame is mismatched, the detection frame matching module further associates the mismatched detection frame with the high-value detection frame and the prediction frame, respectively, and calculates the cosine distance between the mismatched detection frame and the high-value detection frame and the prediction frame. If the minimum cosine distance is less than a set threshold, the corresponding high-value detection frame or prediction frame is considered to be the most appropriate association item, and the mismatched detection frame is switched to a matching state, and it is considered that no occlusion occurs; The high-value detection frame is a detection frame in which a target appears once and a detection frame that exceeds a maximum lifespan; a high-value detection frame is used only once, and when a high-value detection frame is determined as an associated item, it is deleted from the high-value detection frame set.

Images