An attention-based multi-target tracking method with anti-occlusion
By adopting attention-based anti-occlusion method in multi-objective tracking technology, the target features are processed and the target features are enhanced, the problem of low feature distinction in occlusion scenarios is solved, and the accuracy and stability of multi-objective tracking are improved.
Patent Information
- Application Number
- CN202310878608.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-07-17
- Publication Date
- 2024-11-26
- Estimated Expiration
- 2043-07-17
AI Technical Summary
In multi-objective tracking technology, existing methods are difficult to accurately predict target locations in target interaction occlusion scenarios, resulting in reduced feature distinction and impact on tracking performance. In long-term tracking, the target may lose its identity, resulting in tracking loss.
Adopting attention-based anti-occlusion multi-objective tracking method, the target features are extracted through the backbone network, and divided into independent tracking targets and non-independent tracking target features according to the occlusion situation, and using multi-scale dual attention network and timing information aggregation network for Feature processing enhances tracking target features and weakens occlusion target features, and improves feature distinction and motion prediction accuracy.
It improves the feature distinction and robustness in occlusion scenarios, enhances the identification and tracking accuracy of multi-target tracking, reduces target identity changes, and improves the stability and accuracy of tracking.
Smart Images

Figure CN117252908B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of multi-target tracking, and in particular to an attention-based anti-occlusion multi-target tracking method. Background Art
[0002] Multi-target tracking has high application value and has been widely used in many fields, including intelligent video surveillance, autonomous driving, robot navigation, tactical reconnaissance, intelligence collection, etc. Therefore, multi-target tracking has great value and plays an important role in many application fields.
[0003] However, when applying multi-target tracking technology to real-world scenarios, it still faces many challenges due to the complexity and variability of real-world scenarios. Since the imaging conditions of video acquisition equipment are complex and change in real time, factors such as occlusion, deformation, and illumination changes will cause the appearance of the target to change; when multiple targets move in the same area, there will be intersections and overlaps between targets, and mutual occlusion between the background and the target, resulting in changes in the appearance of the target. The changes in the target appearance features caused by these problems will reduce the distinguishability of the appearance features, thereby increasing the difficulty of distinguishing, identifying, and tracking targets. In addition, during long-term tracking, the above problems will cause the target to temporarily leave the field of view. Target movement, camera movement, and other reasons will also cause the target to disappear from the field of view for a long time. When the target reappears in the field of view, it will lose its previous identity, resulting in identity conversion and tracking loss.
[0004] Appearance features are feature vectors that contain target information. When tracking, the same target in the previous and next frames has similar appearance features. In real scenes, the interactive occlusion between targets in the field of view will introduce irrelevant features into the target's appearance features. The features of the tracked target include the features of the occluded target and are affected by irrelevant information, resulting in a decrease in feature differentiation, which in turn affects the results of subsequent similarity measurement and data association, and reduces the accuracy of tracking.
[0005] In multi-target tracking, the target's motion can be approximated as uniform linear motion in a short period of time, and the motion model is used to update and predict the state of the object. However, in the scenario of target interaction occlusion, because the model cannot be updated in time or the information used for updating contains occlusion information, the motion model will be disturbed and the target position cannot be accurately predicted, thus affecting the tracking performance.
[0006] In recent years, the research on multi-target tracking technology has made great progress and achievements. With the development of deep learning and the improvement of hardware computing power, deep learning-based methods have become a common method in multi-target tracking research. Common multi-target tracking frameworks include detection-based tracking frameworks, joint detection and tracking frameworks, and other frameworks.
[0007] A detection-based tracking framework in the prior art includes: performing target detection in target interactive occlusion scenarios and target long-term disappearance scenarios, obtaining the target's position and bounding box information through detection, and then extracting the target's feature information. First, the overall image features are extracted, and then combined with the position information obtained by target detection, the features are further processed to extract the features at the corresponding position.
[0008] The disadvantages of the detection-based tracking framework in the above-mentioned prior art include: in complex scenes where targets are interactively occluded, only extracting features at the target position will be interfered by the occlusion, the extracted features contain irrelevant information of the occluded objects, and the attention is not focused on the tracking target of interest, which reduces the feature discrimination, thereby affecting subsequent data association and reducing the accuracy of multi-target tracking.
[0009] The motion prediction model in multi-target tracking in the prior art includes: based on the assumption of linear motion, the target motion is modeled according to information such as the target's speed, position and acceleration in consecutive video frames. The model predicts the position of the target in the next frame image, and the prediction result is associated and matched with the detection result to form a tracking trajectory.
[0010] The disadvantages of the motion prediction model in multi-target tracking in the above-mentioned prior art include: the motion prediction model will be disturbed in the target interaction occlusion scenario, and the model cannot be updated in time or the update information has low differentiation, resulting in the trajectory motion model being offset, thereby affecting the accuracy of the predicted target position in the next frame, and further affecting the tracking performance. Summary of the invention
[0011] The embodiment of the present invention provides an attention-based anti-occlusion multi-target tracking method to achieve effective multi-target tracking.
[0012] In order to achieve the above object, the present invention adopts the following technical scheme.
[0013] An attention-based anti-occlusion multi-target tracking method, comprising:
[0014] Step S10: input the t-th frame image and the detection result of the t-1-th frame image output by the detection network into the backbone network, and the backbone network obtains the features of the detection target in the t-th frame image;
[0015] Step S20: dividing the features of the detected target in the t-th frame image into independent tracking target features and dependent tracking target features according to the occlusion situation, and dividing the dependent tracking target features into tracking target features and occlusion target features;
[0016] Step S30: performing temporal information aggregation on the tracking target feature and enhancing it through a tracking attention mechanism, performing temporal information aggregation on the occlusion target feature and weakening it through an occlusion attention mechanism, so as to obtain a processed non-independent tracking target feature;
[0017] Step S40: inputting the independent tracking target features and the processed dependent tracking target features into a detection network, and the detection network outputs the predicted position of the target in the tth frame image;
[0018] Step S50: performing data association on the predicted positions of the detection target output by the detection network at the t-1th frame image and the tth frame image to obtain the tracking trajectory of the detection target.
[0019] Preferably, the step S20 of dividing the features of the detected target in the t-th frame image into independent tracking target features and dependent tracking target features according to the occlusion situation includes:
[0020] The target position frame in frame t-1 When updating the trajectory segment of frame t-1, it is determined as the tracking trajectory, corresponding to the tracked object T in frame t. t i , determine each tracked object T according to the intersection ratio between the target position boxes t i The corresponding obstruction Calculate the target position box for each tracked object The intersection over union (IoU) with other target position boxes:
[0021]
[0022] in express and The overlapping area, express and The area occupied by the whole;
[0023] For each tracking target T t i , sort it with the intersection and union ratio of other tracking targets, if the maximum intersection and union ratio is higher than the occlusion threshold, then the target T is judged t i For non-independent tracking targets, the detection box corresponding to the highest intersection-union ratio is selected as the occluded target of this target If the maximum intersection-over-union ratio is lower than the occlusion threshold, the target T is judged to be t i To track the target independently.
[0024] Preferably, the step S20 of dividing the dependent tracking target features into tracking target features and occluding target features includes:
[0025] The non-independent tracking target features are input into a multi-scale dual attention network. After encoding the non-independent tracking target features, the multi-scale dual attention network is divided into tracking target features and occlusion target features. The multi-scale dual attention network consists of an upper branch and a lower branch of two different scales. The input of the lower branch is the features of the detection target in the t-th frame image extracted by the backbone network, and the input of the upper branch is the features of upsampling the features of the detection target in the t-th frame image extracted by the backbone network. The features output by the upper branch and the features output by the lower branch are compressed in channel dimension using maximum pooling and average pooling. The features output by the upper branch and the features output by the lower branch are compressed in channel dimension using 1×1 convolution and 3×3 convolution respectively, so that the features are concentrated in a single channel. The compressed feature values are normalized using the Sigmoid function to obtain spatial attention features of two scales, namely, tracking target features and occlusion target features.
[0026] Preferably, the step S30 aggregating the time series information of the tracking target features and enhancing them through the tracking attention mechanism, aggregating the time series information of the occluded target features and weakening them through the occlusion attention mechanism, to obtain the processed non-independent tracking target features, includes:
[0027] The tracking target features are aggregated through a temporal information aggregation network, and then enhanced through a tracking attention mechanism. The occluded target features are aggregated through a temporal information aggregation network, and then weakened through an occlusion attention mechanism. The temporal information aggregation network is composed of a reset gate, an update gate, and a state update network. The input tracking target features and occluded target features obtain gating information through a reset gate and an update gate. The candidate hidden layer state and the final hidden state are obtained using the gating information. The temporal information aggregation network aggregates the current frame features and historical features with certain weights to achieve selective memory of historical information, and finally outputs the aggregated features of the trajectory after the current frame is updated.
[0028] Preferably, the step of aggregating the time series information of the tracking target features through a time series information aggregation network and aggregating the time series information of the occluded target features through a time series information aggregation network includes:
[0029] The tracking target features and occluded target features of the current t frame Features aggregated with the trajectory in the previous frame Concatenate by channel as preparation variables for generating reset gates and update gates:
[0030]
[0031] Where Cat(·) represents channel-by-channel concatenation. The prepared variables are passed through a convolutional layer and the results are mapped between 0 and 1 through a sigmoid activation function to obtain the gating signals of the reset gate and the update gate respectively:
[0032] r t =σ{Conv r (h1)} (27)
[0033] z t =σ{Conv z (h1)} (28)
[0034] Where σ(·) represents the sigmoid activation function, Conv d (·) represents the convolutional layer, d∈{r,z}, where r and z represent the reset gate and update gate respectively. The reset gate is used to control the extent to which the aggregated information of the trajectory in the previous frame works in this frame, and is aggregated with the features extracted in this frame to obtain the hidden features of this frame:
[0035]
[0036] h t '=tanh{Conv h (h2)} (30)
[0037] Where ⊙ represents the dot product, and the update gate is used to control the degree to which the hidden features and historical features of this frame affect the aggregated features of this frame:
[0038]
[0039] Update gate z t It is a vector ranging from 0 to 1, which aggregates the features of the current frame and the historical features with a certain weight to achieve selective memory of historical information and output the aggregated features of the trajectory after the current frame is updated.
[0040] Preferably, the enhancement by tracking attention mechanism refers to using the time-aware aggregate information extracted from the non-independent tracking target in the past frame to enhance T t i The feature of the occlusion attention mechanism is to use the time-aware aggregation information extracted from the past frame of the target that occludes the non-independent tracking target to suppress the target T t i The specific operation is as follows: t i Enhanced fusion of multi-scale dual attention features and time-aware aggregation features:
[0041]
[0042] in The target T t i The time-aware aggregation features formed by the corresponding positions of the trajectories in the past frames, The target T t i Features extracted by the multi-scale dual attention network. h(·), f(·), ρ(·), and θ(·) represent the four convolutional layers in the enhanced network, which are used to calculate the correlation between features.
[0043] Target T t i Instead of blocking the target The features are used in the same way to suppress network aggregation:
[0044]
[0045] in yes The time-aware aggregated features formed by the corresponding positions of the trajectories in the past frames are used to calculate the target T according to the tracking target enhancement and occluded target suppression. t i Overall attention-based features:
[0046] Attention(T t i )=Enhance(T t i )-Reduce(T t i ) (twenty three)
[0047]
[0048] Where w is the adaptive weight for adjusting the attention feature, and its specific meaning is as follows:
[0049]
[0050] Where IoU(·) represents the intersection over union ratio between two detection boxes. Respectively represent the tracking target T t i and its occluded target Position box, o min Represents the minimum intersection-over-union ratio of hyperparameters. When T t i and The intersection-over-combination ratio between the position boxes exceeds the threshold o minWhen , the weight is a non-zero value, and the more the intersection-over-union ratio exceeds the threshold, the greater the weight, that is, the greater the degree of occlusion of the non-independent tracking target, the higher the proportion of attention features; when T t i and The intersection-over-combination ratio between the position boxes is less than the threshold o min , the weight is reset to zero.
[0051] Preferably, the detection network uses the Faster R-CNN structure, which consists of four parts: convolutional layer, region candidate network, region of interest pooling, and classifier.
[0052] It can be seen from the technical solutions provided by the above-mentioned embodiments of the present invention that the present invention improves the distinguishability of appearance features based on the attention mechanism and enhances the robustness of the algorithm in occluded scenarios; performs similarity calculation and data association of targets between frames based on cross-frame correlation, thereby improving the stability of maintaining the same target ID and reducing identity changes in scenarios where the target disappears for a long time, thereby improving the recognition and tracking accuracy of multi-target tracking and maintaining the stability of tracking.
[0053] Additional aspects and advantages of the present invention will be given in part in the following description, which will become obvious from the following description, or may be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS
[0054] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative work.
[0055] Figure 1 A processing flow chart of an attention-based anti-occlusion multi-target tracking method provided by an embodiment of the present invention;
[0056] Figure 2 A schematic diagram of an implementation principle of an attention-based anti-occlusion multi-target tracking method provided in an embodiment of the present invention;
[0057] Figure 3 A schematic diagram of the structure of a multi-scale dual attention network provided by an embodiment of the present invention;
[0058] Figure 4 A schematic diagram of the structure of a channel attention module network provided by an embodiment of the present invention;
[0059] Figure 5 A structural diagram of a timing information aggregation network provided in an embodiment of the present invention. DETAILED DESCRIPTION
[0060] The embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and cannot be interpreted as limiting the present invention.
[0061] It will be understood by those skilled in the art that, unless expressly stated, the singular forms "one", "said", and "the" used herein may also include plural forms. It should be further understood that the term "comprising" used in the specification of the present invention refers to the presence of the features, integers, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or groups thereof. It should be understood that when we refer to an element as being "connected" or "coupled" to another element, it may be directly connected or coupled to the other element, or there may be intermediate elements. In addition, the "connection" or "coupling" used herein may include wireless connection or coupling. The term "and / or" used herein includes any unit and all combinations of one or more associated listed items.
[0062] It will be understood by those skilled in the art that, unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as those generally understood by those skilled in the art in the art to which the present invention belongs. It should also be understood that terms such as those defined in common dictionaries should be understood to have meanings consistent with the meanings in the context of the prior art, and will not be interpreted with idealized or overly formal meanings unless defined as herein.
[0063] To facilitate understanding of the embodiments of the present invention, several specific embodiments will be further explained below with reference to the accompanying drawings, and each embodiment does not constitute a limitation on the embodiments of the present invention.
[0064] The embodiment of the present invention provides an attention-based anti-occlusion multi-target tracking method. The method starts from the perspective of feature extraction and motion prediction, establishes attention mechanisms for tracking targets and occluding targets respectively, and enhances or weakens the extracted features in a targeted manner according to the current object of attention, thereby enhancing the features of the tracking target and suppressing the features of the occluding target, thereby improving the effectiveness of the features; adopts a time aggregation mechanism to aggregate the past information of the target with the current frame information, thereby effectively utilizing the historical information in the trajectory; and further highlights the area of the target of interest through a multi-scale dual attention module, thereby enhancing the distinguishability of the features. Using the features extracted by the network for motion prediction is conducive to more accurately predicting the position of the target in the next frame in the case of interactive occlusion between targets, thereby improving the ability to predict targets and thus improving the accuracy of tracking.
[0065] The processing flow chart of an attention-based anti-occlusion multi-target tracking method provided by an embodiment of the present invention is as follows: Figure 1 As shown, the process includes the following steps:
[0066] Step S10: input the t-th frame image and the detection result of the t-1-th frame image output by the detection network into the backbone network, and the backbone network obtains the features of the detection target in the t-th frame image;
[0067] Step S20: dividing the features of the detected target in the t-th frame image into independent tracking target features and dependent tracking target features according to the occlusion situation, and dividing the dependent tracking target features into tracking target features and occluded target features.
[0068] Step S30: performing temporal information aggregation on the tracking target feature and enhancing it through a tracking attention mechanism, performing temporal information aggregation on the occlusion target feature and weakening it through an occlusion attention mechanism, so as to obtain a processed non-independent tracking target feature;
[0069] Step S40: inputting the independent tracking target features and the processed dependent tracking target features into a detection network, and the detection network outputs the predicted position of the target in the tth frame image;
[0070] Step S50: performing data association on the predicted positions of the detection target output by the detection network at the t-1th frame image and the tth frame image to obtain the tracking trajectory of the detection target.
[0071] Specifically, the above step S10 includes:
[0072] The implementation principle of the attention-based anti-occlusion multi-target tracking method provided by the embodiment of the present invention is as follows: Figure 2 As shown, Figure 1 Middle D t-1 , D t Respectively represent the detection results of the t-1th frame and the tth frame, B t Denotes the target position in the t-th frame predicted based on the target position in the t-1th frame and the features of the t-th frame. The t-th frame image and the detection result of the t-1th frame image output by the detection network are input into the backbone network, the backbone network obtains the features of the detected target in the t-th frame image, and the detection network outputs the position box of each frame. The detection result D in the t-1th frame t-1 Input to the upper network, after multi-scale dual attention network and tracking attention and occlusion attention mechanism, the position B of the target in the tth frame is finally predicted t ;D t With B t After data association, the tracking trajectory is obtained.
[0073] First, use the target position in the previous frame to predict the target position in the current frame. Filter the position boxes of the previous frame that exceed the edge range of the current frame image and are too small in area, and determine the targets of the remaining position boxes as tracking targets and occlusion targets according to the occlusion situation. Extract the features of the target based on the tracking attention and occlusion attention mechanisms, and then input the features into the Fast R-CNN network to predict the category, confidence score and position box offset of each tracked target in the t-th frame image, filter out the position boxes with confidence scores lower than the confidence threshold, and use non-maximum suppression to further screen the position boxes, delete the position boxes whose boundaries exceed the edge of the image and whose area is too small, and correct the position box of the t-1th frame image input by the network according to the position box offset to obtain the predicted position box B in the t-th frame image. t .
[0074] Figure 3 A schematic diagram of the structure of a multi-scale dual attention network provided for an embodiment of the present invention. In the case of interactive occlusion between targets, different positions of the extracted features may belong to different targets, reducing the discriminability of the features. In order to extract more discriminative features, increase the importance of tracking targets and suppress the importance of occluded targets, the present invention proposes a multi-scale dual attention network to enhance the robustness of features to occlusion.
[0075] Figure 4 A schematic diagram of the structure of a channel attention module network provided by an embodiment of the present invention, Figure 4 The dashed line in the middle represents 1 minus the output value of the channel attention module, and ⊙ represents the dot product. The overall network consists of two branches of different scales. The input of the lower branch is the feature F extracted by the backbone network. in , the input of the upper branch is F in Upsampled features. Since the feature size within the detection box is limited, the features are further scaled during the forward propagation of the network. In order to alleviate the spatial information loss generated in this process, the input features are upsampled:
[0076]
[0077]
[0078] Where Upsample(·) indicates upsampling using bilinear interpolation. Then, pooling is used to compress the channel dimension of the feature. Average pooling calculates the average value of feature points in the neighborhood, which helps to solve the problem of increased variance of the estimated value caused by limited neighborhood size; maximum pooling takes the maximum value of feature points in the neighborhood, which helps to solve the problem of estimated mean shift. Therefore, in this network, both average pooling and maximum pooling are used to compress the number of feature channels:
[0079]
[0080]
[0081]
[0082]
[0083] Where MaxPool(·) and AvgPool(·) represent the maximum pooling and average pooling operations respectively. The features output by the two pooling layers are concatenated in the channel dimension:
[0084]
[0085]
[0086] Among them, Cat(·) represents concatenation along the channel dimension. In order to adapt to features of different sizes, 1×1 convolution and 3×3 convolution are used to compress the features of the upper branch and the lower branch respectively, so that the features are concentrated in a single channel, and the Sigmoid function is used to normalize the values to obtain spatial attention features of two scales:
[0087]
[0088]
[0089] Where σ(·) represents the Sigmoid function, Conv n×n Represents a convolutional layer with a convolution kernel size of n×n. By multiplying the spatial attention weights obtained by the upper and lower branches with their input feature maps, we can obtain feature maps that emphasize the key spatial areas and highlight the representative features of the tracked target at different scales:
[0090]
[0091]
[0092] Where ⊙ represents the matrix dot product. Downsample the upper branch features to restore them to the size before upsampling:
[0093]
[0094] DownSample(·) represents downsampling. Because the feature map of the upper branch can supplement the spatial features of large targets, and the feature map of the lower branch can supplement the spatial features of small targets, the feature maps of the two branches are fused below. The features of the two branches are added together and the weight W is obtained through the channel attention module. channel :
[0095]
[0096] Where Channel(·) represents the channel attention module. channel and 1-W channel As weights, two spatial attention features of different scales are fused:
[0097]
[0098] Where ⊙ represents the matrix dot product. The obtained F attention The attention features output by the multi-scale dual attention network are enhanced in the spatial dimension to make the features more focused on the tracked target.
[0099] Figure 4 ⊙ represents dot product. The network input is the sum of the spatial attention features of two scales, which is divided into two branches: global branch and local branch. The global branch performs global average pooling on the input to compress its space to one pixel as the channel weight, and then uses the bottleneck network structure to perform point-by-point convolution on the features, which can extract the local information of the features while keeping the amount of calculation small. The process is as follows:
[0100]
[0101] Among them, Conv point (·) represents point-by-point convolution, Relu(·) represents activation function, and GlobalAvgPooling(·) represents global average pooling. The local branch also uses the bottleneck network structure to extract local information while keeping the size of the input feature map unchanged. It can retain and highlight the details of the underlying features. The process is as follows:
[0102]
[0103] Then add the features output by the global branch and the local branch and multiply them by the input features as weights:
[0104]
[0105] Where Sigmoid(·) represents the Sigmoid function, and the output F attention is a feature with channel attention.
[0106] Specifically, the above step S20 includes:
[0107] The tracking attention and occlusion attention mechanisms determine independent tracking targets and dependent tracking targets according to the degree of occlusion in the target interactive occlusion scenario, and then extract target features based on the attention mechanism. Since dependent tracking targets are affected by occlusion of other targets, the features output by the feature extraction network contain information about some occluders. Therefore, this method further processes the backbone network features based on the feature enhancement of the occluded target, reduces the impact of irrelevant information introduced by target occlusion on the features, and thus improves the distinguishability of the features.
[0108] First, a target is determined to be an independent tracking target or a non-independent tracking target. The target position frame in the t-1th frame When updating the trajectory segment of frame t-1, it has been determined as the tracking trajectory, corresponding to the tracked object T in frame t. t i , determine each tracked object T according to the intersection ratio between the target position boxes t i The corresponding obstruction That is, calculate the target position box of each tracked object The intersection over union (IoU) with other target location boxes:
[0109]
[0110] in express and The overlapping area, express and The area occupied by the whole. For each tracking target T t i , sort it with the intersection and union ratio of other tracking targets, if the maximum intersection and union ratio is higher than the occlusion threshold, then the target T is judged t i For non-independent tracking targets, the detection box corresponding to the highest intersection-union ratio is selected as the occluded target of this target If the maximum intersection-over-union ratio is lower than the occlusion threshold, the target T is judged to be t i The selection of independent tracking targets and non-independent tracking targets is to carry out targeted processing in the subsequent feature extraction, that is, to use the tracking attention and occlusion attention mechanisms for non-independent tracking targets to extract more accurate features, so as to reduce the impact of occlusion on the non-independent tracking targets.
[0111] Then, feature extraction is performed on independent tracking targets and dependent tracking targets respectively. For independent tracking targets, the features corresponding to the target position in the overall features output by the backbone network are extracted; for dependent tracking targets, the features corresponding to the target position in the overall features output by the backbone network are first extracted, and then the features are input into the multi-scale dual attention network to further extract its more representative features.
[0112]
[0113] Where AttentionNet(·) represents a multi-scale dual attention network, represents the location box of the target in frame t-1, F t Represents the features of the tth frame extracted by the backbone network.
[0114] The tracking attention and occlusion attention mechanisms are divided into two parts: enhancement and suppression. Enhancement refers to using the time-aware aggregate information extracted from the past frames of the non-independent tracking target to enhance T t i The feature of T is suppressed by using the time-aware aggregate information extracted from the past frames of the target that occludes the non-independent tracked target. t i The features of T can be enhanced by introducing aggregate information from historical memory. The specific operation is as follows: t i Enhanced fusion of multi-scale dual attention features and time-aware aggregation features:
[0115]
[0116] in The target T t i The time-aware aggregation features formed by the corresponding positions of the trajectories in the past frames, The target T t i Features extracted by the multi-scale dual attention network, h(·), f(·), ρ(·), θ(·) represent the four convolutional layers in the enhanced network, which are used to calculate the correlation between features. Similarly, T t i Instead of blocking the target The features are used in the same way to suppress network aggregation:
[0117]
[0118] in yes The corresponding positions of the trajectories in the past frames are aggregated to form time-aware aggregate features. and It is an aggregate feature with identity-aware memory. The process of introducing these aggregate features is equivalent to introducing the aggregated non-local attention between the historical reference information of the tracked target and the features obtained in the new frame. Then the target T is calculated based on the tracked target enhancement and occluded target suppression. t i Overall attention-based features:
[0119] Attention(T t i )=Enhance(T t i )-Reduce(T t i ) (twenty three)
[0120]
[0121] where w is the adaptive weight that adjusts the attention feature.
[0122] Adaptive weights are used to adjust the degree of enhancement and suppression in the attention mechanism. The attention mechanism using tracking target enhancement and occluded target suppression can make the extracted features pay more attention to the tracking target, reduce the feature interference from the occluded target, and collect less interfering features for subsequent data association and memory aggregation. Although the enhancement of tracking attention features and the weakening of occlusion features make the features of non-independent tracking targets more concentrated, not all targets in multi-target tracking scenarios require such a mechanism. For example, unoccluded targets or targets with little occlusion can be directly feature extracted without using the above attention mechanism. In addition, non-independent tracking targets with different occlusion conditions should have different enhancement and suppression conditions. For example, non-independent tracking targets with larger occluded areas should be set with larger tracking attention enhancement and larger occlusion attention suppression. However, using a fixed weight w, it is impossible to set different degrees of feature enhancement and suppression according to the occlusion situation, so adaptive attention weights are used for attention features:
[0123]
[0124] Where IoU(·) represents the intersection over union ratio between two detection boxes. Respectively represent the tracking target T t i and its occluded target Position box, o min represents the minimum intersection-over-union ratio of the hyperparameter. t i and The intersection-over-combination ratio between the position boxes exceeds the threshold o min When , the weight is a non-zero value, and the more the intersection-over-union ratio exceeds the threshold, the greater the weight, that is, the greater the degree of occlusion of the non-independent tracking target, the higher the proportion of attention features; when T t i and The intersection-over-combination ratio between the position boxes is less than the threshold o min When , the weight is reset to zero, and the attention feature does not participate in feature weighting. By using this adaptive weight, adaptive adjustment of the attention of the tracked object and the attention of the occluded target can be achieved.
[0125] Figure 5 A structural diagram of a temporal information aggregation network provided in an embodiment of the present invention, the temporal information aggregation network is used to perform identity-aware temporal information aggregation on the information of the target in the trajectory, so as to extract more discriminative attention features. In the above-mentioned tracking target attention mechanism, the features of non-independent tracking targets need to be aggregated with the features of their trajectories in previous frames to enhance the representativeness of the features. However, in scenarios where there is severe occlusion between targets, directly extracting the features of the previous frame as aggregated features usually contains more occlusion information and introduces noise, so the temporal information aggregation network is used to upgrade the trajectory features. Figure 5 The dashed line indicates 1-z t The obtained variables, the network consists of three parts: reset gate, update gate, and state update network. First, the current t-th frame feature Features aggregated with the trajectory in the previous frame Concatenate by channel as preparation variables for generating reset gates and update gates:
[0126]
[0127] Among them, Cat(·) represents channel concatenation. Then the prepared variable is passed through a convolution layer and the result is mapped between 0 and 1 through the sigmoid activation function to obtain the gating signals of the reset gate and the update gate respectively:
[0128] r t =σ{Conv r (h1)} (27)
[0129] z t =σ{Conv z (h1)} (28)
[0130] Where σ(·) represents the sigmoid activation function, Conv d(·) represents the convolutional layer, d∈{r,z}, where r and z represent the reset gate and update gate respectively. The reset gate is used to control the extent to which the aggregated information of the trajectory in the previous frame works in this frame, and is aggregated with the features extracted in this frame to obtain the hidden features of this frame:
[0131]
[0132] h t '=tanh{Conv h (h2)} (30)
[0133] Among them, ⊙ represents the dot product. Finally, the update gate is used to control the degree to which the hidden features and historical features of this frame contribute to the aggregated features of this frame:
[0134]
[0135] Since the update gate z t It is a vector ranging from 0 to 1, which can measure the size of the gate opening, aggregate the current frame features and historical features with a certain weight, realize the selective memory of historical information, and finally output the aggregated features of the trajectory after the current frame is updated.
[0136] Specifically, the above step S50 includes:
[0137] The predicted position B in the t-th frame image t And the test results D t To associate. Select Faster R-CNN as the detection network to obtain the detection results, calculate the IoU between all predicted positions and the detected positions, and if the IoU between a certain predicted position and the detection result is greater than the threshold, then the detection result is judged to be the position of the target track to which the predicted position belongs in the tth frame image. Next, you only need to match the remaining detection results and inactive tracks. Calculate the GIoU between the remaining predicted positions and the detection positions and filter out the associations with GIoU less than the threshold, calculate the cost matrix based on the spatial features, and use the Hungarian algorithm for matching. For the predicted positions and detection boxes that are still not matched after the Hungarian algorithm is matched, initialize the unmatched detection boxes as new tracks, and add the unmatched predicted positions to the inactive track list. Update the existing track list and the inactive track list, and delete the tracks in the inactive track list that have not been updated for a long time. The process is specifically expressed as follows:
[0138]
[0139]
[0140] In summary, the method of the present invention adds a multi-scale dual attention module to the feature extraction network, uses spatial attention and channel attention to enhance the features of the tracked target, and improves the discriminability of the features; adopts a time aggregation mechanism to aggregate the information of the same target in the past frames for predicting the target position in the next frame, and effectively utilizes past information; the above modules jointly improve the discrimination of the features, and ultimately obtain more accurate tracking results.
[0141] The tracking attention and occlusion attention proposed in the present invention process the features in a targeted manner according to the current object of attention, enhance the features of the tracking target, suppress the features of the occluded target, make the extracted features more distinctive, use the features to perform motion prediction, and then perform trajectory correlation matching, ultimately obtaining a more accurate tracking trajectory.
[0142] Those skilled in the art can understand that the accompanying drawings are only schematic diagrams of an embodiment, and the modules or processes in the accompanying drawings are not necessarily required to implement the present invention.
[0143] It can be known from the description of the above implementation methods that those skilled in the art can clearly understand that the present invention can be implemented by means of software plus a necessary general hardware platform. Based on such an understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which can be stored in a storage medium such as ROM / RAM, a magnetic disk, an optical disk, etc., and includes a number of instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in the various embodiments of the present invention or certain parts of the embodiments.
[0144] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device or system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment. The device and system embodiments described above are merely schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.
[0145] The above is only a preferred specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed by the present invention should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.
Claims
1. An attention-based anti-occlusion multi-target tracking method, characterized in that: include: Step S10: input the t-th frame image and the detection result of the t-1-th frame image output by the detection network into the backbone network, and the backbone network obtains the features of the detection target in the t-th frame image; Step S20: dividing the features of the detected target in the t-th frame image into independent tracking target features and dependent tracking target features according to the occlusion situation, and dividing the dependent tracking target features into tracking target features and occlusion target features; Step S30: performing temporal information aggregation on the tracking target feature and enhancing it through a tracking attention mechanism, performing temporal information aggregation on the occlusion target feature and weakening it through an occlusion attention mechanism, so as to obtain a processed non-independent tracking target feature; Step S40: inputting the independent tracking target features and the processed dependent tracking target features into a detection network, and the detection network outputs the predicted position of the target in the tth frame image; Step S50: performing data association on the predicted positions of the detection target output by the detection network at the t-1th frame image and the tth frame image to obtain a tracking trajectory of the detection target; The step S30 of aggregating the time series information of the tracking target features and enhancing them through the tracking attention mechanism, aggregating the time series information of the occluded target features and weakening them through the occlusion attention mechanism, and obtaining the processed non-independent tracking target features, includes: The tracking target features are aggregated through a temporal information aggregation network, and then enhanced through a tracking attention mechanism. The occluded target features are aggregated through a temporal information aggregation network, and then weakened through an occlusion attention mechanism. The temporal information aggregation network is composed of a reset gate, an update gate, and a state update network. The input tracking target features and occluded target features obtain gating information through the reset gate and the update gate. The candidate hidden layer state and the final hidden state are obtained using the gating information. The temporal information aggregation network aggregates the current frame features and historical features with a certain weight to achieve selective memory of historical information, and finally outputs the aggregated features of the trajectory after the current frame is updated; The enhancement by tracking attention mechanism refers to using the time-aware aggregate information extracted from the past frames of the non-independent tracking target to enhance T t i The feature of the occlusion attention mechanism is to use the time-aware aggregation information extracted from the past frame of the target that occludes the non-independent tracking target to suppress the target T t i characteristics.
2. The method according to claim 1, characterized in that The step S20 of dividing the features of the detected target in the t-th frame image into independent tracking target features and dependent tracking target features according to the occlusion situation includes: The target position frame in frame t-1 When updating the trajectory segment of frame t-1, it is determined as the tracking trajectory, corresponding to the tracked object in frame t. Determine each tracked object based on the intersection ratio between the target position boxes The corresponding obstruction Calculate the target position box for each tracked object The intersection over union (IoU) with other target position boxes: in express and The overlapping area, express and The area occupied by the whole; For each tracking target Sort the intersection and union ratios of it and other tracked targets. If the maximum intersection and union ratio is higher than the occlusion threshold, the target is judged as For non-independent tracking targets, the detection box corresponding to the highest intersection-union ratio is selected as the occluded target of this target If the maximum intersection-and-union ratio is lower than the occlusion threshold, the target is judged To track the target independently.
3. The method according to claim 2, characterized in that The step S20 of dividing the dependent tracking target features into tracking target features and occluding target features includes: The non-independent tracking target features are input into a multi-scale dual attention network. After encoding the non-independent tracking target features, the multi-scale dual attention network is divided into tracking target features and occlusion target features. The multi-scale dual attention network consists of an upper branch and a lower branch of two different scales. The input of the lower branch is the features of the detection target in the t-th frame image extracted by the backbone network, and the input of the upper branch is the features of upsampling the features of the detection target in the t-th frame image extracted by the backbone network. The features output by the upper branch and the features output by the lower branch are compressed in channel dimension using maximum pooling and average pooling. The features output by the upper branch and the features output by the lower branch are compressed in channel dimension using 1×1 convolution and 3×3 convolution respectively, so that the features are concentrated in a single channel. The compressed feature values are normalized using the Sigmoid function to obtain spatial attention features of two scales, namely, tracking target features and occlusion target features.
4. The method according to claim 1, characterized in that: The step of aggregating the time series information of the tracking target features through the time series information aggregation network and aggregating the time series information of the occluded target features through the time series information aggregation network includes: The tracking target features and occluded target features of the current t frame Features aggregated with the trajectory in the previous frame Concatenate by channel as preparation variables for generating reset gates and update gates: Where Cat(·) represents channel-by-channel concatenation. The prepared variables are passed through a convolutional layer and the results are mapped between 0 and 1 through a sigmoid activation function to obtain the gating signals of the reset gate and the update gate respectively: r t =σ{Conv r (h1)} (27) z t =σ{Conv z (h1)} (28) Where σ(·) represents the sigmoid activation function, Conv d (·) represents the convolutional layer, d∈{r,z}, where r and z represent the reset gate and update gate respectively. The reset gate is used to control the extent to which the aggregated information of the trajectory in the previous frame works in this frame, and is aggregated with the features extracted in this frame to obtain the hidden features of this frame: h t ′=tanh{Conv h (h2)} (30) Where ⊙ represents the dot product, and the update gate is used to control the degree to which the hidden features and historical features of this frame affect the aggregated features of this frame: Update gate z t It is a vector ranging from 0 to 1, which aggregates the features of the current frame and the historical features with a certain weight to achieve selective memory of historical information and output the aggregated features of the trajectory after the current frame is updated.
5. The method according to claim 1, characterized in that The detection network uses the Faster R-CNN structure, which consists of four parts: convolutional layer, region candidate network, region of interest pooling, and classifier.
Citation Information
Patent Citations
Target object tracking method, device and equipment, and computer readable storage medium
CN112132866A
Multi-target tracking method
CN116128932A
Target motion tracking method based on brain-inspired memory network
GB202218183D0
