Multi-target tracking method and system based on Mask twin network

The Mask Twin Network-based method addresses multi-target tracking challenges by integrating Mask R-CNN and convolutional networks, enhancing Kalman filters, and using dual attention modules for robust and accurate target tracking in complex environments.

CN119068023APending Publication Date: 2024-12-03西安中科立德红外科技有限公司
View PDF 3 Cites 0 Cited by

Patent Information

Application Number
CN202411050530.7
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-01
Publication Date
2024-12-03

AI Technical Summary

Technical Problem

Current multi-target tracking algorithms face challenges in real-world scenarios due to inaccurate target detection, occlusion, and similar appearance features, leading to poor performance and identity number swaps, especially in complex environments with background interference and high target density.

Method used

A multi-target tracking method using a Mask Twin Network that integrates Mask R-CNN and convolutional neural networks, enhances Kalman filters with unscented transformation, and employs dual attention modules for improved feature extraction and trajectory prediction, followed by Hungarian algorithm matching for real-time tracking.

Benefits of technology

Enhances target tracking robustness and accuracy by balancing texture and semantic features, improving Kalman filter nonlinearity, and ensuring consistent target identification in complex scenes.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119068023A_ABST
    Figure CN119068023A_ABST
Patent Text Reader

Abstract

The invention provides a multi-target tracking method and system based on a Mask twinning network. The multi-target tracking method comprises the following steps: inputting an image into a Mask R-CNN network to perform feature extraction on a target in the image; cutting the target feature map based on the candidate region of the target in the target feature map; inputting the feature map into a double attention module for processing; predicting a target track and a target state through an improved Kalman filter; and matching the target prediction trajectory state with the target real trajectory state through a Hungary algorithm module to obtain a target real-time tracking result. The multi-target tracking system comprises a Mask R-CNN network, a segmentation module, a double attention module, an improved Kalman filter and a Hungary algorithm module, the Mask R-CNN network is adopted to realize fusion of a Mask twinning network and a convolutional neural network architecture, nonlinearity of the Kalman filter is enhanced through unscented transformation, and the improved Kalman filter is obtained. And the robustness of the Kalman filter in a real scene is improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to a multi-target tracking method and system based on a Mask twin network. Background Art

[0002] Multi-target tracking technology refers to the technology of accurately locating and tracking multiple targets from continuous image frames while maintaining the consistency of target identity. It is a research hotspot in computer vision technology. Multi-target tracking technology is mainly divided into target detection, trajectory prediction and data association. Multi-target tracking technology is widely used in real-world scenarios. For example, autonomous driving, intelligent monitoring, vehicle flow monitoring and other aspects cannot be separated from the support of multi-target tracking technology.

[0003] With the rapid development of artificial intelligence, multi-target tracking technology based on deep learning has made great breakthroughs compared with multi-target tracking methods based on traditional detection and feature extraction. Currently, multi-target tracking algorithms based on deep learning are mainly divided into multi-target tracking algorithms based on detection and multi-target tracking algorithms based on joint detection.

[0004] The detection-based multi-target tracking algorithm first uses the detector to locate the target, uses the good target detection results as the input of the tracker, and then uses the tracker to extract the apparent features of each detected target and predict the target's inter-frame state to form a predicted trajectory. Finally, the matching algorithm is used to perform data association between the target's predicted state and the actual state to complete target tracking.

[0005] The advantages of the first multi-target tracking algorithm technical solution are: each module can be optimized separately, and the overall tracking accuracy of the multi-target tracking algorithm can be improved by improving the performance of a certain module. The disadvantages are: the degree of correlation between modules is not high, and the training process of each module is independent of each other, which will lead to the problem that the model parameters of each module can converge to the optimal value, but the optimal value of the overall algorithm is not reached.

[0006] The multi-target tracking algorithm of joint detection advocates reducing the complexity of the algorithm by directly integrating modules such as target feature prediction, appearance feature extraction, and data association matching in the tracker with the target detection network.

[0007] The advantages of the second multi-target tracking algorithm technical solution are: short inference time and relatively simple structure. The disadvantages are: different modules are incompatible when learning in the same framework, and training the detection network and the re-identification network at the same time requires a huge data set to improve the generalization ability of the model. In addition, the current joint detection multi-target tracking algorithm performs poorly in terms of temporal consistency.

[0008] In actual scenarios, due to the existence of factors such as background interference and excessive target density, the current research on multi-target tracking algorithms still faces many problems, such as inaccurate target detection and occlusion between similar targets, which leads to poor performance of multi-target tracking algorithms.

[0009] Target detection problem: The tracking performance of the multi-target tracking algorithm is highly dependent on the accuracy of target detection. However, in abnormal weather conditions, dense targets, and other conditions, due to background interference and mutual occlusion between targets, false detections and missed detections often occur during target detection, affecting the data association of the target status and resulting in poor performance of the target tracking algorithm.

[0010] Target occlusion problem: The multi-target tracking algorithm predicts the trajectory state of the target in the next frame by locating the trajectory formed by the target in consecutive image frames. When multiple targets are occluded, the historical tracking trajectory and the newly generated predicted trajectory cannot be successfully matched, resulting in the problem that the identity number changes when the target is detected again.

[0011] Similarity of appearance features: In complex scenes, there may be multiple targets with similar appearance features. When similar-looking targets are too close to each other, they cannot be associated with the correct trajectory due to similar features, and the identity numbers will be swapped. This phenomenon is particularly obvious in low-resolution image frames and long-distance observation targets. Summary of the invention

[0012] The purpose of the present invention is to provide a multi-target tracking method and system based on Mask Twin Network, adopt Mask R-CNN network to realize the fusion of Mask Twin Network and convolutional neural network architecture, enhance the nonlinearity of Kalman filter by unscented transformation to obtain an improved Kalman filter, and improve the robustness of Kalman filter in real scenes.

[0013] The technical solution for achieving the purpose of the present invention is as follows:

[0014] On the one hand, the present invention provides a multi-target tracking method based on Mask Twin Network, comprising the following steps:

[0015] The image is input into the Mask R-CNN network to extract the features of the target in the image and obtain the target feature map;

[0016] Cutting the target feature map based on the candidate region of the target in the target feature map to obtain a feature map with consistent scale;

[0017] Inputting the feature map into a dual attention module for processing to obtain a target feature with channel attention and spatial attention;

[0018] The target trajectory and target state are predicted by improving the Kalman filter to obtain the target predicted trajectory and state prediction information;

[0019] The predicted target trajectory state and the actual target trajectory state are matched through the Hungarian algorithm module to obtain the real-time target tracking result.

[0020] Based on one aspect, in one embodiment of the present invention, the image is input into a Mask R-CNN network to extract features of a target in the image to obtain a target feature map, including:

[0021] Search for the image to be identified through the Mask R-CNN network;

[0022] Use the Mask R-CNN network’s Mask Twin Network and the Mask-based Deep Twin Network to classify, regress, segment, and extract features of the image to be identified to obtain a feature map.

[0023] A dual-branch feature fusion network is used to perform feature fusion on the features extracted from the feature map to obtain a target feature map; wherein the dual-branch feature fusion network extracts features from the feature map through a bottom-up path while performing nearest neighbor upsampling on the feature map from the highest layer through a top-down path, and the dual-branch feature fusion network performs feature fusion on the feature maps generated by the bottom-up and top-down paths through lateral connections.

[0024] Based on one aspect, in one embodiment of the present invention, the Mask R-CNN network and the Mask-based deep twin network are used to classify, regress, segment and extract features of the image to be identified to obtain a feature map, including:

[0025] Mask denoising and feature extraction are performed through a neural encoder based on the Mask Twin Network;

[0026] Through random cropping enhancement, the data of the identified image is randomly enhanced to generate two different views x 1 and x 2 ;

[0027] According to the mask design principle, two different views x 1 and x 2 Perform mask operations separately;

[0028] For two different views x 1 and x 2 80% random grid masking and 20% focal masking in the spatial dimension,

[0029] The spatial masked view is color-dithered and gray-scaled, three independent masks are randomly generated, and 70% independent channel masking operations are performed on the three RGB channels in the channel dimension. Next, Gaussian blur is applied to the two views.

[0030] The two processed views are respectively input into the encoders of their respective branches for feature extraction to obtain feature pyramids of the two views, where the encoder architectures of the two branches are asymmetric.

[0031] Based on one aspect, in one embodiment of the present invention, the target feature map is cut based on the candidate region of the target in the target feature map to obtain a feature map with consistent scale, including:

[0032] Generate a sliding window in the region candidate network to perform convolution on the target feature map, generate an anchor box with an aspect ratio of (1:1, 1:2, 2:1) for each pixel on the target feature map, then offset the output coordinates to all anchor boxes, sort them according to the scores, and finally filter out the candidate regions through non-maximum suppression;

[0033] Each candidate area is traversed and divided into H*W units. Four standard positions are selected in each unit. The standard positions are calculated and adjusted by bilinear interpolation. Then, the maximum pooling operation is used for aggregation. Finally, regression, classification and segmentation are performed through a fully convolutional network to obtain a feature map with consistent scale.

[0034] Based on one aspect, in one embodiment of the present invention, the feature map is input into a dual attention module for processing to obtain a target feature with channel attention and spatial attention, including:

[0035] Construct channel attention module and spatial attention module;

[0036] In the process of channel attention weighting by the channel attention module, the scale of the feature map is changed by the reshape function, the feature map is transposed and weighted summed with the deformed features, a channel attention map is obtained by using the softmax function, and then the channel attention map is restored to the original scale by the reshape function, and the channel attention map retains the information between the channels in the feature map;

[0037] In the spatial attention weighting process, the spatial attention module uses maximum pooling and average pooling to highlight the key features of the feature map, uses a softmax function to obtain a spatial attention map, and restores the spatial attention map to its original scale through convolution;

[0038] A target feature with channel attention and spatial attention is obtained through the channel attention map and the spatial attention map.

[0039] Based on one aspect, in one embodiment of the present invention, the target trajectory and target state are predicted by an improved Kalman filter to obtain the target predicted trajectory and state prediction information, including: taking the trajectory and state of the target in the previous frame as the input of the improved Kalman filter, predicting the target trajectory and target state by the improved Kalman filter, and obtaining the target predicted trajectory and state prediction information of the current frame; wherein:

[0040] The improved Kalman filter introduces unscented transformation on the basis of Kalman filtering, and then calculates the estimated value after unscented transformation by least square method, obtains the new mean and covariance matrix of the observed quantity by weighted summation, and performs state update and covariance update of the system;

[0041] A plurality of improved Kalman filters are used to predict the same target, and the weights of each improved Kalman filter are adjusted by adaptive weights.

[0042] Based on one aspect, in one embodiment of the present invention, the target predicted trajectory state and the target actual trajectory state are matched by the Hungarian algorithm module to obtain the target real-time tracking result, including:

[0043] Detect the current frame image to obtain the target's true state and target's true trajectory;

[0044] According to the target predicted trajectory and the state prediction information of the previous frame image by the improved Kalman filter, and the target real state and the target real trajectory of the current frame image, the internal cost matrix in the Hungarian algorithm module is updated to complete the solution of the minimum weight matching problem, realize the data association between the predicted result and the detection result, that is, the matching is successful, and the real-time tracking result of the target is obtained.

[0045] On the other hand, the present invention also provides a multi-target tracking system based on a Mask twin network. The multi-target tracking system is based on the above-mentioned multi-target tracking method based on a Mask twin network. The multi-target tracking system includes:

[0046] A Mask R-CNN network, wherein the Mask R-CNN network is input with an image, and the Mask R-CNN network extracts features of a target in the image to obtain a target feature map;

[0047] A segmentation module, wherein the segmentation module segments the target feature map based on a candidate region of the target in the target feature map to obtain a feature map with a consistent scale;

[0048] A dual attention module, wherein the dual attention module processes the input feature map to obtain a target feature with channel attention and spatial attention;

[0049] An improved Kalman filter is used to predict a target trajectory and a target state to obtain target prediction trajectory and state prediction information;

[0050] The Hungarian algorithm module matches the target predicted trajectory state with the target real trajectory state to obtain the target real-time tracking result.

[0051] Based on another aspect, in one embodiment of the present invention, the Mask R-CNN network searches for an image to be identified;

[0052] The Mask R-CNN network uses the Mask twin network and the Mask-based deep twin network in the Mask R-CNN network to classify, regress, segment and extract features of the image to be identified to obtain a feature map;

[0053] The Mask R-CNN network uses a dual-branch feature fusion network to perform feature fusion on the features extracted from the feature map to obtain a target feature map.

[0054] Based on another aspect, in one embodiment of the present invention, the dual attention module includes a channel attention module and a spatial attention module;

[0055] In the process of channel attention weighting, the channel attention module changes the scale of the feature map through the reshape function, transposes the feature map and weights the transformed feature, obtains the channel attention map using the softmax function, and then restores the channel attention map to its original scale through the reshape function, and the channel attention map retains the information between the channels in the feature map;

[0056] In the spatial attention weighting process, the spatial attention module uses maximum pooling and average pooling to highlight the key features of the feature map, uses a softmax function to obtain a spatial attention map, and restores the spatial attention map to its original scale through convolution.

[0057] Compared with the prior art, the present invention has the following beneficial effects:

[0058] 1. The present invention adopts the Mask R-CNN network to realize the fusion of the Mask Twin network and the convolutional neural network architecture; specifically, the instance segmentation method is used to enhance the description of the target position and morphology in the image, and the Mask Twin network is added before the encoder of the convolutional neural network to perform random grid masking and focus masking in the spatial dimension and random masking in the channel dimension on the image. The combination of random grid masking and focus masking balances the texture features and semantic features in the image.

[0059] 2. The present invention improves the nonlinearity of the Kalman filter by untraceable transformation to obtain an improved Kalman filter, thereby improving the robustness of the Kalman filter in real scenes; and further improving the tracking performance of the multi-target tracking method.

[0060] 3. The target feature map of the present invention adopts a feature pyramid network structure to improve the accuracy of target tracking. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] Figure 1 A flowchart of a multi-target tracking method based on Mask twin network provided by the present invention;

[0062] Figure 2 A line of sight schematic diagram of a multi-target tracking method based on Mask twin network provided by the present invention;

[0063] Figure 3 The structure diagram of the Mask R-CNN network provided by the present invention;

[0064] Figure 4 A feature enhancement network structure diagram based on feature pyramid provided by the present invention;

[0065] Figure 5 A schematic diagram of the network structure of the dual attention module provided by the present invention;

[0066] Figure 6 An algorithm flow chart of the improved Kalman filter provided by the present invention;

[0067] Figure 7 A principle block diagram of a multi-target tracking system based on a Mask twin network provided by the present invention. DETAILED DESCRIPTION

[0068] The present invention is described in detail below in conjunction with the various embodiments shown in the accompanying drawings, but it should be noted that these embodiments are not limitations of the present invention, and any equivalent transformations or substitutions in functions, methods, or structures made by ordinary technicians in the field based on these embodiments are all within the scope of protection of the present invention.

[0069] See also Figure 1 and Figure 2The embodiment of the present invention discloses a multi-target tracking method based on a Mask twin network, comprising the following steps:

[0070] Step 1: The image is input into the Mask R-CNN network to extract features of the target in the image and obtain the target feature map.

[0071] For more details, see Figure 3 In the embodiment of the present invention, the image input Mask R-CNN network is used to extract features of the target in the image to obtain a target feature map, including: searching for the image to be identified through the Mask R-CNN network; using the Mask twin network and the Mask deep twin network in the Mask R-CNN network to classify, regress, segment and extract features of the image to be identified to obtain a feature map. Among them, mask denoising and feature extraction are performed by a neural encoder based on the Mask twin network; random data enhancement is performed on the image to be identified by random cropping enhancement to generate two different views x 1 and x 2 ; According to the mask design principle, two different views x 1 and x 2 Perform mask operations separately; for two different views x 1 and x 2 An 80% random grid mask operation and a 20% focus mask operation are performed in the spatial dimension. The view after the spatial mask is applied is color jittered and grayscale converted. Three independent masks are randomly generated, and a 70% independent channel mask operation is performed on the three RGB channels in the channel dimension. Next, Gaussian blur is applied to the two views. The two processed views are respectively input into the encoders of their respective branches for feature extraction to obtain feature pyramids of the two views, where the encoder architectures of the two branches are asymmetric.

[0072] A double-branch feature fusion network is used to fuse the features extracted from the feature map to obtain the target feature map; wherein, the double-branch feature fusion network extracts features from the feature map through a bottom-up path, and performs nearest neighbor upsampling on the feature map from the highest layer through a top-down path. The double-branch feature fusion network fuses the feature maps generated by the bottom-up and top-down paths through lateral connections.

[0073] The embodiment of the present invention uses an instance segmentation method to enhance the description of the target position and morphology, adds a mask twin network before the ResNet50 neural network encoder, performs random grid masking and focus masking in the spatial dimension and random masking in the channel dimension on the two views, and uses a feature pyramid enhancement module and a feature fusion module in the branch feature fusion part to fuse the features extracted by the two branches.

[0074] Step 1 In the implementation step in actual application, two different views x are 1 and x 2 Perform mask operations separately. The specific mask design principles are as follows:

[0075]

[0076] Where M represents the mask, z represents the mask area filling value, f represents the important feature, and g represents the secondary feature.

[0077] In addition, in the embodiment of the present invention, when two different views x are viewed 1 and x 2 Before performing 80% random grid masking and 20% focus masking in the spatial dimension, the image is processed with a high-pass filter before applying the mask to eliminate the parasitic edges generated by the random grid mask and overcome the convolution kernel collapse problem after the convolution kernel performs edge detection.

[0078] For two different views x 1 and x 2 An 80% random grid mask operation and a 20% focus mask operation are performed in the spatial dimension, where the combination of the random grid mask and the focus mask in both views is random, balancing the texture features and semantic features in the input image.

[0079] See also Figure 4 In the first step of the embodiment of the present invention, after obtaining the feature pyramids of the two views, the feature pyramids are fused through a dual-branch feature fusion network. Specifically, the feature pyramids of the two views are extracted as the input of the feature pyramid enhancement modules of their respective branches, and the features are upsampled through iterations with step sizes of 64, 32, 16, 8, and 4 to generate an upwardly enhanced feature pyramid, and then the generated feature pyramid is downsampled through iterations with step sizes of 4, 8, 16, 32, and 64 to obtain a downwardly enhanced feature pyramid; the feature pyramids after the two branch enhancements are used as the input of the feature fusion module, and the feature maps of the same size in the feature pyramids are added to obtain a feature pyramid after feature fusion.

[0080] Step 2: Cut the target feature map based on the candidate area of ​​the target in the target feature map to obtain a feature map with consistent scale.

[0081] More specifically, a sliding window is generated in the region candidate network to perform convolution on the target feature map. An anchor box is generated for each pixel on the corresponding target feature map according to the aspect ratio of (1:1, 1:2, 2:1), and then the output coordinates are offset to all anchor boxes, sorted according to the scores, and finally the candidate regions are screened out by non-maximum suppression; each candidate region is traversed, the candidate region is divided into H*W units, four standard positions are selected in each unit, and the standard positions are calculated and adjusted by bilinear interpolation method, and then the maximum pooling operation is used for aggregation, and finally regression, classification and segmentation are performed through the full convolutional network to obtain a feature map with consistent scale.

[0082] In practical applications, the candidate region acquisition, feature classification, regression and mask prediction of the embodiment of the present invention are specifically as follows: the feature pyramid after feature fusion is used as the input of the region candidate network, a basic anchor frame is generated for the feature map of each level of the feature pyramid, and a candidate frame is generated according to the scale of the basic anchor frame using a ratio of (1:1, 1:2, 2:1), and the final candidate region is obtained by non-maximum suppression screening; all candidate regions are evenly divided into H*W sub-regions, four regular sampling points are taken in each sub-region, and the pixel values ​​of the four adjacent sampling points are calculated using bilinear interpolation. The maximum pooling is performed on each sub-region to complete the regional feature aggregation, and the final feature pyramid is obtained; the feature maps of small sizes in the feature pyramid are upsampled to make the sizes of each feature map consistent, and the full convolution network is used for image segmentation, wherein the loss function is:

[0083] L=L cls +L box +L mask (1)

[0084] Among them, L cls represents the loss function of the classification branch, L box represents the loss function of the regression branch, L mask represents the loss function of the segmentation mask branch;

[0085] The classification loss function is:

[0086] L cls = -log[c gt +(1-C gt )(1-c)] (2)

[0087] Among them, c represents the probability of correct prediction classification, c gt Indicates the proportion of the correct classification results to the total number;

[0088] The regression loss function is:

[0089]

[0090] Among them, ρ represents the Euclidean distance between the center position of the predicted box and the center position of the real box, box represents the center position of the predicted box, and box gt represents the center position of the real box, and c represents the diagonal distance of the minimum area containing the predicted box and the real box;

[0091] The segmentation mask uses the average binary cross entropy loss, and the loss function is:

[0092]

[0093] Among them, m*m represents the size of the feature map, M ij Represents the true mask of the (i,j) coordinate position, Represents the predicted mask at the (i,j) coordinate position.

[0094] Step 3: Input the feature map into the dual attention module for processing to obtain the target features with channel attention and spatial attention.

[0095] See also Figure 5 , step three of the embodiment of the present invention includes: constructing a channel attention module and a spatial attention module; in the process of channel attention weighting, the scale of the feature map is changed by the reshape function through the channel attention module, the feature map is transposed and weighted with the deformed features, the channel attention map is obtained by the softmax function, and the channel attention map is restored to the original scale by the reshape function, and the channel attention map retains the information between the channels in the feature map; in the process of spatial attention weighting, the spatial attention module uses maximum pooling and average pooling to highlight the key features of the feature map, the spatial attention map is obtained by the softmax function, and the spatial attention map is restored to the original scale by convolution; the target features with channel attention and spatial attention are obtained through the channel attention map and the spatial attention map.

[0096] Step 3 of the embodiment of the present invention improves feature distinguishability and highlights key features through a dual attention module. Specifically, the tracking features extracted by the twin neural network are used as the input of the attention module, and the output is a feature with channel attention and spatial attention weighting. Please continue to refer to Figure 5 The specific processing process includes:

[0097] In the channel attention weighting part, the input feature map (C*H*W) is reshaped into (C*N) dimensions and input into three branches. The feature map of branch one is transposed and matrix multiplied with the feature map of branch two. The softmax operation is applied to the obtained feature map to obtain the channel-weighted attention map (C*C). The attention map is transposed and matrix multiplied with the reshaped feature map. Then, the reshape function is used to transform the attention map to the size of the input feature map and perform weighted summation with the input feature map to obtain a feature map with channel attention weighting. In the process of channel weighting, no convolution operation is used to retain the relationship between the original channels, thereby improving the distinguishability of features.

[0098] In the spatial attention weighting part, the input feature map is subjected to maximum pooling and average pooling, and the pooled results are weighted and summed. The softmax operation is applied to the obtained feature map to obtain the spatially weighted attention map, which is then convolved to restore the input feature map size and weighted and summed with the input feature map to obtain a feature map with spatial attention weighting. This method enhances the feature representation capability of the model;

[0099] The feature maps output by the two attention branches are element-wise summed to complete feature fusion, and a feature map with channel attention and spatial attention weights is obtained.

[0100] Step 4: Predict the target trajectory and target state by improving the Kalman filter to obtain the target predicted trajectory and state prediction information.

[0101] More specifically, see Figure 6 When the target trajectory and target state are predicted by the improved Kalman filter, the trajectory and state of the target in the previous frame are used as the input of the improved Kalman filter, and the target trajectory and target state are predicted by the improved Kalman filter to obtain the target prediction trajectory and state prediction information of the current frame; wherein: the improved Kalman filter introduces untraceable transformation on the basis of Kalman filtering, and then the estimated value after untraceable transformation is calculated by the least square method, and the new mean and covariance matrix of the observation quantity are obtained by weighted summation, and the state and covariance update of the system are performed;

[0102] Multiple improved Kalman filters are used to predict the same target, and the weights of each improved Kalman filter are adjusted through adaptive weights.

[0103] Please continue reading Figure 6The embodiment of the present invention improves the Kalman filter to perform an untraceable transformation on the linear system in the Kalman filter algorithm, obtains sampling points and weights of the prior distribution, performs nonlinear function transfer based on the state equation, predicts the mean and covariance of the target trajectory state, uses the second untraceable transformation to obtain new sampling points, obtains the observed value based on the observation equation, calculates the predicted mean and covariance, and finally calculates the Kalman gain to update the target trajectory state;

[0104] Three improved Kalman filters are used to update the trajectory state prediction of the same target, and the weights of each filter are adjusted using the adaptive weight method to enhance the robustness of the filter in real scenarios;

[0105] The Kalman filter state equation is:

[0106] x i+1 =f(x i )+w i (5)

[0107] Among them, x i represents the system state matrix at time i, f() represents the nonlinear state transfer function, w i represents the process noise matrix at time i;

[0108] The Kalman filter observation equation is:

[0109] z i =h(x i )+v i (6)

[0110] Among them, z i represents the system observation matrix at time i, h() represents the nonlinear observation function, w i represents the observation noise matrix at time i.

[0111] Step 5: Match the target predicted trajectory state and the target real trajectory state through the Hungarian algorithm module to obtain the target real-time tracking result. The Hungarian algorithm of the embodiment of the present invention belongs to a matching algorithm. The target predicted trajectory state is the same as the target predicted trajectory and state prediction information obtained in the previous step. Whether the target is successfully tracked is determined based on the matching degree between the calculated predicted trajectory state and the real trajectory state. If the match is successful, the KF filter is updated to continue tracking. If the match fails, the trajectory and ID are deleted.

[0112] More specifically, the target predicted trajectory state and the target real trajectory state are matched through the Hungarian algorithm module. The current frame image is first detected to obtain the target real state and the target real trajectory. Then, the target predicted trajectory and state prediction information of the previous frame image are updated according to the improved Kalman filter, and the target real state and the target real trajectory of the current frame image are used to update the internal cost matrix in the Hungarian algorithm module to complete the solution of the minimum weight matching problem, realize the data association between the prediction result and the detection result, that is, the matching is successful, and the real-time tracking result of the target is obtained.

[0113] See also Figure 7 , the embodiment of the present invention also provides a multi-target tracking system based on Mask Twin Network, the multi-target tracking system is based on the above-mentioned multi-target tracking method based on Mask Twin Network, and the multi-target tracking system includes Mask R-CNN network, segmentation module, dual attention module, improved Kalman filter and Hungarian algorithm module. Among them, the Mask R-CNN network inputs an image, and the Mask R-CNN network extracts features of the target in the image to obtain a target feature map; the segmentation module cuts the target feature map based on the candidate area of ​​the target in the target feature map to obtain a feature map with consistent scale; the dual attention module processes the input feature map to obtain target features with channel attention and spatial attention; the improved Kalman filter predicts the target trajectory and target state to obtain target predicted trajectory and state prediction information; the Hungarian algorithm module matches the target predicted trajectory state with the target real trajectory state to obtain the target real-time tracking result.

[0114] The above-mentioned Mask R-CNN network of the embodiment of the present invention searches for the image to be identified; the Mask R-CNN network uses the Mask twin network and the Mask-based deep twin network in the Mask R-CNN network to classify, regress, segment and extract features of the image to be identified to obtain a feature map; the Mask R-CNN network uses a dual-branch feature fusion network to perform feature fusion on the features extracted from the feature map to obtain a target feature map.

[0115] The above-mentioned dual attention module of the embodiment of the present invention includes a channel attention module and a spatial attention module; in the process of channel attention weighting, the channel attention module changes the scale of the feature map through the reshape function, transposes the feature map and weighted sums it with the deformed features, uses the softmax function to obtain the channel attention map, and then restores the channel attention map to the original scale through the reshape function, and the channel attention map retains the information between the channels in the feature map; in the process of spatial attention weighting, the spatial attention module of the embodiment of the present invention uses maximum pooling and average pooling to highlight the key features of the feature map, uses the softmax function to obtain the spatial attention map, and restores the spatial attention map to the original scale through convolution.

[0116] The embodiment of the present invention adopts the Mask R-CNN network to realize the fusion of the Mask Twin network and the convolutional neural network architecture; specifically, the instance segmentation method is used to enhance the description of the target position and morphology in the image, and the Mask Twin network is added before the encoder of the convolutional neural network to perform random grid masking and focus masking in the spatial dimension and random masking in the channel dimension on the image. The combination of random grid masking and focus masking balances the texture features and semantic features in the image.

[0117] The series of detailed descriptions listed above are only specific descriptions of feasible implementation methods of the present invention. They are not intended to limit the scope of protection of the present invention. Any equivalent implementation methods or changes that do not deviate from the technical spirit of the present invention should be included in the scope of protection of the present invention.

[0118] It will be apparent to those skilled in the art that the invention is not limited to the details of the exemplary embodiments described above and that the invention can be implemented in other specific forms without departing from the spirit or essential features of the invention. Therefore, the embodiments should be considered exemplary and non-limiting in all respects, and the scope of the invention is defined by the appended claims rather than the foregoing description, and it is intended that all variations falling within the meaning and scope of the equivalent elements of the claims be included in the invention. Any reference numeral in a claim should not be considered as limiting the claim to which it relates.

[0119] In addition, it should be understood that although the present specification is described according to implementation modes, not every implementation mode contains only one independent technical solution. This description of the specification is only for the sake of clarity. Those skilled in the art should regard the specification as a whole. The technical solutions in each embodiment may also be appropriately combined to form other implementation modes that can be understood by those skilled in the art.

Claims

1. A multi-target tracking method based on Mask Twin Network, characterized in that: The following steps are involved: The image is input into the Mask R-CNN network to extract the features of the target in the image and obtain the target feature map; Cutting the target feature map based on the candidate region of the target in the target feature map to obtain a feature map with consistent scale; Inputting the feature map into a dual attention module for processing to obtain a target feature with channel attention and spatial attention; The target trajectory and target state are predicted by improving the Kalman filter to obtain the target predicted trajectory and state prediction information; The predicted target trajectory state and the actual target trajectory state are matched through the Hungarian algorithm module to obtain the real-time target tracking result.

2. According to claim 1, a multi-target tracking method based on Mask Twin Network is characterized in that: The image is input into the Mask R-CNN network to extract features of the target in the image and obtain a target feature map, including: Search for images to be identified through the Mask R-CNN network; Use the Mask R-CNN network’s Mask Twin Network and the Mask-based Deep Twin Network to classify, regress, segment, and extract features of the image to be identified to obtain a feature map. A dual-branch feature fusion network is used to perform feature fusion on the features extracted from the feature map to obtain a target feature map; wherein the dual-branch feature fusion network extracts features from the feature map through a bottom-up path while performing nearest neighbor upsampling on the feature map from the highest layer through a top-down path, and the dual-branch feature fusion network performs feature fusion on the feature maps generated by the bottom-up and top-down paths through lateral connections.

3. According to claim 2, a multi-target tracking method based on Mask Twin Network is characterized in that: The Mask R-CNN network uses the Mask twin network and the Mask-based deep twin network to classify, regress, segment and extract features of the image to be identified to obtain a feature map, including: Mask denoising and feature extraction are performed through a neural encoder based on the Mask Twin Network; The image to be identified is randomly enhanced by random cropping enhancement to generate two different views x1 and x2; According to the mask design principle, mask operations are performed on two different views x1 and x2 respectively; For two different views x1 and x2, 80% random grid masking and 20% focus masking are performed in the spatial dimension. The spatial masked view is color-dithered and gray-scaled, three independent masks are randomly generated, and 70% independent channel masking operations are performed on the three RGB channels in the channel dimension. Next, Gaussian blur is applied to the two views. The two processed views are respectively input into the encoders of their respective branches for feature extraction to obtain feature pyramids of the two views, where the encoder architectures of the two branches are asymmetric.

4. According to claim 1, a multi-target tracking method based on Mask Twin Network is characterized in that: The step of cutting the target feature map based on the candidate region of the target in the target feature map to obtain a feature map with a consistent scale includes: Generate a sliding window in the region candidate network to perform convolution on the target feature map, generate an anchor box with an aspect ratio of (1:1, 1:2, 2:1) for each pixel on the target feature map, then offset the output coordinates to all anchor boxes, sort them according to the scores, and finally filter out the candidate regions through non-maximum suppression; Each candidate area is traversed and divided into H*W units. Four standard positions are selected in each unit. The standard positions are calculated and adjusted by bilinear interpolation. Then, the maximum pooling operation is used for aggregation. Finally, regression, classification and segmentation are performed through a fully convolutional network to obtain a feature map with consistent scale.

5. According to a multi-target tracking method based on Mask Twin Network according to claim 1, it is characterized in that: The feature map is input into the dual attention module for processing to obtain target features with channel attention and spatial attention, including: Construct channel attention module and spatial attention module; In the process of channel attention weighting by the channel attention module, the scale of the feature map is changed by the reshape function, the feature map is transposed and weighted summed with the deformed features, a channel attention map is obtained by using the softmax function, and then the channel attention map is restored to the original scale by the reshape function, and the channel attention map retains the information between the channels in the feature map; In the spatial attention weighting process, the spatial attention module uses maximum pooling and average pooling to highlight the key features of the feature map, uses a softmax function to obtain a spatial attention map, and restores the spatial attention map to its original scale through convolution; A target feature with channel attention and spatial attention is obtained through the channel attention map and the spatial attention map.

6. According to claim 1, a multi-target tracking method based on Mask Twin Network is characterized in that: The method of predicting the target trajectory and target state by using the improved Kalman filter to obtain the target predicted trajectory and state prediction information includes: taking the trajectory and state of the target in the previous frame as the input of the improved Kalman filter, predicting the target trajectory and target state by using the improved Kalman filter, and obtaining the target predicted trajectory and state prediction information of the current frame; wherein: The improved Kalman filter introduces unscented transformation on the basis of Kalman filtering, and then calculates the estimated value after unscented transformation by least square method, obtains the new mean and covariance matrix of the observed quantity by weighted summation, and performs state update and covariance update of the system; A plurality of improved Kalman filters are used to predict the same target, and the weights of each of the improved Kalman filters are adjusted by adaptive weights.

7. The multi-target tracking method based on Mask Twin Network according to claim 1, characterized in that: The target predicted trajectory state and the target real trajectory state are matched by the Hungarian algorithm module to obtain the target real-time tracking result, including: Detect the current frame image to obtain the target's true state and target's true trajectory; According to the target predicted trajectory and the state prediction information of the previous frame image by the improved Kalman filter, and the target real state and the target real trajectory of the current frame image, the internal cost matrix in the Hungarian algorithm module is updated to complete the solution of the minimum weight matching problem, realize the data association between the predicted result and the detection result, that is, the matching is successful, and the real-time tracking result of the target is obtained.

8. A multi-target tracking system based on a Mask twin network, based on a multi-target tracking method based on a Mask twin network according to any one of claims 1 to 7, characterized in that: include: A Mask R-CNN network, wherein the Mask R-CNN network is input with an image, and the Mask R-CNN network extracts features of a target in the image to obtain a target feature map; A segmentation module, wherein the segmentation module segments the target feature map based on the candidate region of the target in the target feature map to obtain a feature map with a consistent scale; A dual attention module, wherein the dual attention module processes the input feature map to obtain a target feature with channel attention and spatial attention; An improved Kalman filter is used to predict a target trajectory and a target state to obtain target prediction trajectory and state prediction information; The Hungarian algorithm module matches the target predicted trajectory state with the target real trajectory state to obtain the target real-time tracking result.

9. The multi-target tracking system based on Mask Twin Network according to claim 8, characterized in that: The Mask R-CNN network searches for an image to be identified; The Mask R-CNN network uses the Mask twin network and the Mask-based deep twin network in the Mask R-CNN network to classify, regress, segment and extract features of the image to be identified to obtain a feature map; The Mask R-CNN network uses a dual-branch feature fusion network to perform feature fusion on the features extracted from the feature map to obtain a target feature map.

10. The multi-target tracking system based on Mask Twin Network according to claim 8, characterized in that: The dual attention module includes a channel attention module and a spatial attention module; In the process of channel attention weighting, the channel attention module changes the scale of the feature map through the reshape function, transposes the feature map and weights the transformed feature, obtains the channel attention map using the softmax function, and then restores the channel attention map to its original scale through the reshape function, and the channel attention map retains the information between the channels in the feature map; In the spatial attention weighting process, the spatial attention module uses maximum pooling and average pooling to highlight the key features of the feature map, uses a softmax function to obtain a spatial attention map, and restores the spatial attention map to its original scale through convolution.

Citation Information

Patent Citations

  • Multi-target tracking method based on Mask R-CNN and apparent feature fusion

    CN113506317A

  • Segmentation and tracking method based on attention and feature fusion

    CN116596966A

  • Video processing method and device

    WO2023128866A2