Multi-target tracking method and system based on optimal transmission algorithm and attention mechanism

Abstract

The invention relates to the technical field of multi-target tracking, and provides a multi-target tracking method and system based on an optimal transmission algorithm and an attention mechanism, and the method comprises the steps: integrating an attention mechanism module into a feature extraction network, carrying out the weighted fusion of generated features, and achieving the selective emphasis of useful features and the inhibition of useless features; and an optimal transmission algorithm is introduced into the triple loss function, so that the calculation of similarity between targets is enhanced, and the similarity and difference between samples are more accurately measured, thereby improving the accuracy of feature recognition. The matching accuracy is improved by adopting two times of data association matching features and tracks, so that a more accurate track of the tracked target is obtained.

Description

Technical Field

[0001] The present disclosure relates to the technical field related to multi-target tracking, and specifically, to a multi-target tracking method and system based on an optimal transmission algorithm and an attention mechanism. Background Art

[0002] The statements in this section merely provide background information related to the present disclosure and do not necessarily constitute prior art.

[0003] Multi-target tracking (MOT) is an important research direction in the field of computer vision. With the development of artificial intelligence and deep learning technology, the application of multi-target tracking technology in autonomous driving, intelligent monitoring, behavior recognition, smart cities and other fields has become a topic of focus for many researchers. Early classic methods include Meanshift and particle filtering, but the overall accuracy is low, and they are mainly single-target tracking. In recent years, with the improvement of target detection performance, detection-based tracking schemes have quickly become the mainstream framework of current MOT methods. Existing MOT methods based on detection and tracking, these methods first use detectors to detect targets in each frame of the video sequence, crop the targets according to the bounding box, and obtain all the targets in the image. Then, it is transformed into a target association problem between the previous and next frames, and the similarity matrix is ​​constructed through IoU, appearance, etc., and solved by methods such as the Hungarian algorithm and the greedy algorithm.

[0004] The inventors found in their research that the accuracy of the existing MOT method based on detection and tracking depends on the effect of appearance feature extraction and the similarity matrix constructed between the previous and next frames. When the appearance feature extraction effect is not good or the similarity matrix is ​​inaccurate, the tracking effect will be poor. In a multi-target scene, there may be occlusion between the targets, which will reduce the visible part of the target and affect the appearance feature extraction. The appearance of the target may also change due to factors such as changes in lighting and perspective. These appearance changes will cause the accuracy of the similarity measurement matrix to decrease, thereby affecting the tracking accuracy. In summary, the existing MOT method must ensure the accuracy and robustness of video tracking to adapt to its application in intelligent monitoring, autonomous driving and other fields. However, due to the poor effect of appearance feature extraction and similarity matrix, the accuracy and robustness of video tracking are poor. Summary of the invention

[0005] In order to solve the above problems, the present invention proposes a multi-target tracking method and system based on an optimal transmission algorithm and an attention mechanism, which enhances the feature extraction capability, improves the accuracy of measuring the similarity between samples, and improves the tracking effect in complex situations such as target occlusion, appearance change and motion blur, thereby improving the accuracy and robustness of tracking.

[0006] In order to achieve the above objectives, the present disclosure adopts the following technical solutions:

[0007] One or more embodiments provide a multi-target tracking method based on an optimal transmission algorithm and an attention mechanism, comprising the following steps:

[0008] Obtain the video sequence to be tracked, perform target detection to obtain video frames containing detection frames, perform preprocessing to obtain target images, and divide them into video frames with high-confidence detection frames and video frames with low-confidence detection frames according to the detection frame scores;

[0009] The video frames with high confidence detection frames and the preprocessed video frames are subjected to appearance feature extraction through the trained appearance feature extraction model; the appearance feature extraction model uses an improved triplet loss function for network optimization, and adds the optimal transmission Si nkhorn algorithm to the original triplet loss function to adjust the distance matrix;

[0010] Obtain the trajectory of the video frame to be tracked to predict the motion information, obtain the position information, and then obtain the predicted trajectory;

[0011] The extracted appearance features are extracted and the predicted trajectory is subjected to the first data association; the video frame with the low confidence detection box is subjected to the second data association with the trajectory that is not matched by the first data association, and the trajectory of the tracked target is matched.

[0012] One or more embodiments provide a multi-target tracking system based on an optimal transmission algorithm and an attention mechanism, including:

[0013] Preprocessing module: configured to obtain a video sequence to be tracked, perform target detection to obtain a video frame containing a detection frame, perform preprocessing to obtain a target image, and divide the image into a video frame with a high-confidence detection frame and a video frame with a low-confidence detection frame according to the detection frame score;

[0014] Feature extraction module: configured to extract appearance features from video frames with high confidence detection frames and preprocessed video frames through the trained appearance feature extraction model; the appearance feature extraction model uses an improved triplet loss function for network optimization, and adds the optimal transmission Si nkhorn algorithm to the original triplet loss function to adjust the distance matrix;

[0015] Trajectory prediction module: configured to obtain the trajectory of the video frame to be tracked to predict the motion information, obtain the position information, and then obtain the predicted trajectory;

[0016] Matching association module: It is configured to perform a first data association between the extracted appearance feature extraction and the predicted trajectory; perform a second data association between the video frame with the low confidence detection box and the trajectory that is not matched by the first data association, and match the trajectory of the tracked target.

[0017] An electronic device includes a memory and a processor, and computer instructions stored in the memory and running on the processor. When the computer instructions are run by the processor, the steps in the multi-target tracking method based on the optimal transmission algorithm and attention mechanism are completed.

[0018] A computer-readable storage medium is used to store computer instructions. When the computer instructions are executed by a processor, the steps in the multi-target tracking method based on the optimal transmission algorithm and the attention mechanism are completed.

[0019] Compared with the prior art, the present invention has the following beneficial effects:

[0020] In the present invention, the attention mechanism module is integrated into the feature extraction network, and the generated features are weightedly fused to achieve selective emphasis on useful features and suppression of useless features; and the optimal transmission algorithm is introduced into the triplet loss function to enhance the calculation of similarities between targets, ensuring more accurate measurement of similarities and differences between samples, thereby improving the accuracy of feature recognition.

[0021] The advantages of the present disclosure and the advantages of additional aspects will be described in detail in the following specific embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] The accompanying drawings constituting a part of the present disclosure are used to provide a further understanding of the present disclosure. The exemplary embodiments of the present disclosure and the description thereof are used to explain the present disclosure but do not constitute a limitation of the present disclosure.

[0023] Figure 1 is a flowchart of a multi-target tracking method based on an optimal transmission algorithm and an attention mechanism according to Embodiment 1 of the present disclosure;

[0024] Figure 2 It is a schematic diagram of the structure of the spatial attention mechanism and channel attention mechanism modules of Embodiment 1 of the present disclosure;

[0025] Figure 3 Schematic diagram of the calculation process of the improved triplet loss function of Example 1 of the present disclosure. DETAILED DESCRIPTION

[0026] The present disclosure is further described below in conjunction with the accompanying drawings and embodiments.

[0027] It should be noted that the following detailed descriptions are exemplary and are intended to provide further explanation of the present disclosure. Unless otherwise specified, all technical and scientific terms used herein have the same meanings as those commonly understood by those skilled in the art to which the present disclosure belongs.

[0028] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit exemplary embodiments according to the present disclosure. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that when the terms "comprising" and / or "including" are used in this specification, it indicates the presence of features, steps, operations, devices, components and / or combinations thereof. It should be noted that, in the absence of conflict, the various embodiments in the present disclosure and the features in the embodiments can be combined with each other. The embodiments will be described in detail below in conjunction with the accompanying drawings.

[0029] Example 1

[0030] In the technical solutions disclosed in one or more embodiments, Figures 1 to 3 As shown in FIG. 1 , a multi-target tracking method based on an optimal transmission algorithm and an attention mechanism includes the following steps:

[0031] Step 1: Obtain the video sequence to be tracked, perform target detection to obtain video frames containing detection frames, perform preprocessing to obtain target images, and divide them into video frames with high-confidence detection frames and video frames with low-confidence detection frames according to the detection frame scores;

[0032] Step 2: extracting appearance features from the video frames with high confidence detection frames and the preprocessed video frames using the trained appearance feature extraction model;

[0033] The appearance feature extraction model adopts an improved triplet loss function for network optimization, wherein the improved triplet loss function adds an optimal transmission S i nkhorn algorithm to the original triplet loss function to adjust the distance matrix;

[0034] Step 3: Obtain the trajectory of the video frame to be tracked to predict the motion information, obtain the position information, and then obtain the predicted trajectory;

[0035] Step 4: Perform the first data association between the extracted appearance features and the predicted trajectory; perform the second data association between the video frame with the low confidence detection box and the trajectory that is not matched by the first data association, and match the trajectory of the tracked target;

[0036] A further technical solution, in step 2, the appearance feature extraction model includes a newly added attention mechanism module and an improved triplet loss function, the newly added attention mechanism module adds a space and channel attention mechanism module after the last convolutional layer of the backbone network to enhance the ability to capture distinguishing features between different targets with similar appearances, and obtain a feature map; the improved triplet loss function adds an optimal transmission Si nkhorn algorithm to the original triplet loss function to adjust the distance matrix;

[0037] Furthermore, the improved triple loss function calculates the distance matrix between samples through Euclidean distance or cosine distance, constructs the positive and negative sample indicator matrix according to the label information of the samples, uses the weighted instance mining method to calculate the weighted positive and negative sample distances, and uses the optimal transmission Sinkhorn algorithm to determine the sample weights to obtain the positive sample distance and the negative sample distance;

[0038] In this embodiment, by integrating the attention mechanism module into the feature extraction network, the generated features are weighted and fused to achieve selective emphasis on useful features and suppression of useless features; and the optimal transmission algorithm is introduced into the triple loss function to enhance the calculation of similarity between targets, ensuring more accurate measurement of similarities and differences between samples, thereby improving the accuracy of feature recognition. By using two data association matching features and trajectories, the matching accuracy is improved, thereby obtaining a more accurate trajectory of the tracked target.

[0039] In step 1, the acquired video frames are detected and preprocessed, and for the video sequence to be tracked, the targets in the video are extracted for detection, and an image with a detection frame for each tracked object in the video is obtained;

[0040] Optionally, the preprocessing includes: identifying each tracking object in the video frame to be tracked, isolating each tracking object from the original video frame and separating it from other tracking objects, and obtaining a separate image corresponding to each tracking object;

[0041] The video frames obtained by detection and the preprocessed images are input into the appearance feature extraction model. The feature extraction model can include a ResNeSt50 backbone network and a spatial and channel attention mechanism module and an improved loss function. The spatial and channel attention mechanism module is as follows: Figure 2 As shown, the improved loss function is Figure 3 shown.

[0042] In step 1, for the obtained target image, the detection frame is divided into a high-confidence detection frame and a low-confidence detection frame, specifically including:

[0043] Step 11: obtain the detection frame through target detection and obtain the confidence of the corresponding detection frame;

[0044] Step 12: Using a threshold judgment method, divide the video frames according to the detection frame scores;

[0045] Specifically, according to the confidence level, a fixed threshold is set to divide the detection frame into a high-confidence detection frame and a low-confidence detection frame, and different confidence detection frames are processed in different ways;

[0046] In step 2, the appearance feature extraction model includes a backbone network, an aggregation module, a detection head, and a loss function unit;

[0047] Backbone network: extracts features from the input image and increases the weight of the features based on the attention mechanism, and outputs feature maps;

[0048] Aggregation module: Aggregate the features of the backbone network and aggregate the features generated by the backbone network into a global feature;

[0049] Detection head: performs target detection based on aggregated features and outputs a target image with a detection frame;

[0050] Loss function unit: used to calculate the loss function during the training process of the appearance feature extraction model;

[0051] The newly added attention mechanism module in this embodiment adds a spatial attention mechanism module and a channel attention mechanism module after the last convolutional layer of the backbone network to enhance the ability to capture distinguishing features between different targets with similar appearances and obtain a feature map;

[0052] The spatial attention mechanism module and the channel attention mechanism module selectively emphasize the features with rich information and suppress the features with weak information by modeling the interdependence between the channel dimension and the spatial dimension of the feature map. Figure 2 As shown, the spatial and channel attention mechanism module includes two modules: channel attention mechanism and spatial attention mechanism;

[0053] In the spatial attention mechanism module and the channel attention mechanism module, the input channel attention and spatial attention are the same feature map X∈R C×H×W , where C is the number of channels, H and W are the height and width of the feature map; the output is the processed feature map.

[0054] A structure that can be realized, the channel attention mechanism module includes two branches, the first branch includes an adaptive maximum pooling layer, a first convolution layer, a first activation layer, and a second convolution layer connected in sequence; the second branch includes an adaptive average pooling layer, a third convolution layer, a second activation layer, and a fourth convolution layer connected in sequence; the outputs of the second convolution layer and the fourth convolution layer are spliced ​​and connected to the third activation layer;

[0055] A possible implementation structure, the spatial attention mechanism module includes a maximum pooling layer, an average pooling layer, and a fifth convolutional layer and a fourth activation layer connected after the two pooling layers;

[0056] The first activation layer and the second activation layer use Re lu activation function respectively, and the third activation layer and the fourth activation layer use Si gmoi d activation function;

[0057] The channel attention mechanism module and the spatial attention mechanism module are described below respectively;

[0058] The channel attention module first applies adaptive average pooling and adaptive maximum pooling to the input feature map, thereby obtaining two 1×1 feature maps as follows

[0059]

[0060] Next, the two feature maps are respectively reduced in number of channels through the fc1 convolution layer (i.e., the first convolution layer and the third convolution layer), and then nonlinearly activated through the Relu function of the first activation layer and the second activation layer, where X has been processed by average pooling and maximum pooling, and W fc1 is the convolution kernel, and the formula is as follows:

[0061] FC1(X)=ReLU(Conv2D(X,W fc1 ))(3)

[0062] Then the number of channels is restored through the fc2 convolutional layer (i.e., the second convolutional layer and the fourth convolutional layer), where W fc2 is the convolution kernel, and the formula is as follows:

[0063] FC2(X)=Conv2D(FC1(X),W fc2 )(4)

[0064] Next, add the two results together, and finally use the Sigmoid activation function to activate the final channel attention weight. The formula is as follows:

[0065] Attention(X)=Sigmoid(FC2(AAvgPool(X))+FC2(AMaxPool(X)))(5)

[0066] The spatial attention module first performs average pooling and maximum pooling on the input feature map to obtain two feature maps, and then converts them into 2×H×W by maintaining the dimension. The formula is as follows:

[0067]

[0068] Next, the two feature maps are connected in the channel dimension to obtain a 2×H×W feature map, which is then processed through the conv1 convolution layer (i.e. the fifth convolution layer) to output a 1×H×W feature map, where W conv1 is the convolution kernel, and X is the result after average pooling and maximum pooling. The specific formula is as follows:

[0069] Conv1(X)=Conv2D(Concat([AvgPool(X),MaxPool(X)]),W conv1 )(8)

[0070] Finally, the final spatial attention weight is generated through the Sigmoid activation function:

[0071] Attention(X)=Sigmoid(Conv1(X))(9)

[0072] The channel attention weights and spatial attention weights obtained in the channel attention module and the spatial attention module are finally multiplied by the feature map of the original input to obtain a processed new feature map.

[0073] In some embodiments, the loss function set by the loss function unit of the appearance feature extraction model includes a cross entropy loss and a triplet loss based on the optimal transmission Sinkhorn algorithm;

[0074] The improved loss function is used to optimize the training network to enhance the feature extraction capability and improve the sample measurement effect. The improved method is: in the original triplet loss function calculation process, the optimal transmission algorithm is used to adjust the distance between samples to optimize the discriminability of the feature space.

[0075] A further technical solution, an improved method for determining the triplet loss function value, comprises the following steps:

[0076] Step 21: Calculate the distance matrix between samples by Euclidean distance or cosine distance for subsequent sample mining and loss calculation;

[0077] Step 22: construct a positive and negative sample indicator matrix based on the sample label information for subsequent calculation of the positive and negative sample distances;

[0078] Step 23: Use the optimal transmission Sinkhorn algorithm to determine the sample weights;

[0079] Step 24: According to the obtained distance matrix between samples, the positive and negative sample indicator matrix and the sample weights, a difficult sample mining method and a weighted sample mining method are used to calculate and obtain the positive sample distance and the negative sample distance;

[0080] Step 25: Calculate the final loss based on the obtained positive sample distance, negative sample distance and the input expected distance difference;

[0081] Specifically, in step 24, the sample weight is considered when calculating the weighted positive and negative sample distances using the weighted sample mining method;

[0082] In this embodiment, the optimally transmitted Sinkhorn algorithm is used in the weighted sample mining method to determine the sample weights, thereby obtaining more stable positive sample distances and negative sample distances;

[0083] There are five inputs in the calculation process of the improved triplet loss function value, including: the feature vector output by the appearance feature extraction model, the label information of the sample, the expected distance difference between the positive sample and the negative sample, norm_feat, and hard_mining;

[0084] Among them, norm_feat and hard_mining are two Boolean values. norm_feat indicates whether to normalize the feature vector, and hard_mining indicates whether to enable the hard sample mining strategy.

[0085] In step 21, according to the value of norm_feat, choose which method to use to calculate the distance matrix dist_mat; if norm_feat is True, cosine distance calculation is used, otherwise Euclidean distance is used.

[0086] In step 22, positive sample and negative sample indicator matrices are constructed respectively according to whether the label information target of the sample is of the same category: an is_pos indicator matrix is ​​constructed to indicate whether each pair of samples belongs to the same category; and an is_neg indicator matrix is ​​constructed to indicate whether each pair of samples belongs to different categories.

[0087] In step 23, select the appropriate sampling strategy based on the hard_mining parameter, specifically:

[0088] If hard_mining is True, the hard sample mining function is used to calculate the distance between positive and negative sample pairs according to hard_example_mining, otherwise the weighted sample mining function weighted_example_mining is used to calculate the distance between positive and negative samples.

[0089] In the weighted sample mining function, the positive sample distance and negative sample distance are first calculated, and then the positive sample weight and negative sample weight are calculated using the optimal transmission, and then the weighted calculation is performed to obtain the final positive and negative sample distance. Finally, the final loss is calculated based on the obtained positive and negative sample distance and the input expected distance difference margin.

[0090] The input of the Sinkhorn algorithm includes the distance matrix dist_mat, the mask matrix mask, the regularization parameter reg, and the number of iterations iter_num;

[0091] Distance matrix dist_mat: is a square matrix representing the distance between each point in the source set and the target set;

[0092] Mask matrix mask: used to represent the transmission status between point pairs, indicating which point pairs are allowed to transmit. Usually the elements in the mask matrix are 0 or 1, where 1 means that transmission is allowed and 0 means that it is not allowed.

[0093] Regularization parameter reg: used to control the degree of regularization of the algorithm, which helps to ensure the existence and uniqueness of the solution;

[0094] Iteration number iter_num, that is, the number of iterations that the Sinkhorn iterative algorithm needs to execute.

[0095] The output of the Sinkhorn algorithm is transport_plan, the optimal transmission plan, which is a square matrix representing the probability of each point in the source set being transmitted to a point in the target set;

[0096] In step 24, using the optimal transmission Si nkhorn algorithm to determine the sample weights includes the following steps:

[0097] Step 241: Obtain a source set and a target set, wherein the source set is the distance between samples, and the target set is the uniformly distributed weights;

[0098] Step 242: Initialize the acquired set, that is, initialize the source set and the target set to be uniformly distributed;

[0099] Step 243, update the transmission probability between point pairs according to the cost function and the regularization parameter, use the exponential function to transform the distance matrix, combine the regularization parameter, calculate the transmission probability between each pair of samples, and gradually update the transmission plan to ensure that it meets the marginal constraints; at the same time, normalize the transmission plan so that the sum of each row and column is the weight of the corresponding source set and target set, and this iterative process will continue until the specified number of iterations iter_num is reached, so that the transmission plan converges to a stable state, and finally obtains reasonable sample weights.

[0100] Initialize the elements to be used. N represents the dimension of the distance matrix dist_mat, that is, the number of rows or columns of the matrix. a and b are both initialized vectors, all of which are Used for the subsequent normalization step. The optimal transport plan transport_plan is an initialized matrix whose elements are It means that in the absence of other information, the transmission probability between each pair of points is equal.

[0101] Next, a For loop is performed to iterate according to the specified number of iterations iter_num. In each iteration, the optimal transmission plan transport_plan is multiplied by exp(-dist_mat / reg), which is equivalent to updating the transmission probability between point pairs according to the cost and regularization parameter. Then, the rows and columns of the optimal transmission plan transport_plan are normalized respectively to ensure that the sum of each row and column is 1, that is, the total transmission probability of each source point and destination point is 1. Finally, the normalized optimal transmission plan transport_plan is multiplied by the mask matrix mask to obtain the final optimal transmission plan.

[0102] In step 3, the trajectory of the video frame to be tracked is obtained, and the motion information is predicted using Kalman filtering to obtain the position information and the predicted trajectory;

[0103] In step 4, two association matches are performed to perform association matches between the previous and next video frames, update the trajectory information, and obtain the final matching information. The specific association method is as follows:

[0104] Step 41: The first data association uses the spatial information and the appearance information to calculate the similarity matrix, selects the smaller one as the cost matrix, and performs data association through the Hungarian algorithm; specifically, it includes:

[0105] Step 411: Based on the spatial information obtained by Kalman filtering, the intersection-over-union ratio between the predicted bounding box and the true bounding box is calculated by IOU matching to measure the overlap between the predicted bounding box and the true bounding box, thereby obtaining a similarity matrix;

[0106] The spatial information is the location information obtained in step 3;

[0107] Step 412: Calculate the appearance similarity matrix based on the appearance feature information obtained by the appearance feature extraction model;

[0108] Step 413: Based on the minimum similarity matrix, the Hungarian algorithm is used to perform data association to obtain information of successful matches and some unmatched trajectories.

[0109] Step 42: For the tracks that were not successfully matched in the first association, the tracks are matched again with the video frames with low confidence detection frames in the second association by using the Hungarian algorithm, and if the tracks are successfully matched, the track information is updated. Specifically, it includes:

[0110] Step 421, calculating the similarity matrix based on the spatial information by means of IOU matching;

[0111] Step 422: Use the Hungarian algorithm to match the first unmatched trajectories and the video frames of the low-scoring detection frames according to the similarity matrix; update the trajectory information according to the matching information.

[0112] In this embodiment, the spatial and channel attention mechanism modules are used in the appearance feature extraction part to enhance the feature extraction capability, and by modeling the interdependence between the channel dimension and the spatial dimension of the feature map, it is possible to selectively emphasize features with rich information and suppress features with weaker information. At the same time, in network training, an improved triple loss function is used for network optimization. The improved triple loss function calculates the distance matrix between samples through Euclidean distance or cosine distance, constructs a positive and negative sample indicator matrix based on the label information of the samples, uses a weighted instance mining method to calculate the weighted positive and negative sample distances, and uses the optimal transmission Sinkhorn algorithm to determine the sample weights, thereby obtaining more stable positive sample distances and negative sample distances.

[0113] Example 2

[0114] Based on Example 1, this embodiment provides a multi-target tracking system based on an optimal transmission algorithm and an attention mechanism, including:

[0115] Preprocessing module: configured to obtain a video sequence to be tracked, perform target detection to obtain a video frame containing a detection frame, perform preprocessing to obtain a target image, and divide the image into a video frame with a high-confidence detection frame and a video frame with a low-confidence detection frame according to the detection frame score;

[0116] Feature extraction module: configured to extract appearance features from video frames with high confidence detection frames and preprocessed video frames through the trained appearance feature extraction model; the appearance feature extraction model uses an improved triplet loss function for network optimization, and adds the optimal transmission Si nkhorn algorithm to the original triplet loss function to adjust the distance matrix;

[0117] Trajectory prediction module: configured to obtain the trajectory of the video frame to be tracked to predict the motion information, obtain the position information, and then obtain the predicted trajectory;

[0118] Matching association module: It is configured to perform a first data association between the extracted appearance feature extraction and the predicted trajectory; perform a second data association between the video frame with the low confidence detection box and the trajectory that is not matched by the first data association, and match the trajectory of the tracked target.

[0119] It should be noted here that the various modules in this embodiment correspond one-to-one to the various steps in Example 1, and the specific implementation process is the same, which will not be repeated here.

[0120] Example 3

[0121] This embodiment provides an electronic device, including a memory and a processor, and computer instructions stored in the memory and running on the processor. When the computer instructions are executed by the processor, the steps in the multi-target tracking method based on the optimal transmission algorithm and attention mechanism of Example 1 are completed.

[0122] Example 4

[0123] This embodiment provides a computer-readable storage medium for storing computer instructions. When the computer instructions are executed by a processor, the steps in the multi-target tracking method based on the optimal transmission algorithm and attention mechanism of Example 1 are completed.

[0124] The above description is only a preferred embodiment of the present disclosure and is not intended to limit the present disclosure. For those skilled in the art, the present disclosure may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.

[0125] Although the above describes the specific implementation methods of the present disclosure in conjunction with the accompanying drawings, it is not intended to limit the scope of protection of the present disclosure. Technical personnel in the relevant field should understand that on the basis of the technical solution of the present disclosure, various modifications or variations that can be made by those skilled in the art without creative work are still within the scope of protection of the present disclosure.

Claims

1. A multi-target tracking method based on an optimal transmission algorithm and an attention mechanism, characterized in that: The steps include: Obtain the video sequence to be tracked, perform target detection to obtain video frames containing detection frames, perform preprocessing to obtain target images, and divide them into video frames with high-confidence detection frames and video frames with low-confidence detection frames according to the detection frame scores; The video frames with high confidence detection frames and the preprocessed video frames are subjected to appearance feature extraction through the trained appearance feature extraction model; The appearance feature extraction model uses an improved triplet loss function for network optimization, and adds the optimal transmission Sinkhorn algorithm to the original triplet loss function to adjust the distance matrix; Obtain the trajectory of the video frame to be tracked to predict the motion information, obtain the position information, and then obtain the predicted trajectory; The extracted appearance features are extracted and the predicted trajectory is subjected to the first data association; the video frame with the low confidence detection box is subjected to the second data association with the trajectory that is not matched by the first data association, and the trajectory of the tracked target is matched.

2. The multi-target tracking method based on the optimal transmission algorithm and the attention mechanism as claimed in claim 1, characterized in that: The appearance feature extraction model includes a newly added attention mechanism module. The newly added attention mechanism module adds a spatial attention mechanism module and a channel attention mechanism module after the last convolutional layer of the backbone network to enhance the ability to capture distinguishing features between different targets with similar appearances and obtain a feature map.

3. The multi-target tracking method based on the optimal transmission algorithm and the attention mechanism as claimed in claim 2, characterized in that: The channel attention mechanism module includes two branches. The first branch includes an adaptive maximum pooling layer, a first convolutional layer, a first activation layer, and a second convolutional layer connected in sequence. The second branch includes an adaptive average pooling layer, a third convolutional layer, a second activation layer, and a fourth convolutional layer connected in sequence. The outputs of the second convolutional layer and the fourth convolutional layer are concatenated and connected to the third activation layer. Alternatively, the spatial attention mechanism module includes a maximum pooling layer and an average pooling layer, and a fifth convolutional layer and a fourth activation layer connected after the two pooling layers.

4. The multi-target tracking method based on the optimal transmission algorithm and the attention mechanism as claimed in claim 1, characterized in that: During the training process of the appearance feature extraction model, an improved method for determining the triplet loss function value is adopted, which includes the following steps: Calculate the distance matrix between samples using Euclidean distance or cosine distance; According to the label information of the sample, construct the positive and negative sample indicator matrix; The optimal transmission Sinkhorn algorithm is used to determine the sample weights; According to the obtained distance matrix between samples, positive and negative sample indicator matrix and sample weights, the difficult sample mining method and the weighted sample mining method are used to calculate and obtain the positive sample distance and the negative sample distance; The final loss is calculated based on the obtained positive sample distance, negative sample distance and expected distance difference of the input.

5. The multi-target tracking method based on the optimal transmission algorithm and the attention mechanism as claimed in claim 4, characterized in that: The optimal transmission Sinkhorn algorithm is used to determine the sample weights, which includes the following steps: Get the source set and the target set, where the source set is the distance between samples and the target set is the uniformly distributed weights; Initialize the source and target collections to be uniformly distributed; The transmission probability between point pairs is updated according to the cost function and regularization parameters, the distance matrix is ​​transformed using an exponential function, and the transmission probability between each pair of samples is calculated in combination with the regularization parameters, and the transmission plan is gradually updated to meet the marginal constraints; at the same time, the transmission plan is normalized so that the sum of each row and column is the weight of the corresponding source set and target set, and this is obtained in sequence until the specified number of iterations is reached, so that the transmission plan converges to a stable state and the sample weight is obtained.

6. The multi-target tracking method based on the optimal transmission algorithm and the attention mechanism as claimed in claim 1, characterized in that: The appearance feature extraction model includes a backbone network, an aggregation module, a detection head, and a loss function unit; Backbone network: extracts features from the input image and increases the weight of the features based on the attention mechanism, and outputs feature maps; Aggregation module: Aggregate the features of the backbone network and aggregate the features generated by the backbone network into a global feature; Detection head: performs target detection based on aggregated features and outputs a target image with a detection frame; Loss function unit: used to implement loss function calculation in the training process of appearance feature extraction model.

7. The multi-target tracking method based on the optimal transmission algorithm and attention mechanism as claimed in claim 1, characterized in that: The loss function set by the loss function unit of the appearance feature extraction model includes cross entropy loss and triple loss based on the optimal transmission Sinkhorn algorithm.

8. A multi-target tracking system based on an optimal transmission algorithm and an attention mechanism, characterized in that: include: Preprocessing module: configured to obtain a video sequence to be tracked, perform target detection to obtain a video frame containing a detection frame, perform preprocessing to obtain a target image, and divide the image into a video frame with a high-confidence detection frame and a video frame with a low-confidence detection frame according to the detection frame score; Feature extraction module: configured to extract appearance features from the high-resolution video frame and the processed video frame using a trained appearance feature extraction model; The appearance feature extraction model uses an improved triplet loss function for network optimization, and adds the optimal transmission Sinkhorn algorithm to the original triplet loss function to adjust the distance matrix; Trajectory prediction module: configured to obtain the trajectory of the video frame to be tracked to predict the motion information, obtain the position information, and then obtain the predicted trajectory; Matching association module: It is configured to perform a first data association between the extracted appearance feature extraction and the predicted trajectory; perform a second data association between the video frame with the low confidence detection box and the trajectory that is not matched by the first data association, and match the trajectory of the tracked target.

9. An electronic device, characterized in that: The method comprises a memory and a processor and computer instructions stored in the memory and executed on the processor. When the computer instructions are executed by the processor, the steps in the multi-target tracking method based on the optimal transmission algorithm and the attention mechanism as described in any one of claims 1 to 7 are completed.

10. A computer-readable storage medium, characterized in that: Used to store computer instructions, which, when executed by a processor, complete the steps of the multi-target tracking method based on the optimal transmission algorithm and attention mechanism described in any one of claims 1-7.

Images