Learable multi-target tracking method based on distance guidance

Abstract

The invention discloses a learnable multi-target tracking method based on distance guidance, and belongs to the technical field of computer vision. The learnable multi-target tracking method based on distance guidance is mainly realized by a learnable interaction module and a distance mapping module. Distance mapping strategies including a general distance mapping module and a modulation distance mapping module are respectively set according to the interaction content of the learnable interaction module. The participation weight of the historical information is effectively determined according to the mapping result, reasonable utilization of the historical information by the multi-target tracking model is enhanced, the tracking capability of the multi-target tracking model in a complex tracking scene is improved, and finally, the distance attention weight and the learnable interaction result are fused in a weighted summation mode, so that the learning effect of the multi-target tracking model is improved. And distance information is used as priori knowledge to guide learning optimization of a multi-target tracking model. Therefore, the tracking robustness of the multi-target tracking model is enhanced, the research and development cost of the multi-target tracking model is reduced, and the application scene is expanded.

Description

Technical Field

[0001] The present invention relates to the technical field of computer vision, and more specifically to a learnable multi-target tracking method based on distance guidance. Background Art

[0002] In recent years, computer vision has been used more and more widely in practical scenarios. As a comprehensive task of classification, positioning and target re-identification, multi-target tracking plays an important role in autonomous driving, security monitoring and video understanding. With the development of deep learning, multi-target tracking models have begun to design concise learnable interaction modules to adaptively interact with detection information and tracking information, and realize the mining and utilization of deep tracking clues. However, in existing methods, the objects of adaptive information interaction are all detection information and all tracking information. This feature makes the training of interaction modules a global optimization problem. On the one hand, it reduces the learning efficiency of the multi-target tracking model and requires a large number of iterative optimizations to obtain ideal performance. On the other hand, the multi-target tracking model cannot learn useful tracking clues in crowded, occluded, complex motion and other scenes, which makes the tracking robustness of the multi-target tracking model worse and limits its usage scenarios. Summary of the invention

[0003] 1. Technical issues to be solved

[0004] In order to solve the technical problems raised by the above-mentioned background technology, the present invention provides a learnable multi-target tracking method based on distance guidance, which uses the position information of the object to guide the optimization of the learnable interaction module, improves the optimization efficiency of the learnable interaction module, enhances the tracking robustness of the multi-target tracking model, thereby reducing the deployment cost of the multi-target tracking model and expanding the application scenarios.

[0005] 2. Technical solution

[0006] The purpose of the present invention is achieved through the following technical solutions.

[0007] A learnable multi-target tracking method based on distance guidance includes the following steps:

[0008] Constructing a multi-target tracking model, the multi-target tracking model includes a target detector, a detection result encoding module, a learnable interaction module, a distance mapping module and a tracking decoding module;

[0009] Obtain the video to be detected, input the video with detection into the target detector for frame-by-frame detection, obtain the detection result of each frame image, input the detection result into the detection result encoding module for encoding processing, and obtain the detection object feature expression vector;

[0010] For the first frame image in the video to be detected, the detected object feature expression vector is input into the tracking decoding module to obtain the tracking result; the tracking result is processed to obtain the effective tracking result feature expression vector and the historical feature expression vector;

[0011] Starting from the second frame image in the video to be detected, the feature expression vector of the detected object, the feature expression vector of the effective tracking result and the historical feature expression vector are input into the learnable interaction module for interaction to obtain the learnable interaction expression; the position information corresponding to the feature expression vector of the detected object and the feature expression vector of the effective tracking result is input into the distance mapping module to obtain the distance attention weight; based on the distance attention weight, the feature expression vector of the detected object and the feature expression vector of the effective tracking result are guided to interact to obtain the feature expression vector of the detected object and the feature expression vector of the effective tracking result after interaction; the feature expression vector of the detected object and the feature expression vector of the effective tracking result after interaction are input into the tracking decoding module to obtain the tracking result.

[0012] As a further improvement of the present invention, the detection result encoding module includes a feature mapping submodule and an encoder mapping submodule;

[0013] The feature mapping submodule is represented as:

[0014] M f =Linear3(ReLU(Linear2(ReLU(Linear1(x)))))

[0015] Among them, M f represents the feature mapping submodule, x represents the input information obtained from the detection result, Linear1, Linear2 and Linear3 represent different linear mapping modules, and ReLU represents the rectified linear unit;

[0016] The encoder mapping submodule is represented as:

[0017] M e ={Conv2d(di,d,kernel size =1),i=1,2,..L}

[0018] Among them, M e represents the encoder mapping submodule, i represents a natural number, L represents the number of selected feature layers, di represents the input channel, d represents the output channel, kernel size Represents the convolution kernel size, and Conv2d represents a two-dimensional convolution kernel.

[0019] As a further improvement of the present invention, the feature mapping submodule maps the detection result to obtain a detection object feature expression vector; the detection object feature expression vector is expressed as:

[0020] D q =M f (Sa(box,Bf))

[0021] Among them, D q Represents the detected object feature expression vector, Sa represents the feature sampling of the output features of the target detector backbone network according to the location box of the detection result, box represents the location box of the detection result, and Bf represents the output features of the target detector backbone network;

[0022] The output features of the target detector backbone network are input into the encoder mapping submodule to obtain the encoding basic features; the encoding basic features are expressed as:

[0023] tgt=M e (Bf)

[0024] Among them, tgt represents the basic encoding feature.

[0025] As a further improvement of the present invention, the input and output forms of the tracking and decoding module are:

[0026]

[0027] Among them, AD e Represents the tracking decoding module, represents the feature expression vector of object X, box1 represents the position box corresponding to object X, represents the tracking feature expression vector corresponding to object X, and box2 represents the tracking position box corresponding to object X;

[0028] In the first frame image, the feature expression vector of the detected object is input into the tracking decoding module to obtain the tracking result. The tracking process is expressed as:

[0029] (O d ,box′ d )=AD e (tgt,D q ,box d )

[0030] Among them, D q Represents the feature expression vector of the detected object, box d Represents the location box of the detected object, O d Represents the tracking feature expression vector corresponding to the detected object, box′ d Indicates the tracking position box corresponding to the detected object.

[0031] As a further improvement of the present invention, the processing of the tracking result to obtain the effective tracking result feature expression vector includes: setting the classification confidence positive and negative thresholds, if the classification confidence of the tracking result is higher than the classification confidence positive and negative thresholds, then the tracking result is a valid tracking result, adding the tracking sequence, marking its tracking feature expression vector as a valid tracking result feature expression vector; the effective tracking result feature expression vector is expressed as:

[0032]

[0033] Among them, T q Represents the effective tracking result feature expression vector, Represents the tracking feature expression vector corresponding to the input i-th detection result, Represents the classification confidence corresponding to the i-th detection result of the input, τ n Indicates the positive and negative thresholds of classification confidence;

[0034] The processing of the tracking results to obtain the historical feature expression vector includes: recording and updating the historical feature expression vector of the effective tracking results by a sliding average method; the historical feature expression vector is expressed as:

[0035]

[0036] Among them, H q represents the historical feature expression vector, w represents the update weight, t represents the time, Represents the historical feature expression vector at time t, Represents the historical feature expression vector at time t-1, and * represents a multiplication operation.

[0037] As a further improvement of the present invention, the learnable interaction module is expressed as:

[0038] LIM(Q,K,V,V base ,M)=O4

[0039] O1=MHAM(Q,K,V,M)

[0040] O2=Norm1(O1+V base )

[0041] O3=Linear2(Dropout(Activation(Linear1(O2)))

[0042] O4=Norm2(O2+O3)

[0043] Among them, LIM represents a learnable interaction module, Q, K, V, V baseRepresents different input vectors, M represents the attention matrix, O1, O2, O3 and O4 represent different intermediate vectors, MHAM represents a multi-head attention module containing attention weights, Norm1 and Norm2 represent different normalization modules, Linear1 and Linear2 represent different linear mapping modules, Dropout represents a random loss module, and Activation represents a nonlinear mapping module.

[0044] As a further improvement of the present invention, the detection object feature expression vector, the effective tracking result feature expression vector and the history feature expression vector are input into the learnable interaction module for interaction to obtain the learnable interaction expression, including:

[0045] Update the effective tracking result feature expression vector, and the update formula is:

[0046]

[0047] Among them, NTq represents the updated effective tracking result feature expression vector, LIM1 represents the first learnable interaction module, M t represents the distance attention weight between the effective tracking result feature expression vector and the detected object feature expression vector and the historical feature expression vector. concat represents the connection operation. Represents the effective tracking result feature expression vector with location information, Represents the feature expression vector of the detected object with position information, Represents the historical feature expression vector with location information;

[0048] Update the feature expression vector of the detected object, and the update formula is:

[0049]

[0050] Among them, ND q represents the updated detection object feature expression vector, LIM2 represents the second learnable interaction module, N q represents a vector composed of random noise, M d represents the generation result of the general distance mapping module;

[0051] The effective tracking result feature expression vector and the detected object feature expression vector are comprehensively updated, and the update formula is:

[0052]

[0053] Among them, Q t represents the effective tracking result feature expression vector after comprehensive update, Q drepresents the updated detection object feature expression vector, LIM3 represents the third learnable interaction module, represents the updated effective tracking result feature expression vector with position information, represents the updated detected object feature expression vector with position information, M a Represents another result generated by the general distance mapping module.

[0054] As a further improvement of the present invention, the distance mapping module includes a general distance mapping module and a modulation distance mapping module;

[0055] The general distance mapping module is expressed as:

[0056] DM g (box1,box2)=M

[0057] dist = 1-IOU(box1, box2)

[0058]

[0059] Among them, DM g Represents the general distance mapping module, dist represents the distance information obtained based on the intersection-over-union ratio of box1 and box2, IOU represents the intersection-over-union ratio function, e represents the natural constant, and σ represents the hyperparameter;

[0060] The modulation distance mapping module is expressed as:

[0061] DM m (box1, box2) = M m

[0062] dist = 1-IOU(box1,box2)

[0063] o ij =BiM(dist)

[0064] IOU m [~o ij ]=IOU[~o ij ]*0.5

[0065] IOU m [o ij ]=IOU[o ij ]

[0066] Among them, DM m represents the modulation distance mapping module, M m represents the intermediate variable, o ij Represents the object pair matched by the optimal bipartite matching algorithm, BiM represents the optimal bipartite matching algorithm, IOUm represents the I / O function after modulation;

[0067] The distance attention weight is calculated by the modulated intersection-over-parameter function, and the calculation formula is:

[0068] dist m =1-IOU m

[0069]

[0070] Among them, dist m Indicates the distance after modulation.

[0071] As a further improvement of the present invention, the detected object feature expression vector and the effective tracking result feature expression vector after interaction are input into the tracking decoding module to obtain the tracking result, which is expressed as:

[0072] (O M ,box′ M )=AD e (tgt,M q ,box M )

[0073] M q =concat(Q t ,Q d )

[0074] box M =concat(box t box d )

[0075] Among them, O′ M =concat(Q′ t , Q′ d ), which correspond to the effective tracking result feature expression vector and the detected object feature expression vector after comprehensive update, box′ M =concat(box′ t , box′ d ), which correspond to the location boxes of the effective tracking result feature expression vector and the detected object feature expression vector after comprehensive update.

[0076] As a further improvement of the present invention, for the tracking result output by the detection object feature expression vector, if the classification confidence of the tracking result is higher than the classification confidence positive and negative thresholds, the tracking result is marked as a newly appeared object, a unique ID is assigned, and the tracking result is added to the tracking sequence;

[0077] For the tracking result output by the effective tracking result feature expression vector after interaction, if the classification confidence of the tracking result is lower than the positive and negative thresholds of the classification confidence, the tracking result is marked as disappeared, and the tracked object that is in the disappeared state for T consecutive frames is removed from the tracking sequence.

[0078] 3. Beneficial effects

[0079] Compared with the prior art, the advantages of the present invention are:

[0080] The present invention discloses a learnable multi-target tracking method based on distance guidance. The proposed learnable multi-target tracking method based on distance guidance can use distance information to help the learnable interaction module suppress irrelevant noise and accelerate the learning efficiency of the multi-target tracking model. At the same time, the attention of the learnable interaction module is limited to an effective range through distance information, and the learning of the relationship between objects in challenging scenes such as occlusion and crowding is enhanced through distance modulation strategy. In addition, the participation weight of historical information is determined based on the current distance information, the rational use of historical information by the multi-target tracking model is strengthened, and the tracking ability of the multi-target tracking model in complex tracking scenes is improved, which has strong practicality and wide applicability. BRIEF DESCRIPTION OF THE DRAWINGS

[0081] Figure 1 This is a schematic diagram of the structure of a multi-target tracking model according to an embodiment of the present invention;

[0082] Figure 2 This is a schematic diagram of the output result of the distance mapping module according to an embodiment of the present invention;

[0083] Figure 3 Schematic diagram of historical information weight of the distance mapping module according to an embodiment of the present invention. DETAILED DESCRIPTION

[0084] The present invention is described in detail below in conjunction with the accompanying drawings and specific embodiments.

[0085] Example

[0086] like Figure 1 As shown, the multi-target tracking model constructed in this embodiment includes a target detector, a detection result encoding module, a learnable interaction module, a distance mapping module and a tracking decoding module.

[0087] First, in the tracking scene, a pre-trained target detector is used to detect the video frames frame by frame to obtain the detection results of each frame. It should be noted that in this embodiment, the target detector used can be a pre-trained target detector based on CNN or a pre-trained target detector based on transformer. In this embodiment, the existing pre-trained target detection model YOLOX is selected as the target detector.

[0088] For the detection result encoding module, the detection result encoding module includes a feature mapping submodule and an encoder mapping submodule. In this embodiment, the feature mapping submodule is represented as:

[0089] M f =Linear3(ReLU(Linear2(ReLU(Linear1(x)))))

[0090] Among them, M f represents a feature mapping submodule, x represents input information obtained from the detection result, Linear1, Linear2 and Linear3 represent different linear mapping modules, and ReLU represents a rectified linear unit. In this embodiment, Linear1 is a 2240×960 linear mapping module, Linear2 is a 960×960 linear mapping module, and Linear3 is a 960×256 linear mapping module.

[0091] In this embodiment, the encoder mapping submodule is represented as:

[0092] M e = {Conv2d(di, d, kernel size =1), i=1, 2, ..L}

[0093] Among them, M e represents the encoder mapping submodule, i represents a natural number, L represents the number of selected feature layers, di represents the input channel, d represents the output channel, kernel size Represents the convolution kernel size, Conv2d represents a two-dimensional convolution kernel. For the encoder mapping submodule, there are:

[0094] M e ={Conv2d(1280, 256, kernel size =1),

[0095] Conv2d(640, 256, kernel size =1),

[0096] Conv2d(320, 256, kernel size =1)}

[0097] Then, input the detection result into the detection result encoding module for encoding to obtain the detection object feature expression vector. Specifically, establish a rectangular coordinate system, encode the detection result through the detection result encoding module, and record the position box of the detection result as box = (x1, y1, x2, y2), where (x1, y1) are the horizontal and vertical coordinates of the upper left corner of the position box, and (x2, y2) are the horizontal and vertical coordinates of the lower right corner of the position box. Feature sampling is performed on the output features of the target detector backbone network according to the position box. The detection result, that is, the sampling result, is mapped through the feature mapping submodule to obtain the detection object feature expression vector. In this embodiment, the detection object feature expression vector is expressed as:

[0098] D q =M f (Sa(box,Bf))

[0099] Among them, D q Represents the feature expression vector of the detected object, Sa represents the feature sampling of the output features of the target detector backbone network according to the location box of the detection result, box represents the location box of the detection result, and Bf represents the output features of the target detector backbone network.

[0100] The output features of the target detector backbone network are input into the encoder mapping submodule to obtain the encoding basic features. In this embodiment, the encoding basic features are expressed as:

[0101] tgt=M e (Bf)

[0102] Wherein, tgt represents the encoding basic feature. It should be noted that, in this embodiment, the last three feature layers of the output feature are selected for mapping.

[0103] For the tracking decoding module, the tracking decoding module is composed of a decoder network based on Transformer. In this embodiment, the tracking decoding module directly uses the decoder network of the DETR type target detector. For example, the decoder network of the DAB-DETR target detector is used. Therefore, its input and output forms are:

[0104]

[0105] Among them, AD e Represents the tracking decoding module, represents the feature expression vector of object X, box1 represents the position box corresponding to object X, represents the tracking feature expression vector corresponding to object X, and box2 represents the tracking position box corresponding to object X.

[0106] Therefore, in this embodiment, for the first frame image in the video to be detected, the detected object feature expression vector is input into the tracking decoding module to obtain the tracking result. The tracking result is processed to obtain the effective tracking result feature expression vector and the historical feature expression vector. Specifically, in the first frame image, the detected object feature expression vector is input into the tracking decoding module to obtain the tracking result, and the calculation formula is:

[0107] (O d , box′ d )=AD e (tgt′D q ,box d )

[0108] Among them, D q Represents the feature expression vector of the detected object, box d Represents the location box of the detected object, O d Represents the tracking feature expression vector corresponding to the detected object, box′ d Indicates the tracking position box corresponding to the detected object.

[0109] Furthermore, the tracking result is processed to obtain an effective tracking result feature expression vector. Specifically, a classification confidence positive and negative threshold is set. If the classification confidence of the tracking result is higher than the classification confidence positive and negative threshold, the tracking result is an effective tracking result, and its tracking feature expression vector is marked as an effective tracking result feature expression vector. At the same time, it is marked as a newly appeared object, given a unique ID, and added to the tracking sequence. In this embodiment, the effective tracking result feature expression vector T q It is expressed as:

[0110]

[0111] Among them, T q Represents the effective tracking result feature expression vector, Represents the tracking result feature expression vector corresponding to the input i-th detection result, Represents the classification confidence corresponding to the i-th detection result of the input, τ n Indicates the classification confidence positive and negative thresholds. In this embodiment, the classification confidence positive and negative thresholds τ are set n is 0.5.

[0112] Furthermore, the tracking results are processed to obtain a historical feature expression vector. Specifically, as time goes by, the historical feature expression vector of the effective tracking results is recorded and updated by a sliding average method to supplement the historical information. In this embodiment, the historical feature expression vector is expressed as:

[0113]

[0114] Among them, H q represents the historical feature expression vector, w represents the update weight, t represents the time, Represents the historical feature expression vector at time t, represents the historical feature expression vector at time t-1, and * represents a multiplication operation. In this embodiment, the update weight w is set to 0.3.

[0115] For the learnable interaction module, the learnable interaction module is expressed as:

[0116] LIM(Q,K,V,V base , M)=O4

[0117] O1=MHAM(Q,K,V,M)

[0118] O2=Norm1(O1+V base )

[0119] O3=Linear2(Dropout(Activation(Linear1(O2))))

[0120] O4=Norm2(O2+O3)

[0121] Among them, LIM represents a learnable interaction module, Q, K, V, V base represents different input vectors, Q represents an I×d vector, K represents a J×d vector, V and V base All represent I×d vectors, M represents an I×J attention matrix. In this embodiment, the dimension of d is set to 256, and I and J are determined by the number of objects participating in the interaction. O1, O2, O3, and O4 represent different intermediate vectors, MHAM represents a multi-head attention module containing attention weights, Norm1 and Norm2 represent different normalization modules. In this embodiment, Norm1 and Norm2 are normalization modules of dimension 256. Dropout represents a random deactivation module. In this embodiment, Dropout is a random deactivation module with a deactivation probability of 0. Activation represents a nonlinear mapping module, which is set to a rectified linear unit. Linear1 and Linea2 represent different linear mapping modules. In this embodiment, Linear1 is an I×256 linear mapping module, and Linear2 is a 256×I linear mapping module.

[0122] In this embodiment, the multi-head attention module MHAM including attention weights is expressed as:

[0123] MHAM(Q,KV,M)=Attention(QKM)·V

[0124]

[0125] Among them, i and j both represent natural numbers, and softmax represents the normalized exponential function.

[0126] The distance mapping module includes a general distance mapping module and a modulated distance mapping module. In this embodiment, the general distance mapping module is implemented by a Gaussian kernel function, which is expressed as:

[0127] DM g (box1,box2)=M

[0128] dist = 1-IOU(box1, box2)

[0129]

[0130] Among them, DM g represents a general distance mapping module, dist represents the distance information obtained based on the intersection-and-union ratio of box1 and box2, IOU represents the intersection-and-union ratio function, e represents a natural constant, σ represents a hyperparameter, and IOU(box1, box2) represents the intersection-and-union ratio function of the position box box1 and the position box box2. In this embodiment, the hyperparameter σ is set to 0.25.

[0131] The modulated distance mapping module is composed of the distance modulation module introduced on the basis of the general distance mapping module. The distance modulation module mainly modulates the distance through the optimal bisection matching algorithm BiM, which is expressed as:

[0132] DM m (box1, box2) = M m

[0133] dist = 1-IOU(box1,box2)

[0134] o ij =BiM(dist)

[0135] IOU m [~o ij ]=IOU[~o ij ]*0.5

[0136] IOU m [o ij ]=IOU[o ij ]

[0137] Among them, DM m represents the modulation distance mapping module, M m represents the intermediate variable, o ij Represents the object pair matched by the optimal bipartite matching algorithm, BiM represents the optimal bipartite matching algorithm, IOU mRepresents the modulated intersection-and-joint ratio function. In this embodiment, the optimal bipartite matching algorithm BiM is implemented by the Hungarian algorithm. Further, the distance attention weight is calculated by the modulated intersection-and-joint ratio function, and the calculation formula is:

[0138] dist m =1-IOU m

[0139]

[0140] Among them, dist m Represents the distance after modulation and is an intermediate variable.

[0141] Therefore, in this embodiment, starting from the second frame image in the video to be detected, the detection object feature expression vector, the effective tracking result feature expression vector and the historical feature expression vector are input into the learnable interaction module for interaction to obtain a learnable interaction expression.

[0142] In this embodiment, the learnable interactive expression includes updating the effective tracking result feature expression vector, updating the detected object feature expression vector, and comprehensively updating the effective tracking result feature expression vector and the detected object feature expression vector.

[0143] Then, the position information corresponding to the feature expression vector of the detected object and the feature expression vector of the effective tracking result is input into the distance mapping module to obtain the distance attention weight. In this embodiment, the distance attention weight is used to guide the interaction between the detection result and the tracking result. The distance attention weight is used to guide the interaction between the feature expression vector of the detected object and the feature expression vector of the effective tracking result to obtain the feature expression vector of the detected object and the feature expression vector of the effective tracking result after the interaction. Specifically, the effective tracking result feature expression vector is updated, and the update formula is:

[0144]

[0145] Among them, NTq represents the updated effective tracking result feature expression vector, LIM1 represents the first learnable interaction module, M t represents the distance attention weight between the effective tracking result feature expression vector and the detected object feature expression vector and the historical feature expression vector. concat represents the connection operation. Represents the effective tracking result feature expression vector with location information, Represents the feature expression vector of the detected object with position information, Represents the historical feature expression vector with location information.

[0146] For the detected object feature expression vector with location information and the historical feature expression vector with location information, we have:

[0147]

[0148] Wherein, PE represents the position encoding based on the position box. In this embodiment, the sine-cosine position encoding method is selected to implement the position encoding PE based on the position box. t Represents the location box of the effective tracking result feature expression vector, box d Represents the location box of the detected object feature expression vector, box h The location box representing the historical feature expression vector. It should be noted that, in this embodiment, the box is set h =box t , that is, the historical feature expression vector and the effective tracking result feature expression vector share the same location box.

[0149] In this embodiment, for M t , then:

[0150]

[0151] in, Represents the distance attention weight between the feature expression vector of the detected object and the feature expression vector of the effective tracking result. For the distance attention weight between the feature expression vector of the detected object and the feature expression vector of the effective tracking result, we have:

[0152]

[0153] Among them, DM m Indicates the modulation distance mapping module, box1 = box t ,box2=box d .

[0154] Represents the distance attention weight between the effective tracking result feature expression vector and the historical feature expression vector, aiming to enhance the model's use of historical information. This part is based on the modulated intersection-over-union function in the process of generating the distance attention weight between the detected object feature expression vector and the effective tracking result feature expression vector:

[0155]

[0156] in, represents the maximum value of each row in the modulated I / O function, and diag(·) represents the diagonal matrix formed by these values ​​as diagonal elements. Therefore, we have:

[0157] dist = 1-EIOU

[0158]

[0159] Among them, dist represents the distance information obtained based on the intersection and union ratio of box1 and box2.

[0160] Update the feature expression vector of the detected object, and the update formula is:

[0161]

[0162] Among them, ND q represents the updated detection object feature expression vector, LIM2 represents the second learnable interaction module, M d Represents the generation result of the general distance mapping module; N q Represents a vector composed of random noise, which is used to eliminate the feature expression vector of the detected object that is similar to the feature expression vector of the effective tracking result.

[0163] In this embodiment, M d Generated by the general distance mapping module, M d =DM g (box1, box2), where DM g Represents the general distance mapping module, box1 = box d ,box2=concat(box d ,box t ).

[0164] The effective tracking result feature expression vector and the detected object feature expression vector are comprehensively updated, and the update formula is:

[0165]

[0166] Among them, Q t represents the effective tracking result feature expression vector after comprehensive update, Q d represents the updated detection object feature expression vector, LIM3 represents the third learnable interaction module, represents the updated effective tracking result feature expression vector with position information, represents the updated detected object feature expression vector with position information, M a Represents another result generated by the general distance mapping module.

[0167] In this embodiment, M a By the general distance mapping module DM g Generate, which is expressed as:

[0168] M a =DM g (box1,box2)

[0169] box1=box2=concat(box t ,box d )

[0170] Furthermore, the detected object feature expression vector and the effective tracking result feature expression vector after interaction are input into the tracking decoding module to obtain the tracking result. In this embodiment, the tracking result is expressed as:

[0171] (O′ M ,box′ M )=AD e (tgt,M q ,box M )

[0172] M q =concat(Q t Q d )

[0173] box M =concat(box t ,box d )

[0174] Among them, O′ M =concat(Q′ t , Q′ d ), which correspond to the effective tracking result feature expression vector and the detected object feature expression vector after comprehensive update, box′ M =concat(box′ t , box′ d ), which correspond to the position box of the effective tracking result feature expression vector and the detected object feature expression vector after comprehensive update. τ and Q′ d Input the classification head network composed of linear mapping networks respectively to obtain the classification confidence s t and d .

[0175] Therefore, in this embodiment, for the tracking result output by the detection object feature expression vector, if the classification confidence of the tracking result is higher than the classification confidence positive and negative threshold, the tracking result is marked as a new object, assigned a unique ID, and added to the tracking sequence. d Greater than τ n Q′ d Mark it as a newly appeared object, assign a unique ID, add it to the tracking sequence, and its corresponding tracking result feature expression vector is expressed as: Use classification confidence s t Greater than τ n Q′ tDirectly update the effective tracking result feature expression vector, which is expressed as:

[0176] For the tracking result output by the interactive effective tracking result feature expression vector, if the classification confidence of the tracking result is lower than the classification confidence positive and negative threshold, the tracking result is marked as disappeared, and the tracked object that is in the disappeared state for T consecutive frames is removed from the tracking sequence. In this embodiment, T=20.

[0177] In this embodiment, Figure 2 and Figure 3 The figure shows the output results of the distance mapping module and the historical information weight diagram. Figure 2 and Figure 3 In the figure, the human-shaped symbol represents the tracking result, and the raised circle represents the detection result. The gray circle represents the detection result that is suppressed due to the long distance from the tracked object, and the other colored circles represent the detection results involved in the actual interaction. The concave circle represents the historical tracking information.

[0178] It should be noted that, in the present embodiment, during the training phase, the multi-target tracking model is trained in units of video clips, the tracking results and annotation information are combined, the loss function is calculated, and the model is iteratively optimized by the parameter update method of gradient descent. Specifically, the video clip is formed by randomly selecting a frame in the video frame as the central frame, and cutting C-1 frames at equal distances before and after the central frame with a random sampling interval τ, thereby forming a video clip of length C together with the central frame. The multi-target tracking model in the video clip is tracked according to the above steps to obtain the tracking results. In the present embodiment, C is set to 5. The sampling interval τ is a random integer between 1 and 10 that obeys a uniform distribution.

[0179] In this embodiment, the annotation information in the video clip is denoted as GT, and the loss function L is expressed as:

[0180]

[0181] L(O,GT) π =(λ cls L cls +λ l1 L l1 +λ giou L giou ) π

[0182] Among them, L cls represents the classification loss function, L l1 represents the L1 loss function, L giou represents the GIOU loss function, λ cls represents the weight of the classification loss function. In this embodiment, the weight of the classification loss function is set to 2, λl1 represents the weight of the L1 loss function. In this embodiment, the weight of the L1 loss function is set to 5, λ giou represents the weight of the GIOU loss function. In this embodiment, the weight of the GIOU loss function is set to 2. π represents the mapping relationship between the predicted value and the annotation information. The predicted value of the feature expression vector of the detected object and the annotation information of the unmatched object are optimally matched by binary matching. The result is recorded as π n , π n Fixed as the mapping π from the predicted value of the feature expression vector of the next frame's effective tracking result to its annotation information o , n i It should be noted that, in this embodiment, the existing AdamW algorithm is used to optimize and update the parameters of the multi-target tracking model.

[0183] In the inference and testing phases, the above steps of the distance-guided learnable multi-target tracking method are repeated until all video frames are processed.

[0184] Therefore, the present embodiment provides a distance-guided learnable multi-target tracking method, which utilizes the position information of the object to guide the optimization of the learnable interaction module, improves the optimization efficiency of the learnable interaction module, enhances the tracking robustness of the multi-target tracking model, thereby reducing the research and development cost of the multi-target tracking model and expanding the application scenarios, and has strong practicality and wide applicability.

[0185] The above schematically describes the invention and its implementation methods, which is not restrictive. Without departing from the spirit or basic features of the invention, the invention can be implemented in other specific forms. What is shown in the accompanying drawings is only one of the implementation methods of the invention. The actual structure is not limited thereto, and any figure mark in the claims should not limit the claims involved. Therefore, if a person of ordinary skill in the art is inspired by it, without departing from the purpose of the invention, a structural method and an embodiment similar to the technical solution are designed without creativity, which should all belong to the protection scope of the present invention. In addition, the word "including" does not exclude other elements or steps, and the word "one" before the element does not exclude the inclusion of "multiple" elements. The multiple elements stated in the product claim can also be implemented by one element through software or hardware. The words first, second, etc. are used to indicate the name, and do not indicate any specific order.

Claims

1. A learnable multi-target tracking method based on distance guidance, comprising the following steps: Constructing a multi-target tracking model, the multi-target tracking model includes a target detector, a detection result encoding module, a learnable interaction module, a distance mapping module and a tracking decoding module; Obtain the video to be detected, input the video with detection into the target detector for frame-by-frame detection, obtain the detection result of each frame image, input the detection result into the detection result encoding module for encoding processing, and obtain the detection object feature expression vector; For the first frame image in the video to be detected, the detected object feature expression vector is input into the tracking decoding module to obtain the tracking result; Processing the tracking results to obtain effective tracking result feature expression vectors and historical feature expression vectors; Starting from the second frame image in the video to be detected, the detected object feature expression vector, the effective tracking result feature expression vector and the historical feature expression vector are input into the learnable interaction module for interaction to obtain a learnable interactive expression; The position information corresponding to the feature expression vector of the detected object and the feature expression vector of the effective tracking result is input into the distance mapping module to obtain the distance attention weight; Based on the distance attention weight, the detected object feature expression vector and the effective tracking result feature expression vector are guided to interact, and the detected object feature expression vector and the effective tracking result feature expression vector after interaction are obtained; the detected object feature expression vector and the effective tracking result feature expression vector after interaction are input into the tracking decoding module to obtain the tracking result.

2. A learnable multi-target tracking method based on distance guidance according to claim 1, characterized in that: The detection result encoding module includes a feature mapping submodule and an encoder mapping submodule; The feature mapping submodule is represented as: M f =Linear3(ReLU(Linear2(ReLU(Linear1(x))))) Among them, M f represents the feature mapping submodule, x represents the input information obtained from the detection result, Linear1, Linear2 and Linear3 represent different linear mapping modules, and ReLU represents the rectified linear unit; The encoder mapping submodule is represented as: M e ={Conv2d(di,d,kernel size =1), i=1, 2,..L} Among them, M e represents the encoder mapping submodule, i represents a natural number, L represents the number of selected feature layers, di represents the input channel, d represents the output channel, kernel size Represents the convolution kernel size, and Conv2d represents a two-dimensional convolution kernel.

3. The method for learning multi-target tracking based on distance guidance according to claim 2, characterized in that: The feature mapping submodule maps the detection results to obtain a detection object feature expression vector; the detection object feature expression vector is expressed as: D q =M f (In(box, Bf)) Among them, D q Represents the detected object feature expression vector, Sa represents the feature sampling of the output features of the target detector backbone network according to the location box of the detection result, box represents the location box of the detection result, and Bf represents the output features of the target detector backbone network; The output features of the target detector backbone network are input into the encoder mapping submodule to obtain the encoding basic features; the encoding basic features are expressed as: tgt=M e (Bf) Among them, tgt represents the basic encoding feature.

4. The method for learning multi-target tracking based on distance guidance according to claim 3, characterized in that: The input and output forms of the tracking and decoding module are: Among them, AD e Represents the tracking decoding module, represents the feature expression vector of object X, box1 represents the position box corresponding to object X, represents the tracking feature expression vector corresponding to object X, and box2 represents the tracking position box corresponding to object X; In the first frame image, the feature expression vector of the detected object is input into the tracking decoding module to obtain the tracking result. The tracking process is expressed as: (O d ,box′ d )=AD e (tgt,D q ,box d ) Among them, D q represents the feature expression vector of the detected object, boxd represents the location box of the detected object, d Represents the tracking feature expression vector corresponding to the detected object, box′ d Indicates the tracking position box corresponding to the detected object.

5. The method for learning multi-target tracking based on distance guidance according to claim 4, characterized in that: The processing of the tracking result to obtain the effective tracking result feature expression vector includes: setting the classification confidence positive and negative thresholds, if the classification confidence of the tracking result is higher than the classification confidence positive and negative thresholds, then the tracking result is an effective tracking result, adding the tracking sequence, marking its tracking feature expression vector as an effective tracking result feature expression vector; the effective tracking result feature expression vector is expressed as: Among them, T q Represents the effective tracking result feature expression vector, Represents the tracking feature expression vector corresponding to the i-th detection result of the input, Represents the classification confidence corresponding to the i-th detection result of the input, τ n Indicates the positive and negative thresholds of classification confidence; The processing of the tracking results to obtain the historical feature expression vector includes: recording and updating the historical feature expression vector of the effective tracking results by a sliding average method; the historical feature expression vector is expressed as: Among them, H q represents the historical feature expression vector, w represents the update weight, t represents the time, Represents the historical feature expression vector at time t, Represents the historical feature expression vector at time t-1, and * represents a multiplication operation.

6. The method for learning multi-target tracking based on distance guidance according to claim 5, characterized in that: The learnable interaction module is represented as: LIM(Q,K,V,V base ,M)=O4 O1=MHAM(Q,K,V,M) O2=Norm1(O1+V base ) O3=Linear2(Dropout(Activation(Linear1(O2)))) O4=Norm2(O2+O3) Among them, LIM represents a learnable interaction module, Q, K, V, V base Represents different input vectors, M represents the attention matrix, O1, O2, O3 and O4 represent different intermediate vectors, MHAM represents a multi-head attention module containing attention weights, Norm1 and Norm2 represent different normalization modules, Linear1 and Linear2 represent different linear mapping modules, Dropout represents a random loss module, and Activation represents a nonlinear mapping module.

7. The method for learning multi-target tracking based on distance guidance according to claim 6, characterized in that: The detecting object feature expression vector, the effective tracking result feature expression vector and the historical feature expression vector are input into the learnable interaction module for interaction to obtain the learnable interaction expression, including: Update the effective tracking result feature expression vector, and the update formula is: Among them, NTq represents the updated effective tracking result feature expression vector, LIM1 represents the first learnable interaction module, M t represents the distance attention weight between the effective tracking result feature expression vector and the detected object feature expression vector and the historical feature expression vector. concat represents the connection operation. Represents the effective tracking result feature expression vector with location information, Represents the feature expression vector of the detected object with position information, Represents the historical feature expression vector with location information; Update the feature expression vector of the detected object, and the update formula is: Among them, ND q represents the updated detection object feature expression vector, LIM2 represents the second learnable interaction module, N q represents a vector composed of random noise, M d represents the generation result of the general distance mapping module; The effective tracking result feature expression vector and the detected object feature expression vector are comprehensively updated, and the update formula is: Among them, Q t represents the effective tracking result feature expression vector after comprehensive update, Q d represents the updated detection object feature expression vector, LIM3 represents the third learnable interaction module, represents the updated effective tracking result feature expression vector with position information, represents the updated detected object feature expression vector with position information, M a Represents another result generated by the general distance mapping module.

8. The method for learning multi-target tracking based on distance guidance according to claim 7, characterized in that: The distance mapping module includes a general distance mapping module and a modulation distance mapping module; The general distance mapping module is expressed as: DM g (box1,box2)=M dist = 1-IOU(box1,box2) Among them, DM g Represents the general distance mapping module, dist represents the distance information obtained based on the intersection-and-union ratio of box1 and box2, IOU represents the intersection-and-union ratio function, e represents the natural constant, and σ represents the hyperparameter; The modulation distance mapping module is expressed as: DM m (box1,box2)=M m dist = 1-IOU(box1,box2) o ij =BiM(dist) IOU m [~o ij ]=IOU[~o ij ]*0.5 IOU m [o ij] =IOU[o ij ] Among them, DM m represents the modulation distance mapping module, M m represents the intermediate variable, o ij Represents the object pair matched by the optimal bipartite matching algorithm, BiM represents the optimal bipartite matching algorithm, IOU m represents the I / O function after modulation; The distance attention weight is calculated by the modulated intersection-over-parameter function, and the calculation formula is: dist m =1-IOU m Among them, dist m Indicates the distance after modulation.

9. The method for learning multi-target tracking based on distance guidance according to claim 8, characterized in that: The detected object feature expression vector and the effective tracking result feature expression vector after interaction are input into the tracking decoding module to obtain the tracking result, which is expressed as: (O′ M ,box′ M )=AD e (tgt,M q ,box M ) M q =concat(Q t ,Q d ) box M =concat(box t ,box d ) Among them, O′ M =concat(Q′ t , Q′ d ), which correspond to the effective tracking result feature expression vector and the detected object feature expression vector after comprehensive update, box′ M =concat(box′ t , box′ d ), which correspond to the location boxes of the effective tracking result feature expression vector and the detected object feature expression vector after comprehensive update.

10. The method for learning multi-target tracking based on distance guidance according to claim 9, characterized in that: For the tracking result output by the feature expression vector of the detected object, if the classification confidence of the tracking result is higher than the positive and negative thresholds of the classification confidence, the tracking result is marked as a new object, assigned a unique ID, and added to the tracking sequence; For the tracking result output by the effective tracking result feature expression vector after interaction, if the classification confidence of the tracking result is lower than the positive and negative thresholds of the classification confidence, the tracking result is marked as disappeared, and the tracked object that is in the disappeared state for T consecutive frames is removed from the tracking sequence.

Images