A multi-target tracker and tracking method for unmanned vehicles based on self-query

Abstract

The present invention discloses a multi-target tracker and tracking method for unmanned vehicles based on self-query. After the video frames captured by the camera are subjected to the ResNet‑50 network to extract feature information, the features are respectively input into the detection branch and the self-query branch. The detection result of the detection branch is calculated by the intersection and union ratio with the result of the self-query branch to complete the tracking. The present invention proposes a detection branch based on heat map response, which greatly simplifies the post-processing process of the detection branch and effectively improves the multi-target tracking speed of the intelligent vehicle; the self-query branch is added to improve the multi-target tracking accuracy of the intelligent vehicle; the association of the object trajectory can be completed by only using the intersection and union ratio matching, which reduces the post-processing process of the entire model and effectively improves the real-time performance of the multi-target tracking of the intelligent vehicle; the structure of the tracker designed by the present invention is modularly constructed, and the corresponding number of modules can be modified according to the different working conditions in actual use, ensuring the reusability and maintainability of the algorithm.

Description

Technical Field

[0001] The invention belongs to the technical field of intelligent automobile vision, and particularly designs a multi-target tracker and a tracking method for an unmanned vehicle based on self-query. Background Art

[0002] Smart cars are complex systems that include perception, decision-making, and control. Environmental perception is an important prerequisite for path planning and decision-making control. Camera-based multi-target tracking is one of the key aspects of environmental perception, and has an important impact on the development of ADAS systems such as vehicle driving and pedestrian prediction, as well as path planning for autonomous driving. The main task of multiple object tracking (MOT) technology is to simultaneously locate multiple objects of interest in a given video, maintain their identities, and record their trajectories. Multi-object tracking technology allows smart cars to better understand the environmental information around them and make more accurate control decisions. Existing multi-target tracking algorithms often have the disadvantage of complex post-processing, which often makes it difficult to deploy on vehicle platforms. Summary of the invention

[0003] From the current research contents of domestic and foreign scholars, how intelligent vehicles can keep tracking the objects of interest in front (usually pedestrians and cars) during driving has an important impact on the subsequent development of intelligent vehicles. The present invention aims at the problem that it is difficult for intelligent vehicles to achieve real-time tracking of objects of interest under urban road conditions, solves the shortcomings of the current algorithm in the complex post-processing process, and proposes a multi-target tracker and tracking method for unmanned vehicles based on a self-query mechanism. The technical solution of the present invention is as follows:

[0004] A multi-target tracker for unmanned vehicles based on self-query, comprising:

[0005] Feature extraction network module, detection branch module, self-query branch module;

[0006] The feature extraction network module is used to extract the features of the input image and generate outputs of four layers;

[0007] The detection branch module includes: an upsampling module, a dimension transformation module, a heat map module, a boundary module and a correction module; the detection branch module includes four input branches:

[0008] Input branch 1, which is used to fuse the features of Layer 1 of the feature extraction network module. This branch consists of a single dimensional transformation module with 256 input channels and 256 output channels;

[0009] Input branch 2, which is used to fuse the features of Layer 2 of the feature extraction network module. This branch consists of an upsampling module and a dimension transformation module. The input channel of this branch is 512 and the output channel is 256;

[0010] Input branch 3, which is used to fuse the features of Layer 3 of the feature extraction network module. This branch consists of two upsampling modules and two dimensional transformation modules. The input channel of this branch is 1024 and the output channel is 256. Among them, the output channel of the first dimensional transformation module is 512, and the output channel of the second dimensional transformation module is 256.

[0011] Input branch 4, which is used to fuse the features of Layer 4 of the feature extraction network module. This branch consists of two upsampling modules and two dimensional transformation modules. The input channel of this branch is 2048 and the output channel is 256. Among them, the output channel of the first dimensional transformation module is 1024, the output channel of the second dimensional transformation module is 512, and the output channel of the third dimensional transformation module is 256.

[0012] The detection branch module stacks the output results of input branches 1, 2, 3, and 4 and outputs them to the heat map module, the boundary module, and the correction module;

[0013] The self-query branch module includes a dimensionality transformation module, an encoder, a decoder, and a multi-layer perceptron; the dimensionality transformation module has an input channel of 2048 and an output channel of 512; the encoder is based on the Transformer structure, with an input dimension of 512 and 6 stacked layers, wherein the number of multi-head attention of the Transformer is 8; the decoder is based on the Transformer structure, with an input dimension of 512 and 6 stacked layers, wherein the number of multi-head attention of the Transformer is 8; the multi-layer perceptron output generates the final matching result.

[0014] Furthermore, the feature extraction network module adopts the ResNet-50 structure, and the outputs of its four layers are respectively 1 ,O 2 ,O 3 ,O 4 ; Among them O 1 The size is in,

[0015] Furthermore, the upsampling module includes a deformable convolution kernel, a batch normalization layer (Batchnormal), a Relu activation layer, a deconvolution kernel (ConvTranspose2d), a batch normalization layer (Batchnormal), and a Relu activation layer in sequence; a single upsampling module can double the resolution of the feature map (i.e., the width and height are each twice the original size);

[0016] Among them, the convolution kernel size of the deformable convolution kernel is 3*3, the step size is 1, the padding is 1, and the number of output channels is equal to the number of input channels; the convolution kernel size of the deconvolution kernel is 4*4, the step size is 2, the padding is 2, and the number of output channels is equal to the number of input channels.

[0017] Furthermore, the dimension transformation module includes a convolution layer (Conv2d), the convolution kernel size of the convolution layer (Conv2d) is 1*1, the step size defaults to 1, and the number of output channels defaults to 256.

[0018] Furthermore, the heat map module includes a dimensionality transformation module, a batch normalization (Batchnormal) layer, a Relu activation layer, and a convolution layer (Conv2d) with a convolution kernel size of 3*3. The output channel number of the dimensionality transformation module is 256, and the output channel number of the 3*3 convolution layer is 2.

[0019] Furthermore, the boundary module includes a dimensionality transformation module, a batch normalization (Batchnormal) layer, a Relu activation layer, and a convolution layer (Conv2d) with a convolution kernel size of 3*3. The output channel number of the dimensionality transformation module is 256, and the output channel number of the 3*3 convolution layer is 4.

[0020] Furthermore, the correction module includes a dimensionality transformation module, a batch normalization (Batchnormal) layer, a Relu activation layer, and a convolution layer (Conv2d) with a convolution kernel size of 3*3. The output channel number of the dimensionality transformation module is 256, and the output channel number of the 3*3 convolution layer is 2.

[0021] Furthermore, the multilayer perceptron includes two linear layers, a batch normalization layer, and two Relu activation layers in sequence, wherein the input channel of the first linear layer is 512 and the output channel is 256, and the input channel of the second linear layer 2 is 256 and the output channel is 4.

[0022] The present invention also proposes a tracking method for a multi-target tracker for an unmanned vehicle based on self-query, comprising:

[0023] S1: Generate features based on the feature extraction network module;

[0024] Assume that at time t, the video input image frame I∈R 1*3*H*W , H, W, 3, and 1 are the height, width, number of channels, and number of images of the input matrix respectively, which are input into the ResNet-50 feature extraction network to obtain the features of the image;

[0025] After using ResNet-50 feature extraction, the output features of layer 1 (Layer 1), layer 2 (Layer 2), layer 3 (Layer 3), and layer 4 (Layer 4) are obtained, which are denoted as O 1 ,O 2 ,O 3 ,O 4 ; Among them O 1 The size is in, Suppose the historical identity feature at the current moment is F t-1 ∈R 500*1*512 ;

[0026] S2: Generate detection results according to the detection branch module;

[0027] For the upsampling module, the calculation formula is as follows:

[0028] up opt =up(up in )=Relu(cont(Relu(bn(DCN(up in )))))

[0029] Among them, up in Indicates the input of this module, up opt Represents the output of this module, up() represents the calculation of this module, cont() represents the deconvolution calculation, Relu() represents the Relu activation function, bn() represents the batch normalization calculation, and DCN() represents the deformable convolution calculation;

[0030] O 1 Input to the input branch 1 of the detection branch, and the pre-input can be obtained through the dimension transformation module The calculation formula is as follows:

[0031] pre_do 1 =conv(O 1 )

[0032] Among them, O 1 For input value, pre_do 1Represents the output of this part, conv represents a two-dimensional convolution calculation, its input dimension is 256, the output dimension is 256, and the convolution kernel size is 1;

[0033] O 2 Input to the input branch 2 of the detection branch, after the upsampling module and the dimension transformation module, the pre-input The calculation formula is as follows:

[0034] pre_do 2 =conv(up(O 2 ))

[0035] Among them, O 2 For input value, pre_do 2 Represents the output of this part, conv represents two-dimensional convolution calculation, its input dimension is 512, output dimension is 256, convolution kernel size is 1, up() represents the calculation of upsampling module;

[0036] O 3 Input to the input branch 3 of the detection branch, and you can get the pre-input The calculation formula is as follows:

[0037] pre_do 3 =conv2(up(conv1(up(O 3 ))))

[0038] Among them, O 3 For input value, pre_do 3 Represents the output of this part, up() represents the calculation of the upsampling module, where conv1() represents the convolution calculation of the first convolution layer, that is, the convolution calculation of the first dimensional transformation module, with an input dimension of 1024, an output dimension of 512, and a convolution kernel size of 1; conv2() represents the convolution calculation of the second convolution layer, that is, the convolution calculation of the second dimensional transformation module, with an input dimension of 512, an output dimension of 256, and a convolution kernel size of 1;

[0039] O 4 Input to the input branch 4 of the detection branch, and you can get the pre-input The calculation formula is as follows:

[0040] pre_do 4 =conv3(up(conv2(up(conv1(up(O 4 ))))))

[0041] Among them, O 4 For input value, pre_do 4Represents the output of this part, up() represents the calculation of the upsampling module, where conv1() represents the convolution calculation of the first convolution layer, that is, the convolution calculation of the first dimension transformation module, whose input dimension is 2048, output dimension is 1024, and convolution kernel size is 1; conv2() represents the convolution calculation of the second convolution layer, that is, the convolution calculation of the second dimension transformation module, whose input dimension is 1024, output dimension is 512, and convolution kernel size is 1; conv3() represents the convolution calculation of the third convolution layer, that is, the convolution calculation of the third dimension transformation module, whose input dimension is 512, output dimension is 256, and convolution kernel size is 1;

[0042] Then, pre_do 1 ,pre_do 2 ,pre_do 3 ,pre_do 4 Construct the detection branch input vector along the first dimension The calculation formula is as follows:

[0043] d = cat(pre_do 1 ,pre_do 2 ,pre_do 3 ,pre_do 4 )

[0044] Among them, cat() represents the concatenation operation, and the concatenation dimension is the first dimension;

[0045] Duplicate the input vector d of the detection branch into three copies to get d 1 =d 2 =d 3 =d, change d 1 ,d 2 ,d 3 Input into the heat map module, boundary module and correction module respectively to obtain the signals of each module;

[0046] Among them, for the heat map module, there is a calculation formula:

[0047] hp=dp(d 1 )=conv2(Relu(bn(conv1(d 1 ))))

[0048] Among them, d 1 is the input value, hp represents the output signal of the module. dp represents the calculation of the heat map module; conv1() represents the convolution calculation of the first convolution layer, that is, the convolution calculation of the dimension transformation module, whose input dimension is 1024, output dimension is 256, and convolution kernel size is 1; conv2() represents the convolution calculation of the second convolution layer in the heat map module, and its convolution kernel size is 3×3; Relu() represents the Relu activation function, and bn() is the batch normalization calculation; thus, the signal output value of the heat map module can be calculated;

[0049] Among them, for the boundary module, there is a calculation formula:

[0050] bp=db(d 2 )=conv2(Relu(bn(conv1(d 2 ))))

[0051] Among them, d 2 is the input value, bp represents the output signal of the module. db represents the calculation of the boundary module; conv1() represents the convolution calculation of the first convolution layer, that is, the convolution calculation of the dimension transformation module, with an input dimension of 1024, an output dimension of 256, and a convolution kernel size of 1; conv2() represents the convolution calculation of the second convolution layer in the boundary module, with a convolution kernel size of 3×3. Relu() represents the Relu activation function, and bn() is the batch normalization calculation; thus, the signal output value of the boundary module can be calculated;

[0052] Among them, for the correction module, there is a calculation formula:

[0053] ap=da(d 3 )=conv2(Relu(bn(conv1(d 3 ))))

[0054] Among them, d 3 is the input value, ap represents the output signal of the module. da represents the calculation of the correction module; conv1() represents the convolution calculation of the first convolution layer, that is, the convolution calculation of the dimension transformation module, with an input dimension of 1024, an output dimension of 256, and a convolution kernel size of 1; conv2() represents the convolution calculation of the second convolution layer in the correction module, with a convolution kernel size of 3×3; Relu() represents the Relu activation function, and bn() is the batch normalization calculation, from which the signal output value of the correction module can be calculated;

[0055] Subsequently, the output signal hp of the heat map module is sigmoid activated to obtain the activation value hp', that is, hp' = sigmoid (hp), where sigmoid represents the sigmoid activation function; then the top k maximum values ​​of hp' can be obtained by the k-term maximum value algorithm (topk), and the k target center points, that is, k objects, can be obtained. The corresponding position index topk_index is the temporary coordinate position of the center point, recorded as H' t ;in, x i Represents the horizontal coordinate of the i-th center point, y i represents the ordinate of the i-th center point, X is the set of abscissas, and Y is the set of ordinates;

[0056] The output signal ap of the correction module is aggregated by the index topk_index to obtain the correction value of the center point, which is recorded as A = {a 1 ,a 2 …a k};

[0057] Then, the correction value A of the center point is compared with the temporary coordinate position H′ of the center point. t Add together and you can get the center point coordinates H t ;

[0058] The output signal bp of the boundary module is indexed topk index By aggregating the dimensions, we can get the offset distance between the boundary corresponding to the center point and the center point. Among them, b′ i represents the offset distance of each boundary coordinate corresponding to the i-th center point, box′ represents the set of offset distances; b′ tl Indicates the horizontal distance between the left boundary of the i-th center point and the center point, b′ tt Indicates the vertical distance between the upper boundary of the boundary of the i-th center point and the center point, b′ dr Indicates the horizontal distance between the right boundary of the i-th center point and the center point, b′ dd Represents the vertical distance between the lower boundary of the boundary of the i-th center point and the center point; the boundary coordinate Box can be calculated by the following formula:

[0059]

[0060] Finally, the center point coordinates H t The detection coordinates of the first k detection objects in the current frame can be obtained by combining the boundary coordinate Box output, which is recorded as Dec, where Dec = {H t ,Box};

[0061] S3: Generate matching results based on the self-query branch module;

[0062] O 4 and historical identity characteristics t-1 Input to the self-query branch module, and get the pre-input from the dimension transformation module The calculation formula is as follows:

[0063] pre_po 4 = flatten(conv(O 4 ),dim=2).transpose(0,1)

[0064] Among them, O 4 is the input value, pre_po 4 Indicates the output of this part; conv represents a two-dimensional convolution calculation, with an input dimension of 2048, an output dimension of 512, and a convolution kernel size of 1; flatten represents a stretching transformation, flattening the original feature map by row; transpose() means exchanging elements of two corresponding dimensions, here the elements of the 0th dimension and the 1st dimension of the feature map are exchanged;

[0065] Then, pre_po 4 Input encoder, historical identity feature F t-1 Input the decoder as the historical identity feature, and you can get the pre-output pre_po′∈R 500*1*512 ; Swap the elements of the 0th dimension and the 1st dimension of the pre-output pre_po′, and we can get pre_po″∈R 1*500*512 , and its calculation formula is as follows:

[0066]

[0067] Among them, pre_po 4 、F t-1 is the input value, pre_po″ is the output of this part, transformer() means transformer calculation, and transpose() means exchanging the elements of two corresponding dimensions;

[0068] Finally, pre_po″ is input into the multi-layer perceptron to obtain the matching result p∈R of the historical identity in the current frame. 1 *500*4 , and its calculation formula is as follows:

[0069] p=Relu(linear2(Relu(bn(linear1(pre_po″)))))

[0070] Where pre_po″ is the input value, p is the output of this part; Relu() represents the Relu activation function, bn() is the batch normalization calculation, and linear() represents the linear calculation of the fully connected layer; linear1() represents the linear calculation of the first fully connected layer, and linear2() represents the linear calculation of the second fully connected layer;

[0071] S4: target tracking;

[0072] The detection result Dec is calculated by intersection and union ratio with the matching result p. The calculation formula is:

[0073]

[0074] Where Dec[box] represents the coordinates of the detection box, ∩ represents the calculation of the intersection area of ​​two overlapping boxes, and ∪ represents the calculation of the area of ​​the union of the overlapping boxes.

[0075] The intersection-over-union ratio threshold is set to 0.8, that is, only when the intersection-over-union ratio of the detection frame and the query frame is not less than 0.8, the objects corresponding to the two frames can be regarded as the same object. At this time, the object identity corresponding to the query frame is assigned to the object corresponding to the detection frame, thereby completing the object tracking; for the query frame that failed to complete the pairing, it is saved in the storage unit, and the identity features of the previous frame that follows are input. For the detection frame that failed to complete the pairing, a new tracking identity is generated for it.

[0076] Further, the S4 also includes:

[0077] If a detection signal in a storage unit is not associated for 30 consecutive frames, it is considered that the target has disappeared, and the storage space is released; if there are 500 objects stored in the storage unit, the storage target with the earliest generation time is released.

[0078] Beneficial effects of the present invention:

[0079] (1) A detection branch based on heat map response is proposed, which greatly simplifies the post-processing process of the detection branch and effectively improves the multi-target tracking speed of intelligent vehicles.

[0080] (2) The self-query branch is added to improve the multi-target tracking accuracy of intelligent vehicles.

[0081] (3) The association of object trajectories can be completed by using only intersection-over-union matching, which greatly reduces the post-processing process of the entire model and effectively improves the real-time performance of multi-target tracking of intelligent vehicles.

[0082] (4) The tracker of the present invention is modularly constructed, and the number of modules can be modified according to the different working conditions in actual use, thereby ensuring the reusability and maintainability of the algorithm.

[0083] (5) The network model of the present invention is very practical. After training, under the conditions of GPU model: RTX2080Ti and camera pixel: 1920*1080, a relatively high tracking speed (25+fps) can be achieved, which meets engineering practical requirements. BRIEF DESCRIPTION OF THE DRAWINGS

[0084] Figure 1 . Algorithm flow chart of the present invention

[0085] Figure 2 . Upsample-block architecture diagram of the present invention

[0086] Figure 3 .Dimension transformation module architecture diagram of the present invention

[0087] Figure 4 .The heat map module architecture diagram of the present invention

[0088] Figure 5 . Detection branch architecture diagram of the present invention

[0089] Figure 6 .The multi-layer perceptron architecture diagram of the present invention

[0090] Figure 7 .Self-query branch architecture diagram of the present invention DETAILED DESCRIPTION

[0091] The present invention will be further described below in conjunction with the accompanying drawings.

[0092] like Figure 1 The overall process of the tracker implementation of the present invention is shown. After the video frame captured by the camera is extracted with feature information by the ResNet-50 network, the features are input into the detection branch and the self-query branch respectively. The detection result of the detection branch is calculated by intersection and union with the result of the self-query branch to complete the tracking.

[0093] The present invention first explains the name:

[0094] ResNet-50 is a general deep learning network built on a residual structure. It solves the common gradient explosion / vanishing problem caused by linear stacking through skip connections, and improves the network's ability to extract features.

[0095] Transformer is an attention model based on the self-attention mechanism, which enables the model to make predictions based on the correlation between the current input and the given features. This type of model can be calculated in parallel and can effectively process sequence information.

[0096] The deformable convolution kernel is a new type of convolution structure whose shape can be changed. The deformed convolution kernel can make the neural network pay more attention to the area corresponding to its task, making the recognized features better.

[0097] Batch normalization is a neural network structure model that makes the distribution of data consistent, making it easier and more stable to train deep network models.

[0098] The Relu activation layer is a common neural network activation function, which not only improves the neural network's ability to fit nonlinear models, but also effectively avoids the problem of gradient disappearance and significantly reduces the time spent on training neural networks.

[0099] The deconvolution kernel (ConvTranspose2d) is an inverse operation of the convolution layer, which can calculate the approximate input of the convolution layer from the convolution result.

[0100] The tracker of the present invention is implemented by the following process:

[0101] Step 1: Generate feature extraction network (ResNet-50)

[0102] The feature extraction network of the present invention is based on the improved general ResNet-50 structure. Assume that at time t, the video input image frame I (I∈R 3*H*W ), H, W, and 3 are the height, width, and number of channels of the input matrix respectively), and then input it into the feature extraction network to obtain the features of the image. At this time, the output feature matrix is ​​O(O∈R 2048*H′*W′ ), where H′, W′, 2048 are the height, width, and number of channels of the output feature matrix respectively.

[0103] Step 2: Design the upsample-block module

[0104] like Figure 2 As shown in the figure, the upsampling module is composed of a deformable convolution kernel (DCN), a batch normalization layer (Batchnormal), a Relu activation layer, a deconvolution kernel (ConvTranspose2d), a batch normalization layer (Batchnormal), and a Relu activation layer in sequence. A single upsampling module can double the resolution of the feature map (i.e., the width and height are twice the original size).

[0105] Among them, the convolution kernel size of the deformable convolution kernel (DCN) is 3*3, the stride is 1, the padding is 1, and the number of output channels is equal to the number of input channels. The convolution kernel size of the deconvolution kernel is 4*4, the stride is 2, the padding is 2, and the number of output channels is equal to the number of input channels.

[0106] Step 3: Design dimension transformation module

[0107] like Figure 3 As shown, the dimension transformation module consists of a convolutional layer (Conv2d).

[0108] Among them, the convolution kernel size of the convolution layer (Conv2d) defaults to 1*1, the step size defaults to 1, and the number of output channels defaults to 256.

[0109] Step 4: Design heatmap, boundary, and correction modules

[0110] like Figure 4 As shown in the figure, the heat map module consists of a dimension transformation module, a batch normalization layer, a Relu activation layer, and a convolution layer (Conv2d) with a convolution kernel size of 3*3. The output channel number of the dimension transformation module is 256. The output channel number of the 3*3 convolution layer is 2.

[0111] The architecture of the boundary module is similar to that of the heat map module. The boundary module consists of a dimension transformation module, a batch normalization layer, a Relu activation layer, and a convolution layer (Conv2d) with a convolution kernel size of 3*3. The output channels of the dimension transformation module are 256. The output channels of the 3*3 convolution layer are 4.

[0112] The architecture of the correction module is similar to that of the heat map module. The correction module consists of a dimension transformation module, a batch normalization layer, a Relu activation layer, and a convolution layer (Conv2d) with a convolution kernel size of 3*3. The output channel number of the dimension transformation module is 256. The output channel number of the 3*3 convolution layer is 2.

[0113] Step 5: Design detection branch module

[0114] like Figure 5 As shown in FIG, the detection branch module is constructed by multiple modules such as the upsampling module, the dimension transformation module, the heat map module, the boundary module and the correction module. Indicates that multiple input values ​​are stacked bit by bit and output.

[0115] First, construct input branch 1, which is used to fuse the features of Layer 1 of the feature extraction layer. This branch consists of a single dimensional transformation module with 256 input channels and 256 output channels.

[0116] After that, input branch 2 is constructed, which is used to fuse the features of Layer 2 of the feature extraction layer. This branch consists of an upsampling module and a dimension transformation module. The input channel of this branch is 512 and the output channel is 256.

[0117] After that, input branch 3 is constructed, which is used to fuse the features of Layer 3 of the feature extraction layer. This branch consists of two upsampling modules and two dimension transformation modules. The input channel of this branch is 1024 and the output channel is 256. Among them, the output channel of dimension transformation module 1 is 512, and the output channel of dimension transformation module 2 is 256.

[0118] After that, input branch 4 is constructed, which is used to fuse the features of Layer 4 of the feature extraction layer. This branch consists of two upsampling modules and three dimension transformation modules. The input channel of this branch is 2048 and the output channel is 256. Among them, the output channel of dimension transformation module 1 is 1024, the output channel of dimension transformation module 2 is 512, and the output channel of dimension transformation module 3 is 256.

[0119] Afterwards, the output results of input branches 1, 2, 3, and 4 are stacked and output to the heat map module, boundary module, and correction module.

[0120] Step 6: Design a multi-layer perceptron

[0121] like Figure 6 As shown in the figure, the multilayer perceptron consists of two linear layers, a batch normalization layer, and two Relu activation layers in series. The input channel of linear layer 1 is 512, and the output channel is 256. The input channel of linear layer 2 is 256, and the output channel is 4.

[0122] Step 7: Design the self-query branch module

[0123] like Figure 7 As shown, the self-query branch module includes a dimension transformation module, an encoder, a decoder, and a multi-layer perceptron in sequence.

[0124] First, a dimensionality transformation module is constructed with 2048 input channels and 512 output channels.

[0125] After that, an encoder is constructed based on the Transformer structure, with an input dimension of 512 and 6 stacking layers. The number of multi-head attention of the Transformer is 8.

[0126] After that, a decoder is constructed based on the Transformer structure, with an input dimension of 512 and 6 stacked layers. The number of multi-head attention of the Transformer is 8.

[0127] Afterwards, a multi-layer perceptron is constructed to generate the final matching results.

[0128] The tracker designed by the above process has the following multi-target tracking methods:

[0129] Step 8: Feature generation of feature extraction network module

[0130] Assume that at time t, the video input image frame I∈R 1*3*H*W , H, W, 3, and 1 are the height, width, number of channels, and number of images of the input matrix, respectively. In the embodiment of the present invention, H=1024 and W=1024 are input into ResNet-50 to obtain the features of the image. After using ResNet-50 feature extraction, the output features of layer 1, layer 2, layer 3, and layer 4 can be obtained, which are respectively denoted as O 1 ,O 2 ,O 3 ,O 4 . Among them O 1 The size is in, Suppose the historical identity feature at the current moment is F t-1 ∈R 500*1*512 .

[0131] Step 9: Generation of detection results of detection branch module

[0132] For the upsampling module, the calculation formula is as follows:

[0133] up opt =up(up in )=Relu(cont(Relu(bn(DCN(up in )))))

[0134] Among them, up in Indicates the input of this module, up opt Indicates the output of this module, up() indicates the calculation of this module, cont() indicates deconvolution calculation, Relu() indicates Relu activation function, bn() indicates batch normalization calculation, and DCN() indicates deformable convolution calculation.

[0135] O 1Input to the input branch 1 of the detection branch, and the pre-input can be obtained through the dimension transformation module The calculation formula is as follows:

[0136] pre_do 1 =conv(O 1 )

[0137] Among them, O 1 For input value, pre_do 1 Indicates the output of this part. conv represents a two-dimensional convolution calculation, with an input dimension of 256, an output dimension of 256, and a convolution kernel size of 1.

[0138] O 2 Input to the input branch 2 of the detection branch, after the upsampling module and the dimension transformation module, the pre-input The calculation formula is as follows:

[0139] pre_do 2 =conv(up(O 2 ))

[0140] Among them, O 2 For input value, pre_do 2 Indicates the output of this part. conv represents a two-dimensional convolution calculation, with an input dimension of 512, an output dimension of 256, and a convolution kernel size of 1. up() represents the calculation of the upsampling module.

[0141] O 3 Input to the input branch 3 of the detection branch, and you can get the pre-input The calculation formula is as follows:

[0142] pre_do 3 =conv2(up(conv1(up(O 3 ))))

[0143] Among them, O 3 For input value, pre_do 3 Indicates the output of this part. conv represents the two-dimensional convolution calculation, and up() represents the calculation of the upsampling module. Among them, conv1() represents the convolution calculation of the first convolution layer, that is, the convolution calculation of the dimension transformation module 1, whose input dimension is 1024, output dimension is 512, and convolution kernel size is 1. conv2() represents the convolution calculation of the second convolution layer, that is, the convolution calculation of the dimension transformation module 2, whose input dimension is 512, output dimension is 256, and convolution kernel size is 1.

[0144] O 4 Input to the input branch 4 of the detection branch, and you can get the pre-input The calculation formula is as follows:

[0145] pre_do 4 =conv3(up(conv2(up(conv1(up(O 4 ))))))

[0146] Among them, O 4 For input value, pre_do 4 Indicates the output of this part. conv represents the two-dimensional convolution calculation, and up() represents the calculation of the upsampling module. Among them, conv1() represents the convolution calculation of the first convolution layer, that is, the convolution calculation of the dimension transformation module 1, whose input dimension is 2048, output dimension is 1024, and convolution kernel size is 1. conv2() represents the convolution calculation of the second convolution layer, that is, the convolution calculation of the dimension transformation module 2, whose input dimension is 1024, output dimension is 512, and convolution kernel size is 1. conv3() represents the convolution calculation of the third convolution layer, that is, the convolution calculation of the dimension transformation module 3, whose input dimension is 512, output dimension is 256, and convolution kernel size is 1.

[0147] Then, pre_do 1 ,pre_do 2 ,pre_do 3 ,pre_do 4 Construct the detection branch input vector along the first dimension The calculation formula is as follows:

[0148] d = cat(pre_do 1 ,pre_do 2 ,pre_do 3 ,pre_do 4 )

[0149] Among them, cat() represents the concatenation operation, and the concatenation dimension is the first dimension.

[0150] Duplicate the input vector d of the detection branch into three copies to get d 1 =d 2 =d 3 =d, change d 1 ,d 2 ,d 3 Input into the heat map module, boundary module and correction module respectively to obtain the signals of each module.

[0151] Among them, for the heat map module, there is a calculation formula:

[0152] hp=dp(d 1 )=conv2(Relu(bn(conv1(d1 ))))

[0153] Among them, d 1 is the input value, hp represents the output value of the module. dp represents the calculation of the heat map module. Among them, conv1() represents the convolution calculation of the first convolution layer, that is, the convolution calculation of the dimension transformation module, with an input dimension of 1024, an output dimension of 256, and a convolution kernel size of 1. conv2() represents the convolution calculation of the second convolution layer in the heat map module, with a convolution kernel size of 3×3. Relu() represents the Relu activation function, and bn() is the batch normalization calculation. In this way, the signal output value of the heat map module can be calculated.

[0154] Among them, for the boundary module, there is a calculation formula:

[0155] bp=db(d 2 )=conv2(Relu(bn(conv1(d 2 ))))

[0156] Among them, d 2 is the input value, bp represents the output value of the module. db represents the calculation of the boundary module. Among them, conv1() represents the convolution calculation of the first convolution layer, that is, the convolution calculation of the dimension transformation module, with an input dimension of 1024, an output dimension of 256, and a convolution kernel size of 1. conv2() represents the convolution calculation of the second convolution layer in the boundary module, with a convolution kernel size of 3×3. Relu() represents the Relu activation function, and bn() is the batch normalization calculation. In this way, the signal output value of the boundary module can be calculated.

[0157] Among them, for the correction module, there is a calculation formula:

[0158] ap=da(d 3 )=conv2(Relu(bn(conv1(d 3 ))))

[0159] Among them, d 3 is the input value, ap represents the output value of the module. ap represents the calculation of the correction module. Among them, conv1() represents the convolution calculation of the first convolution layer, that is, the convolution calculation of the dimension transformation module, with an input dimension of 1024, an output dimension of 256, and a convolution kernel size of 1. conv2() represents the convolution calculation of the second convolution layer in the correction module, with a convolution kernel size of 3×3. Relu() represents the Relu activation function, and bn() is the batch normalization calculation. In this way, the signal output value of the correction module can be calculated.

[0160] Then, the output signal hp of the heat map module is sigmoid activated to obtain the activation value hp', that is, hp' = sigmoid (hp), where sigmoid represents the sigmoid activation function. Then, the top k maximum values ​​of hp' can be obtained by the k-term maximum algorithm (topk), and the k target center points, that is, k objects, can be obtained. The corresponding position index topk_index is the temporary coordinate position of the center point, recorded as H' t .in, x i Represents the horizontal coordinate of the i-th center point, y i Represents the ordinate of the i-th center point, X is the set of horizontal coordinates, and Y is the set of vertical coordinates.

[0161] The output signal ap of the correction module is aggregated by the index topk_index to obtain the correction value of the center point, which is recorded as A = {a 1 ,a 2 …a k}, where a i ∈{1,2,3…k} represents the network's predicted value of the correction value for the i-th center point.

[0162] Then, the correction value A of the center point is compared with the temporary coordinate position H′ of the center point. t Add together to get the final position of the center point H t .

[0163] The output signal bp of the boundary module is indexed topk index By aggregating the dimensions, we can get the offset distance between the boundary corresponding to the center point and the center point. Among them, b′ i represents the offset distance of each boundary coordinate corresponding to the i-th center point, and box′ represents the set of offset distances. b′ tl Indicates the horizontal distance between the left boundary of the i-th center point and the center point, b′ tt Indicates the vertical distance between the upper boundary of the boundary of the i-th center point and the center point, b′ dr Indicates the horizontal distance between the right boundary of the i-th center point and the center point, b′ ddIndicates the vertical distance between the lower boundary of the boundary of the i-th center point and the center point. The boundary coordinate Box can be calculated by the following formula:

[0164]

[0165] Finally, the center point coordinates H t With the boundary coordinate Box output, the detection coordinates of the first k detection objects in the current frame can be obtained, denoted as Dec. Where Dec = {H t ,Box}.

[0166] Step 10: Generation of matching results from the self-query branch module

[0167] O 4 and historical identity characteristics t-1 Input to the self-query branch module.

[0168] First, O 4 Incoming Figure 7 After processing the left part (the left part is the dimension transformation module), the pre-input The calculation formula is as follows:

[0169] pre_po 4 = flatten(conv(O 4 ),dim=2).transpose(0,1)

[0170] Among them, O 4 is the input value, pre_po 4 Indicates the output of this part. conv indicates a two-dimensional convolution calculation, with an input dimension of 2048, an output dimension of 512, and a convolution kernel size of 1. flatten indicates a stretching transformation, where the original feature map is flattened row by row. transpose() indicates exchanging elements of two corresponding dimensions, where the elements of the 0th dimension and the 1st dimension of the feature map are exchanged.

[0171] Then, pre_po 4 Input to Figure 7 The encoder of the historical identity feature F t-1 Input to Figure 7 The decoder of is used as the historical identity feature, and the pre-output pre_po′∈R 500*1*512 . By exchanging the elements of the 0th dimension and the 1st dimension of the pre-output pre_po′, we can get pre_po″∈R 1*500*512 The calculation formula is as follows:

[0172]

[0173] Among them, pre_po4 、F t-1 is the input value, and pre_po″ is the output of this part. transformer() means transformer calculation. transpose() means exchanging the elements of two corresponding dimensions.

[0174] Finally, pre_po″ is input into the multi-layer perceptron to obtain the matching result p∈R of the historical identity in the current frame. 1 *500*4 The calculation formula is as follows:

[0175] p=Relu(linear2(Relu(bn(linear1(pre_po″)))))

[0176] Among them, pre_po″ is the input value, p is the output of this part. Relu() represents the Relu activation function, bn() is the batch normalization calculation, and linear() represents the linear calculation of the fully connected layer. Among them, linear1() represents the linear calculation of the first fully connected layer, and linear2() represents the linear calculation of the second fully connected layer.

[0177] Step 11: Target tracking

[0178] The detection result Dec is calculated by intersection and union ratio with the matching result p. The calculation formula is:

[0179]

[0180] Among them, Dec[box] represents the coordinates of the detection box, ∩ represents the calculation of the intersection area of ​​two overlapping boxes, and ∪ represents the calculation of the area of ​​the union of the overlapping boxes.

[0181] The intersection-over-union ratio threshold is set to 0.8, that is, only when the intersection-over-union ratio of the detection frame and the query frame is not less than 0.8, the objects corresponding to the two frames can be regarded as the same object. At this time, the object identity corresponding to the query frame is assigned to the object corresponding to the detection frame, thereby completing the object tracking. For query frames that fail to complete the pairing, they are saved in the storage unit, and the identity features of the previous frame that follows are input. For detection frames that fail to complete the pairing, a new tracking identity is generated for them. Among them, in order to ensure the running speed, if a detection signal in the storage unit is not associated for 30 consecutive frames, it is considered that the target has disappeared, and the storage space is released. If there are 500 objects stored in the storage unit, the storage target with the earliest generation time is released.

[0182] The series of detailed descriptions listed above are only specific descriptions of feasible implementation methods of the present invention. They are not intended to limit the scope of protection of the present invention. All equivalent methods or changes that do not deviate from the technical creation of the present invention should be included in the scope of protection of the present invention.

Claims

1. A multi-target tracker for unmanned vehicles based on self-query, characterized in that: include: Feature extraction network module, detection branch module, self-query branch module; The feature extraction network module is used to extract the features of the input image and generate outputs of four layers; The detection branch module includes: an upsampling module, a dimension transformation module, a heat map module, a boundary module and a correction module; the detection branch module includes four input branches: Input branch 1, which is used to fuse the features of Layer 1 of the feature extraction network module. This branch consists of a single dimensional transformation module with 256 input channels and 256 output channels; Input branch 2, which is used to fuse the features of Layer 2 of the feature extraction network module. This branch consists of an upsampling module and a dimension transformation module. The input channel of this branch is 512 and the output channel is 256; Input branch 3, which is used to fuse the features of Layer 3 of the feature extraction network module. This branch consists of two upsampling modules and two dimensional transformation modules. The input channel of this branch is 1024 and the output channel is 256. Among them, the output channel of the first dimensional transformation module is 512, and the output channel of the second dimensional transformation module is 256. Input branch 4, which is used to fuse the features of Layer 4 of the feature extraction network module. This branch consists of two upsampling modules and two dimensional transformation modules. The input channel of this branch is 2048 and the output channel is 256. Among them, the output channel of the first dimensional transformation module is 1024, the output channel of the second dimensional transformation module is 512, and the output channel of the third dimensional transformation module is 256. The detection branch module stacks the output results of input branches 1, 2, 3, and 4 and outputs them to the heat map module, the boundary module, and the correction module; The self-query branch module includes a dimensionality transformation module, an encoder, a decoder, and a multi-layer perceptron; the dimensionality transformation module has an input channel of 2048 and an output channel of 512; the encoder is based on the Transformer structure, with an input dimension of 512 and 6 stacked layers, wherein the number of multi-head attention of the Transformer is 8; the decoder is based on the Transformer structure, with an input dimension of 512 and 6 stacked layers, wherein the number of multi-head attention of the Transformer is 8; the multi-layer perceptron output generates the final matching result.

2. The multi-target tracker for unmanned vehicles based on self-query according to claim 1, characterized in that: The feature extraction network module adopts the ResNet-50 structure, and the outputs of its four layers are O1, O2, O3, and O4 respectively; the size of O1 is in, H and W represent the height and width of the input matrix.

3. The multi-target tracker for unmanned vehicles based on self-query according to claim 1, characterized in that: The upsampling module includes a deformable convolution kernel, a batch normalization layer, a Relu activation layer, a deconvolution kernel, a batch normalization layer, and a Relu activation layer in sequence; a single upsampling module can double the resolution of the feature map; Among them, the convolution kernel size of the deformable convolution kernel is 3*3, the step size is 1, the padding is 1, and the number of output channels is equal to the number of input channels; the convolution kernel size of the deconvolution kernel is 4*4, the step size is 2, the padding is 2, and the number of output channels is equal to the number of input channels.

4. The multi-target tracker for unmanned vehicles based on self-query according to claim 1, characterized in that: The dimension transformation module includes a convolution layer whose convolution kernel size is 1*1, the step size is 1 by default, and the number of output channels is 256 by default.

5. The self-query based multi-target tracker for unmanned vehicles according to claim 4, characterized in that: The heat map module includes a dimensionality transformation module, a batch normalization layer, a Relu activation layer, and a convolution layer with a convolution kernel size of 3*3. The output channel number of the dimensionality transformation module is 256, and the output channel number of the 3*3 convolution layer is 2.

6. The multi-target tracker for unmanned vehicles based on self-query according to claim 4, characterized in that: The boundary module includes a dimensionality transformation module, a batch normalization layer, a Relu activation layer, and a convolution layer with a convolution kernel size of 3*3 in sequence. The number of output channels of the dimensionality transformation module is 256, and the number of output channels of the 3*3 convolution layer is 4.

7. The multi-target tracker for unmanned vehicles based on self-query according to claim 4, characterized in that: The correction module includes a dimensionality transformation module, a batch normalization layer, a Relu activation layer, and a convolution layer with a convolution kernel size of 3*3 in sequence. The output channel number of the dimensionality transformation module is 256, and the output channel number of the 3*3 convolution layer is 2.

8. The self-query based multi-target tracker for unmanned vehicles according to claim 1, characterized in that: The multilayer perceptron includes two linear layers, a batch normalization layer, and two Relu activation layers in sequence, wherein the input channel of the first linear layer is 512 and the output channel is 256, and the input channel of the second linear layer 2 is 256 and the output channel is 4.

9. A tracking method for a multi-target tracker for an unmanned vehicle based on self-query according to any one of claims 1 to 8, characterized in that: include: S1: Generate features based on the feature extraction network module; Assume that at time t, the video input image frame I∈R 1*3*H*W , H, W, 3, and 1 are the height, width, number of channels, and number of images of the input matrix respectively, which are input into the ResNet-50 feature extraction network to obtain the features of the image; After using ResNet-50 feature extraction, the output features of layer 1, layer 2, layer 3, and layer 4 are obtained, which are recorded as O1, O2, O3, and O4 respectively; the size of O1 is in, Suppose the historical identity feature at the current moment is F t-1 ∈R 500*1*512 ; S2: Generate detection results according to the detection branch module; For the upsampling module, the calculation formula is as follows: up opt =up(up in )=Relu(cont(Relu(bn(DCN(up in ))))) Among them, up in Indicates the input of this module, up opt Represents the output of this module, up() represents the calculation of this module, cont() represents the deconvolution calculation, Relu() represents the Relu activation function, bn() represents the batch normalization calculation, and DCN() represents the deformable convolution calculation; Input O1 into the input branch 1 of the detection branch, and obtain the pre-input through the dimension transformation module The calculation formula is as follows: pre_do1 = conv(O1) Among them, O1 is the input value, pre_do1 represents the output of this part, conv represents two-dimensional convolution calculation, its input dimension is 256, output dimension is 256, and convolution kernel size is 1; Input O2 to the input branch 2 of the detection branch, and obtain the pre-input through the upsampling module and the dimension transformation module The calculation formula is as follows: pre_do2 = conv(up(O2)) Among them, O2 is the input value, pre_do2 represents the output of this part, conv represents the two-dimensional convolution calculation, its input dimension is 512, the output dimension is 256, the convolution kernel size is 1, and up() represents the calculation of the upsampling module; Input O3 into the input branch 3 of the detection branch to obtain the pre-input The calculation formula is as follows: pre_do3=conv2(up(conv1(up(O3)))) Among them, O3 is the input value, pre_do3 represents the output of this part, up() represents the calculation of the upsampling module, where conv1() represents the convolution calculation of the first convolution layer, that is, the convolution calculation of the first dimension transformation module, whose input dimension is 1024, output dimension is 512, and convolution kernel size is 1; conv2() represents the convolution calculation of the second convolution layer, that is, the convolution calculation of the second dimension transformation module, whose input dimension is 512, output dimension is 256, and convolution kernel size is 1; Input O4 into the input branch 4 of the detection branch to obtain the pre-input The calculation formula is as follows: pre_do4=conv3(up(conv2(up(conv1(up(O4))))))) Among them, O4 is the input value, pre_do4 represents the output of this part, up() represents the calculation of the upsampling module, among which conv1() represents the convolution calculation of the first convolution layer, that is, the convolution calculation of the first dimension transformation module, whose input dimension is 2048, output dimension is 1024, and convolution kernel size is 1; conv2() represents the convolution calculation of the second convolution layer, that is, the convolution calculation of the second dimension transformation module, whose input dimension is 1024, output dimension is 512, and convolution kernel size is 1; conv3() represents the convolution calculation of the third convolution layer, that is, the convolution calculation of the third dimension transformation module, whose input dimension is 512, output dimension is 256, and convolution kernel size is 1; Then, the pre-inputs pre_do1, pre_do2, pre_do3, and pre_do4 are concatenated along the first dimension to form the detection branch input vector The calculation formula is as follows: do=cat(pre_do1, pre_do2, pre_do3, pre_do4) Among them, cat() represents the concatenation operation, and the concatenation dimension is the first dimension; The input value of the detection branch input vector d is copied into three copies to obtain d1=d2=d3=d, and d1, d2, and d3 are respectively input to the heat map module, the boundary module, and the correction module to obtain the signals of each module; Among them, for the heat map module, there is a calculation formula: hp=dp(d1)=conv2(Relu(bn(conv1(d1)))) Where d1 is the input value, hp represents the output signal of the module. dp represents the calculation of the heat map module; conv1() represents the convolution calculation of the first convolution layer, that is, the convolution calculation of the dimension transformation module, whose input dimension is 1024, output dimension is 256, and convolution kernel size is 1; conv2() represents the convolution calculation of the second convolution layer in the heat map module, and its convolution kernel size is 3×3; Relu() represents the Relu activation function, and bn() is the batch normalization calculation; thus, the signal output value of the heat map module is calculated; Among them, for the boundary module, there is a calculation formula: bp=db(d2)=conv2(Relu(bn(conv1(d2)))) Among them, d2 is the input value, bp represents the output signal of the module. db represents the calculation of the boundary module; conv1() represents the convolution calculation of the first convolution layer, that is, the convolution calculation of the dimension transformation module, whose input dimension is 1024, output dimension is 256, and convolution kernel size is 1; conv2() represents the convolution calculation of the second convolution layer in the boundary module, and its convolution kernel size is 3×3; Relu() represents the Relu activation function, and bn() is the batch normalization calculation; thus, the signal output value of the boundary module is calculated; Among them, for the correction module, there is a calculation formula: ap=da(d3)=conv2(Relu(bn(conv1(d3)))) Among them, d3 is the input value, ap represents the output signal of the module. da represents the calculation of the correction module; conv1() represents the convolution calculation of the first convolution layer, that is, the convolution calculation of the dimension transformation module, with an input dimension of 1024, an output dimension of 256, and a convolution kernel size of 1; conv2() represents the convolution calculation of the second convolution layer in the correction module, with a convolution kernel size of 3×3; Relu() represents the Relu activation function, and bn() is the batch normalization calculation, thereby calculating the signal output value of the correction module; Subsequently, the output signal hp of the heat map module is sigmoid activated to obtain the activation value hp', that is, hp' = sigmoid (hp), where sigmoid represents the sigmoid activation function; then the first k maximum values ​​of hp' are obtained by the k-term maximum value algorithm to obtain k target center points, that is, k objects, and their corresponding position index topk_index is the temporary coordinate position of the center point, denoted as H' t ;in, x i Represents the horizontal coordinate of the i-th center point, y i represents the ordinate of the i-th center point, X is the set of abscissas, and Y is the set of ordinates; The output signal ap of the correction module is aggregated by the index topk_index to obtain the correction value of the center point, which is recorded as A = {a1, a2…a k }; where a i ∈{1, 2, 3...k} represents the predicted value of the network for the correction value of the i-th center point; Then, the correction value A of the center point is compared with the temporary coordinate position H′ of the center point. t Add together to get the center point coordinates H t ; The output signal bp of the boundary module is indexed topk index Aggregate the dimensions to get the offset distance between the boundary corresponding to the center point and the center point Among them, b′ i represents the offset distance of each boundary coordinate corresponding to the i-th center point, box′ represents the set of offset distances; b′ tl Indicates the horizontal distance between the left boundary of the i-th center point and the center point, b′ tt Indicates the vertical distance between the upper boundary of the boundary of the i-th center point and the center point, b′ dr Indicates the horizontal distance between the right boundary of the i-th center point and the center point, b′ dd Represents the vertical distance between the lower boundary of the boundary of the i-th center point and the center point; the boundary coordinate Box is calculated by the following formula: Finally, the center point coordinates H t The detection coordinates of the first k detection objects in the current frame are obtained by combining with the boundary coordinate Box output, which is recorded as Dec, where Dec = {H t ,Box}; S3: Generate matching results based on the self-query branch module; O4 and historical identity feature F t-1 Input to the self-query branch module, and get the pre-input from the dimension transformation module The calculation formula is as follows: pre_po4=flatten(conv(O4),dim=2).transpose(0,1) Among them, O4 is the input value, pre_po4 represents the output of this part; conv represents two-dimensional convolution calculation, its input dimension is 2048, output dimension is 512, and convolution kernel size is 1; flatten represents stretching transformation, flattening the original feature map by row; transpose() represents exchanging elements of two corresponding dimensions, here the elements of the 0th dimension and the 1st dimension of the feature map are exchanged; Then, pre_po4 is input into the encoder, and the historical identity feature F t-1 Input the decoder as the historical identity feature, that is, the pre-output pre_po′∈R 500*1*512 ; Swap the elements of the 0th dimension and the 1st dimension of the pre-output pre_po′ to obtain pre_po″∈R 1*500*512 , and its calculation formula is as follows: Among them, pre_po4, F t-1 is the input value, pre_po″ is the output of this part, transformer() means transformer calculation, and transpose() means exchanging the elements of two corresponding dimensions; Finally, pre_po″ is input into the multi-layer perceptron to obtain the matching result p∈R of the historical identity in the current frame 1*500*4 , and its calculation formula is as follows: p=Relu(linear2(Relu(bn(linear1(pre_po″))))) Where pre_po″ is the input value, p is the output of this part; Relu() represents the Relu activation function, bn() is the batch normalization calculation, and linear() represents the linear calculation of the fully connected layer; linear1() represents the linear calculation of the first fully connected layer, and linear2() represents the linear calculation of the second fully connected layer; S4: target tracking; The detection result Dec is calculated by intersection and union ratio with the matching result p. The calculation formula is: Where Dec[box] represents the coordinates of the detection box, ∩ represents the calculation of the intersection area of ​​two overlapping boxes, and ∪ represents the calculation of the area of ​​the union of the overlapping boxes. The intersection-over-union ratio threshold is set to 0.8, that is, only when the intersection-over-union ratio of the detection frame and the query frame is not less than 0.8, the objects corresponding to the two frames are regarded as the same object. At this time, the object identity corresponding to the query frame is assigned to the object corresponding to the detection frame, thereby completing the object tracking; for the query frame that failed to complete the pairing, it is saved in the storage unit, and the identity features of the previous frame that followed are input. For the detection frame that failed to complete the pairing, a new tracking identity is generated for it.

10. The tracking method according to claim 9, characterized in that: The S4 further comprises: If a detection signal in a storage unit is not associated for 30 consecutive frames, it is considered that the target has disappeared, and the storage space is released; if there are 500 objects stored in the storage unit, the storage target with the earliest generation time is released.

Images