A multi-target tracking method based on three-dimensional depth information decoupling

Abstract

The application discloses a multi-target tracking method based on three-dimensional depth information decoupling, first constructs a multi-target tracking model based on three-dimensional depth information decoupling and carries out training, then clusters two-dimensional target detection frames to complete detection of dense occlusion, then obtains target three-dimensional depth information through a self-supervision depth estimation method, decouples dense occlusion by using the depth information, and completes multi-target tracking based on spatial information. The method of the application does not need to solve the problem that target appearance feature information extraction is inaccurate due to dense occlusion, proposes MOT based on depth information, only uses IOU matching to realize association, solves the occlusion problem through the advantage of depth information, does not need appearance features, proposes an algorithm based on three-dimensional depth estimation, realizes hierarchical decoupling of dense occlusion targets based on the difference of depth information, carries out data association in different levels, different levels of targets cannot influence each other, and a robust multi-target tracking effect is obtained.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of computer vision tasks, and particularly relates to a multi-target tracking method based on three-dimensional depth information decoupling. BACKGROUND

[0002] Multiple object tracking (MOT) is a key computer vision task that aims to track the motion trajectories of multiple targets (such as pedestrians) in video sequences in real time. With the rapid development of deep learning and artificial intelligence technologies, MOT is increasingly widely applied in various fields. For example, in intelligent monitoring systems, multi-pedestrian target tracking technology is widely used in public security monitoring. By deploying cameras at key locations in the city, the motion trajectories of crowds are tracked in real time using MOT technology, and security personnel can promptly detect abnormal behavior, such as fighting, suspicious package left, etc., thereby improving the level of urban security management. In the field of autonomous driving, autonomous vehicles need to identify and track multiple dynamic targets such as pedestrians and bicycles in complex road environments to avoid collisions and ensure the safety of passengers and pedestrians. Through deep learning algorithms, MOT systems can accurately locate and predict the motion trajectories of pedestrians in real-time traffic scenes, thereby making reasonable driving decisions. Intelligent traffic management is another important application field of multi-pedestrian target tracking technology. In crowded traffic environments, MOT technology can help traffic management departments monitor road conditions in real time, identify and track the motion of pedestrians, optimize traffic signal control strategies, alleviate traffic congestion, and improve traffic efficiency. In addition, this task is also the basis for advanced computer vision tasks such as pose estimation, behavior recognition, behavior analysis, and video analysis. In summary, multi-target tracking technology has a wide and far-reaching application background in the field of artificial intelligence. With the continuous progress of computer vision and deep learning technologies, MOT technology will further promote the intelligent development of various industries and bring more convenience and safety to society.

[0003] The main task of multi-target tracking is to output the motion trajectory of all targets from a given video and maintain the identity (ID) of each target. Among them, the tracked target can be a pedestrian, a vehicle or other objects. The current mainstream tracking method divides the MOT task into three sub-tasks of target detection, feature extraction and data association. This idea has been well developed. However, due to the challenges of occlusion and similar object interference in the actual tracking process, maintaining robust tracking is still a research difficulty. In order to meet the requirements of accurate, robust and real-time tracking of multiple targets in complex scenes, further research and improvement of the MOT algorithm are needed. However, robust tracking in complex scenes is still a research difficulty, mainly in the following three aspects: ① Frequent occlusion in the tracking process makes it difficult to accurately locate the target ② Different targets may have high appearance similarity, increasing the difficulty of maintaining target ID ③ Target interaction may cause tracking box drift.

[0004] The existing multi-target tracking algorithm usually relies on the joint action of the detection module, the appearance feature module and the position prediction module. The existing basic detection algorithm can extract the deep feature information of the target to complete accurate target detection and obtain accurate detection results. However, for the appearance feature extraction module, on the one hand, the feature of the pedestrian target itself is not high in distinguishability, and on the other hand, under the condition of dense occlusion, the appearance feature information extracted by the detection box is inaccurate, so under the condition of occlusion and target overlap, frequent ID switch occurs. SUMMARY

[0005] To solve the above technical problems, the present application provides a multi-target tracking method based on three-dimensional depth information decoupling, which solves the problem of inaccurate IOU (Intersection over Union) matching caused by dense occlusion and realizes multi-target tracking in complex scenes of multiple pedestrian targets.

[0006] The technical scheme adopted by the present application is as follows: a multi-target tracking method based on three-dimensional depth information decoupling, the specific steps are as follows:

[0007] S1, a multi-target tracking model based on three-dimensional depth information decoupling is constructed;

[0008] The multi-target tracking model comprises: 1 feature extraction network based on attention mechanism, 1 camera pose estimation network, 3 depth estimation prediction heads and 3 target detection prediction heads.

[0009] S2, the multi-target tracking model constructed in step S1 is trained;

[0010] S3, based on the trained multi-target tracking model obtained in step S2, inputting an image at a current t moment to obtain target detection data of the image at the t moment and three-dimensional depth information at the t moment;

[0011] Each detected target is represented by a bounding box without ID information, and the box without ID is a detection box; the target detection prediction head outputs multiple pedestrian target boxes for each input image.

[0012] S4, based on step S3, clustering the multiple detection boxes at the t moment and the track boxes at the t-1 moment by using the improved Mean-shift clustering algorithm, identifying dense occlusion, and each class is a dense occlusion;

[0013] When the number of targets in each clustering result exceeds 5, it is considered as dense occlusion; for the "dense occlusion" with less than 5, it is considered as a target with no serious or even no occlusion, i.e. a non-dense occlusion target.

[0014] S5, based on step S4, realizing data association of non-dense occlusion targets and dense occlusion targets;

[0015] S6, processing the unmatched detection box targets and track box targets in step S5, frame-by-frame tracking the targets to obtain target trajectories, and realizing multi-target tracking based on depth information decoupling.

[0016] Further, the step S2 is specifically as follows:

[0017] S21, extracting depth feature information by using a feature extraction network based on an attention mechanism;

[0018] The feature extraction network is a U-shaped backbone network, which includes a Conv-stem convolution module, an average pooling layer, a stage 2 convolution layer, a stage 3 convolution layer, a stage 4 convolution layer, and three up-sampling and convolution modules.

[0019] The convolution kernel size of the stage 1-4 convolution layer is 3x3, and the step size is 2; the stage 2 convolution layer and the stage 3 convolution layer each include a down-sampling layer, three consecutive dilated convolution modules CDC, and a local-global feature interaction module LGFI; the stage 4 convolution layer includes a down-sampling layer, six consecutive dilated convolution modules CDC, and a local-global feature interaction module LGFI. The up-sampling and convolution module includes an up-sampling layer and a convolution layer.

[0020] The feature extraction network is used to extract depth feature information, i.e. the encoder of like-UNet is used to extract depth feature information, which is divided into four stages and specifically as follows:

[0021] 1) First stage:

[0022] The original image of size H×W×3 is used as the input feature and input into the Conv-stem convolutional module for downsampling to obtain a feature of size H / 2×W / 2×C1. tmp1 The output feature map.

[0023] Where H represents the height of the image, W represents the width of the image, and C represents the number of channels of the image; the Conv-stem module consists of three convolutional layers. The first layer uses a 3×3 kernel with a stride of 2 to achieve convolution, which achieves the effect of downsampling. The latter two layers use a 3×3 kernel with a stride of 1 to achieve convolution, which achieves the effect of local feature extraction.

[0024] 2) Second stage:

[0025] The original image is processed by an average pooling layer to obtain a feature map of size H / 2×W / 2×3, which is compared with the feature map of size H / 2×W / 2×C1 obtained in the first stage. tmp1 The feature maps are concatenated to obtain a feature map of size H / 2×W / 2×C1, which is then used as the input feature map. This input feature map is then fed into a convolutional layer with a kernel size of 3×3 and a stride of 2 for downsampling, resulting in a feature map of size H / 4×W / 4×C2. tmp1 The downsampled features are then processed through three consecutively dilated convolutional modules (CDC) and one local-global feature interaction module (LGFI) to obtain a sample of size H / 4×W / 4×C2. tmp2 The output feature map.

[0026] In this stage, the window size of the average pooling layer is set to H / 2×H / 2, and the stride is set to H / 2. That is, the original image with size H×W×3 will output a size of H / 2×W / 2×3 after passing through this average pooling layer.

[0027] 3) Third stage:

[0028] The size obtained in the second stage is H / 4×W / 4×C2. tmp1 The initial downsampling feature map and the final output size of the stage are H / 4×W / 4×C2. tmp2 The feature map of the original image and the feature map of the original image after average pooling are concatenated to obtain a feature map of size H / 4×W / 4×3 as the input feature. This feature map is then input into a convolutional layer with a kernel size of 3×3 and a stride of 2 for downsampling to obtain a feature map of size H / 8×W / 8×C3. tmp1 The downsampled feature map is then processed through 3 CDC modules and 1 LGFI module to obtain a size of H / 8×W / 8×C3. tmp2 The output feature map.

[0029] Wherein, the average pooling window size of this stage is set to H / 4xH / 4, and the step size is set to H / 4, and the output size of the average pooling layer is H / 4xW / 4x3.

[0030] 4) The fourth stage:

[0031] The preliminary down-sampling feature map obtained in the third stage has a size of H / 8xW / 8xC3 tmp1 , the size of the feature map output at the end of the stage is H / 8xW / 8xC3 tmp2 , and the size of the feature map obtained after the original image is processed by the average pooling layer is H / 8xW / 8x3. The feature maps are spliced to obtain a feature map with a size of H / 8xW / 8xC3 as an input feature, and then input to a convolution layer with a kernel size of 3x3 and a step size of 2 for down-sampling, and the sampling result is input to 6 CDC modules and 1 LGFI module to obtain an output feature map with a size of H / 16xW / 16xC4.

[0032] Wherein, the average pooling window size of this stage is set to H / 8xH / 8, and the step size is set to H / 8, and the output size of the average pooling layer is H / 8xW / 8x3.

[0033] S22, through the depth estimation prediction head, the multi-scale feature information extracted in step S21 is fused to obtain depth estimation information of different scales, and then the camera pose estimation network Pose Net is used to complete camera pose estimation, and the multi-scale depth estimation information is fused and reconstructed to obtain the final three-dimensional depth information, so as to realize three-dimensional depth feature information reconstruction;

[0034] S221, through the feature fusion of the different stage features obtained in step S21, three different scale depth estimation information, i.e. multi-scale depth estimation information, is obtained through the depth estimation prediction head.

[0035] (1) The feature with a size of H / 16xW / 16xC4 obtained in the fourth stage is input into the first up-sampling and convolution module for up-sampling and residual connection with the feature with a size of H / 8xW / 8xC3 tmp2 obtained in the third stage, and then a layer of convolution layer is used to obtain a fusion feature with a size of H / 8xW / 8xC3 ~ , which is input into the depth estimation prediction head to finally obtain depth estimation information with a size of H / 4xW / 4x1.

[0036] Wherein, the depth estimation prediction head includes a layer of convolution layer Conv, an up-sampling layer and an activation layer Sigmoid.

[0037] (2) The feature with a size of H / 8xW / 8xC3 ~The fused feature of H / 4xW / 4xC2 is input into a second upsampling and convolution module to be upsampled and residual connected to the second stage with a size of H / 4xW / 4xC2. tmp2 The fused feature of H / 4xW / 4xC2 is input into a second upsampling and convolution module to be upsampled and residual connected to the second stage with a size of H / 4xW / 4xC2. ~ The fused feature of H / 4xW / 4xC2 is input into a second upsampling and convolution module to be upsampled and residual connected to the second stage with a size of H / 4xW / 4xC2.

[0038] (3) The fused feature of H / 4xW / 4xC2 is input into a third upsampling and convolution module to be upsampled, and then directly through a convolution layer to obtain a feature H / 2xW / 2xC1. ~ ~ The fused feature of H / 4xW / 4xC2 is input into a second upsampling and convolution module to be upsampled and residual connected to the second stage with a size of H / 4xW / 4xC2.

[0039] S222, obtain camera pose estimation information of adjacent two frames, that is, complete camera pose estimation through a camera pose estimation network Pose Net;

[0040] For monocular camera training, the camera pose estimation network is formed by ResNet18, the input is a pair of color images or six channels, and a four-layer convolution pose decoder is used to estimate the relative pose of six degrees of freedom between adjacent two frames of images. At the same time, horizontal inversion and random brightness, contrast, saturation and hue jitter with a range of ±0.2, ±0.2, ±0.2 and ±0.1 are trained.

[0041] The six degrees of freedom are the translational degrees of freedom of the x-axis, y-axis and z-axis in the three-dimensional coordinate system and the rotational degrees of freedom, and the pose of the camera is represented by a 4x4 homogeneous transformation matrix T, that is, a 3x3 rotation matrix R and a 3x1 translation vector z.

[0042] S223, correct the original rough three-dimensional depth information obtained in step S21, that is, based on step S222, fuse and reconstruct the multi-scale depth estimation information obtained in step S221 to obtain the final three-dimensional depth information, and realize three-dimensional depth feature information reconstruction through projection transformation and linear interpolation.

[0043] Firstly, projection transformation is performed, that is, each pixel point in the depth map is projected into the world coordinate system, and then projected into the image plane of the target view according to the camera pose transformation matrix.

[0044] ​For each pixel point (u, v), a pixel coordinate grid is created, the pixel coordinates are converted to normalized coordinates, and then the normalized coordinates are multiplied by the depth estimation information d of the corresponding pixel point to obtain a 3D point in the world coordinate system. Then, the 3D point is projected from the current view to the target view using the pose transformation matrix T of the camera, and the 3D point is projected back to the image plane using the camera intrinsic matrix K of the target view. The projection coordinates are normalized to the image plane, i.e., from three-dimensional coordinates (x, y, z) to two-dimensional coordinates (x / z, y / z, 1).

[0045] Then, bilinear interpolation is performed to map the pixel value of the projection point to the target image in the image plane of the target view.

[0046] The projection coordinates are normalized to the range [−1, 1] of the image plane, and the pixel value corresponding to the normalized projection coordinates is interpolated from the image of the target view using the bilinear interpolation method. Finally, the depth estimation information corrected by the camera pose estimation is obtained, i.e., the final three-dimensional depth estimation map, and the calculation expression is as follows:

[0047]

[0048] wherein, , and represent the horizontal and vertical decimal parts of the normalized projection coordinates, represent the nearest four pixel values.

[0049] The depth estimation prediction head learns the modeling of the target, i.e., minimizes the image reconstruction loss between the target image and the synthesized target image , and the edge-aware smoothing loss constrained on the predicted depth map . .

[0050] The loss function expression of the depth estimation prediction head is as follows:

[0051]

[0052]

[0053]

[0054]

[0055] wherein, represents the image reconstruction loss, represents the target image, represents the synthesized target image, represents a fixed threshold value 0.85, represents the inter-pixel structural similarity index, Indicates the distance between pixels Similarity property, used to characterize the similarity of pixel-level mapping relationships. This represents the minimum photometric loss in the source image when processing out-of-view pixels and occluded objects. This indicates the previous or next frame of the source / target image. This represents the weighted edge-aware smoothing loss, which aims to ensure that the predicted depth map remains smooth in edge regions while maintaining a certain degree of continuity in non-edge regions. This represents the average normalized depth. This represents the weight information used for the weighted edge-aware smoothing loss. This represents the sum of loss functions that include features at multiple scales.

[0056] S23. Based on the features of different scales obtained from the fusion of the second, third, and fourth stages in step S21, i.e., the size obtained by the input upsampling and convolution module fusion in step S23 is H×W×C1 ~ H / 2×W / 2×C2 ~ H / 4×W / 4×C3 ~ The fusion features are used to feed the features of each size into an object detection prediction head to achieve the classification subtask and the bounding box regression subtask;

[0057] The target detection prediction head includes: a classification prediction head and a bounding box regression prediction head.

[0058] The classification prediction head includes: two convolutional layers with C 3×3 kernels, one ReLU activation layer, one convolutional layer with 2A 3×3 kernels, and a Sigmoid activation function. The bounding box regression prediction head includes: two convolutional layers with C 3×3 kernels, one ReLU activation layer, one convolutional layer with 4A 3×3 kernels, and a Sigmoid activation function. The classification prediction head predicts the probability of the A anchor boxes at each pixel location belonging to the following categories: background target and foreground pedestrian target. The bounding box regression prediction head predicts the (center x-coordinate, center y-coordinate, width w, height) of each anchor box. The offset of ).

[0059] The classification prediction head first uses two convolutional layers with C 3×3 kernels each. The output features are then fed into a ReLU activation layer, and the output is passed through a convolutional layer with 2A 3×3 kernels. Finally, the output is passed through a Sigmoid activation function, producing 2A binary predictions for each spatial location, indicating whether the target is in the background or foreground pedestrian.

[0060] Wherein, A is set to 9, and the size of C is determined according to the channel number of the actual input feature.

[0061] Similarly, the bounding regression prediction head outputs 4A linear results at each spatial position. For A anchor boxes at each spatial position, the 4 outputs represent the relative offset between the predicted anchor box and the real box.

[0062] The loss function of the target detection prediction head That is, the focal Loss for cross-entropy And L1 Loss for regression task Complete the training of the prediction head, the expression is as follows:

[0063]

[0064]

[0065]

[0066] Wherein, Indicates the predicted probability, when the sample is a positive sample , is , Indicates the direct output probability of the prediction head, Indicates the balance factor, which is used to balance the influence of positive and negative samples, For difficult sample adjustment, When the weight of simple samples is reduced, the attention to difficult samples is enhanced. Indicates the true value, and Indicates the predicted value, and N indicates the number of samples. , Indicate the weight information of the two loss functions respectively, and are set to 0.45 and 0.55 respectively.

[0067] Further, the step S4 is specifically as follows:

[0068] The detection frame obtained from the current t time image is subjected to OCR Mean-shift clustering, and the dense occlusion and non-dense occlusion in the detection frame are classified, and the trajectory frame of the t-1 time image is continuously subjected to OCR Mean-shift clustering, and the dense occlusion and non-dense occlusion in the trajectory frame are classified.

[0069] Wherein, the trajectory information at the t-1 time is known data, that is, each target in the t-1 time image is represented by a bounding box and a unique ID number, the frame with ID is a trajectory frame, and the three-dimensional depth information of the t-1 time image is known.

[0070] For all the target bounding boxes detected at each moment, the center point coordinates are taken as the sample points to be clustered to calculate the expression as follows:

[0071]

[0072]

[0073]

[0074] wherein, , represents a calculation factor for measuring the characteristics of whether the target is occluded and the connection between the targets in the occlusion group; represents a certain sample point to be clustered, represents the Euclidean distance between the sample points the nearest sample point, represents the class probability directly output by the classification prediction head, represents the sample point and the IOU value between the bboxes corresponding to the sample points, represents the area of the bbox corresponding to the sample point, represents the improved sample point weight, the weight factor of the two factors, which is fixedly set to , and represents a Gaussian kernel function, the bandwidth controls the range of the kernel function, is used to complete the mean-shift clustering and update the drift point position.

[0075] Further, the step S5 is specifically as follows:

[0076] S51, data association for non-dense occluded targets is realized;

[0077] Based on the non-dense occluded targets obtained in step S4, the basic association method of SORT is directly used to calculate the IOU similarity between the detection frame at t moment and the track frame at t-1 moment, and the Hungarian algorithm is used to realize the association of the non-dense occluded targets depending on the similarity matrix, and the IOU threshold of the association is set to 0.5. When the IOU similarity is less than 0.5, it is considered that no association occurs.

[0078] S52, data association for dense occluded targets is realized;

[0079] Based on the multiple dense occlusions obtained in step S4, the dense occlusion centroids of the detection frame and the trajectory frame are calculated, and the matching of the dense occlusions is completed by the Euclidean distance of the multiple centroids of the two frames. That is, the data association of the occlusions in the detection frame and the trajectory frame is achieved. The association strategy is based on the three-dimensional depth information of the t time obtained in step S3.

[0080] The three-dimensional depth information of the targets in the dense occlusions is used to achieve the hierarchical uniform division of the depth in the dense occlusions. That is, according to the depth information of each target in the dense occlusions, the deepest depth and the shallowest depth are found. The four-level association regions are uniformly divided between the deepest depth and the shallowest depth. And for the two dense occlusions of the detection frame at t time and the trajectory frame at t-1 time, only the targets of the same level can achieve data association by calculating the IOU.

[0081] Then, according to the hierarchical association strategy, the targets are associated layer by layer. The unassociated targets will be sequentially extended to the next layer for the same IOU association, until the four-level cascade association is completed, and the identity ID is assigned to the matched targets.

[0082] Further, the step S6 is specifically as follows:

[0083] For the targets that have not completed the association in step S5, the last data association is performed. A global IOU similarity matrix is established for the targets in the detection frame at t time and the trajectory frame at t-1 time that have not been associated. Then, unlike the cascade matching in step S6, the matching result is obtained by the Hungarian algorithm once this time, and the calculation threshold of the IOU is still set to 0.5. The identity ID of the associated target is assigned to the detection frame target, and a new identity ID is assigned to the unassociated detection frame ID. For the trajectory frame target that has not completed the association, the part of the ID is discarded.

[0084] Finally, the target trajectory is obtained by frame-by-frame tracking of the target, and the multi-target tracking based on the depth information decoupling is achieved.

[0085] The method of the present application first constructs a multi-target tracking model based on three-dimensional depth information decoupling and trains it, then clusters two-dimensional target detection boxes to complete detection of dense occlusion, and then obtains target three-dimensional depth information through a self-supervised depth estimation method, decouples dense occlusion using the depth information, and completes multi-target tracking based on spatial information. The method of the present application does not need to solve the problem of inaccurate extraction of target appearance feature information caused by dense occlusion, proposes MOT based on depth information, only uses IOU matching to realize association, solves the occlusion problem through the advantage of depth information, does not need appearance features, proposes an algorithm based on three-dimensional depth estimation, realizes hierarchical decoupling of dense occlusion targets based on the difference in depth information, and performs data association in different levels. Different levels of targets cannot affect each other, and a robust multi-target tracking effect is obtained. BRIEF DESCRIPTION OF DRAWINGS

[0086] Figure 1 A flowchart of a multi-target tracking method based on three-dimensional depth information decoupling of the present application.

[0087] Figure 2 A flowchart based on target detection and depth structure feature fusion in an embodiment of the present application.

[0088] Figure 3 A depth information extraction effect diagram in an embodiment of the present application.

[0089] Figure 4 A clustering diagram of the OCR Mean-shift algorithm in an embodiment of the present application.

[0090] Figure 5 An effect diagram of the OCR Mean-shift algorithm in an embodiment of the present application.

[0091] Figure 6 A pseudo code algorithm flowchart of the OCR Mean-Shift algorithm in an embodiment of the present application.

[0092] Figure 7 An overall multi-target tracking effect diagram in an embodiment of the present application. DETAILED DESCRIPTION

[0093] The method of the present application will be further described below in combination with the drawings and embodiments.

[0094] As shown in the flowchart of the multi-target tracking method based on three-dimensional depth information decoupling of the present application, the specific steps are as follows: Figure 1 S1, a multi-target tracking model based on three-dimensional depth information decoupling is constructed;

[0095]

[0096] ​The multi-target tracking model includes: one attention-based feature extraction network, one camera pose estimation network, three depth estimation prediction heads, and three target detection prediction heads.

[0097] S2, Training the multi-target tracking model constructed in step S1;

[0098] S3. Based on the trained multi-target tracking model obtained in step S2, input the image at time t to obtain the target detection data of the image at time t and the three-dimensional depth information at time t.

[0099] Each detected target is represented by a bounding box without an ID; the box without an ID is the detection box. The target detection prediction head outputs multiple pedestrian target boxes for each input image. 3D depth information helps to achieve decoupling.

[0100] S4. Based on step S3, the improved Mean-shift clustering algorithm (OCR Mean-shift algorithm) is used to cluster multiple detection boxes at time t and trajectory boxes at time t-1 to identify dense occlusions, and each class is a dense occlusion.

[0101] In this clustering, if the number of targets in each cluster exceeds 5, it is considered as dense occlusion; if the number of targets is less than 5, it is considered as a target with mild or no occlusion, i.e., a non-dense occlusion target.

[0102] S5. Based on step S4, realize the data association between non-densely occluded targets and densely occluded targets;

[0103] S6. Process the unmatched detection box targets and trajectory box targets in step S5, track the targets frame by frame to obtain the target trajectory, and realize multi-target tracking based on depth information decoupling.

[0104] like Figure 2 As shown, the process is based on the fusion of target detection and deep structural features, namely the multi-target tracking model constructed in training step S1.

[0105] In this embodiment, step S2 is specifically as follows:

[0106] S21. Extract deep feature information using an attention-based feature extraction network;

[0107] The feature extraction network is a U-shaped backbone network, including: Conv-stem convolutional module, average pooling layer, stage 2 convolutional layer, stage 3 convolutional layer, stage 4 convolutional layer, and 3 upsampling and convolutional modules.

[0108] The convolution kernel size of the stage 1-4 convolution layer is 3x3, and the step is 2; the stage 2 convolution layer and the stage 3 convolution layer each include one down-sampling layer, three continuous dilated convolution modules CDC and one local-global feature interaction module LGFI; the stage 4 convolution layer includes one down-sampling layer, six continuous dilated convolution modules CDC and one local-global feature interaction module LGFI. The up-sampling and convolution module includes one up-sampling layer and one convolution layer.

[0109] The deep feature information is extracted by using the feature extraction network, that is, the deep feature information is extracted by using the encoder of the like-UNet, and is divided into four stages, and the specific process is as follows:

[0110] 1) The first stage:

[0111] An original image with a size of HxWx3 is input as an input feature into a Conv-stem convolution module for down-sampling to obtain an output feature map with a size of H / 2xW / 2xC1. tmp1

[0112] H represents the height of the image, W represents the width of the image, and C represents the channel number of the image; the Conv-stem module is composed of three convolution layers, the first layer is convolved with a convolution kernel with a size of 3x3 and a step of 2 to achieve down-sampling, and the last two layers are convolved with a convolution kernel with a size of 3x3 but a step of 1 to achieve local feature extraction.

[0113] 2) The second stage:

[0114] The original image is processed by an average pooling layer to obtain a feature map with a size of H / 2xW / 2x3, which is spliced with the feature map with a size of H / 2xW / 2xC1 tmp1 obtained in the first stage to obtain a feature map with a size of H / 2xW / 2xC1 as an input feature, and then input into a convolution layer with a convolution kernel size of 3x3 and a step of 2 for down-sampling to obtain a down-sampled feature with a size of H / 4xW / 4xC2 tmp1 , and then pass through three continuous dilated convolution modules CDC and one local-global feature interaction module LGFI to obtain an output feature map with a size of H / 4xW / 4xC2 tmp2 .

[0115] In this stage, the window size of the average pooling layer is set to H / 2xH / 2, and the step is set to H / 2, that is, the original image with a size of HxWx3 is processed by the average pooling layer, and the output size is H / 2xW / 2x3.

[0116] ​The reason for concatenating the output features of the first stage with the average-pooled input image features is that this can reduce the spatial information loss caused by feature size reduction, which will also be used in subsequent feature processing. The CDC and LGFI modules are referenced from the Lite-Mono implementation, the former uses a dilated convolution to extract multi-scale local features, and the latter uses a cross-covariance attention mechanism to focus on feature changes in the channel. Using these two modules can learn rich hierarchical feature information.

[0117] 3) Third stage:

[0118] The preliminary down-sampled feature map obtained in the second stage has a size of H / 4 x W / 4 x C2 tmp1 , the final output of the stage has a size of H / 4 x W / 4 x C2 tmp2 , and the feature map of the original image after being processed by the average pooling layer has a size of H / 4 x W / 4 x 3. These three feature maps are concatenated to obtain a feature map with a size of H / 4 x W / 4 x C2 as input features, and then input to a convolutional layer with a kernel size of 3 x 3 and a stride of 2 for down-sampling, to obtain a down-sampled feature map with a size of H / 8 x W / 8 x C3 tmp1 , and then pass through 3 CDC modules and 1 LGFI module to obtain an output feature map with a size of H / 8 x W / 8 x C3 tmp2 .

[0119] In this stage, the average pooling window size is set to H / 4 x H / 4, the stride is set to H / 4, and the output size of the average pooling layer is H / 4 x W / 4 x 3.

[0120] 4) Fourth stage:

[0121] The preliminary down-sampled feature map obtained in the third stage has a size of H / 8 x W / 8 x C3 tmp1 , the final output of the stage has a size of H / 8 x W / 8 x C3 tmp2 , and the feature map of the original image after being processed by the average pooling layer has a size of H / 8 x W / 8 x 3. These three feature maps are concatenated to obtain a feature map with a size of H / 8 x W / 8 x C3 as input features, and then input to a convolutional layer with a kernel size of 3 x 3 and a stride of 2 for down-sampling, and the sampling result is input to 6 CDC modules and 1 LGFI module to obtain an output feature map with a size of H / 16 x W / 16 x C4.

[0122] In this stage, the average pooling window size is set to H / 8 x H / 8, the stride is set to H / 8, and the output size of the average pooling layer is H / 8 x W / 8 x 3.

[0123] In the above description, the input of the third and fourth stages needs to splice the features of the down-sampling layer output of the previous stage. This design is similar to the residual connection proposed in ResNet, and can better model the cross-stage feature correlation.

[0124] Then, bilinear up-sampling is used for the feature maps of the second, third and fourth stages to increase the spatial dimension, and a convolution layer is used to connect the features of the three stages of the encoder, i.e., after each up-sampling and convolution module, a depth estimation prediction head and a target detection prediction head are followed. The depth estimation prediction head will output inverse depth map features at resolutions of 1, 1 / 2 and 1 / 4 of the original image resolution, respectively, and this part of the feature will be combined with the camera pose estimation information to obtain the final depth estimation result.

[0125] S22, through the depth estimation prediction head, the multi-scale feature information extracted in step S21 is fused to obtain depth estimation information at different scales, and then the camera pose estimation information obtained by the camera pose estimation network Pose Net is used to fuse and reconstruct the multi-scale depth estimation information to obtain the final three-dimensional depth information (depth estimation result), so as to realize the reconstruction of the three-dimensional depth feature information;

[0126] S221, through the feature fusion of the different stage features obtained in step S21, three different scale depth estimation information, i.e., multi-scale depth estimation information, is obtained through the depth estimation prediction head;

[0127] (1) The size H / 16×W / 16×C4 feature obtained in the fourth stage is input into the first up-sampling and convolution module for up-sampling and residual connection with the size H / 8×W / 8×C3 tmp2 feature of the third stage, and then a convolution layer is used to obtain a size H / 8×W / 8×C3 ~ fusion feature, which is input into the depth estimation prediction head to finally obtain a size H / 4×W / 4×1 depth estimation information.

[0128] Among them, the depth estimation prediction head includes a convolution layer Conv, an up-sampling layer and an activation layer Sigmoid.

[0129] (2) The size H / 8×W / 8×C3 ~ fusion feature is input into the second up-sampling and convolution module for up-sampling and residual connection with the size H / 4×W / 4×C2 tmp2 feature of the second stage, and then a convolution layer is used to obtain a size H / 4×W / 4×C2 ~ fusion feature, which is input into another depth estimation prediction head with the same structure to finally obtain a size H / 2×W / 2×1 depth estimation information.

[0130] (3) the fused features with a size of H / 4xW / 4xC2 ~ are input into a third upsampling and convolution module for upsampling, and then directly pass through a convolution layer to obtain feature H / 2xW / 2xC1 ~ , and are input into a depth estimation prediction head with the same structure, and finally obtain depth estimation information with a size of HxWx1.

[0131] S222, camera pose estimation information of adjacent two frames is obtained, that is, camera pose estimation is completed through a camera pose estimation network Pose Net;

[0132] For monocular camera training, the camera pose estimation network is formed by ResNet18, the input is a pair of color images or six channels, and a four-layer convolutional pose decoder is used to estimate the relative pose of six degrees of freedom between adjacent two frames of images. At the same time, horizontal inversion and random brightness, contrast, saturation and hue jitter with a range of ±0.2, ±0.2, ±0.2 and ±0.1 are trained respectively to ensure the robustness of camera pose estimation.

[0133] Among them, the ResNet18 network is pre-trained; the six degrees of freedom are the translational degrees of freedom of the x-axis, y-axis and z-axis in the three-dimensional coordinate system and the rotational degrees of freedom, and the pose of the camera can be represented by a 4x4 homogeneous transformation matrix T through the six degrees of freedom, that is, a 3x3 rotation matrix R and a 3x1 translation vector z. This camera pose matrix can represent the pose change of the camera between different frames.

[0134] S223, the original rough three-dimensional depth information obtained in step S21 is corrected, that is, the multi-scale depth estimation information obtained in step S221 is fused and reconstructed to obtain the final three-dimensional depth information based on step S222, and three-dimensional depth feature information is reconstructed through projection transformation and linear interpolation;

[0135] Firstly, projection transformation is performed, that is, each pixel point in the depth map is projected into the world coordinate system, and then is projected into the image plane of the target view according to the camera pose transformation matrix.

[0136] For each pixel point (u, v), a pixel coordinate grid is created, the pixel coordinates are converted into normalized coordinates, then the normalized coordinates are multiplied by the depth estimation information d of the corresponding pixel point to obtain a 3D point in the world coordinate system, and then the 3D point is projected from the current view to the target view using the pose transformation matrix T of the camera, and the 3D point is projected back to the image plane using the camera intrinsic matrix K (known) of the target view. The projection coordinates are normalized to the image plane, that is, from three-dimensional coordinates (x, y, z) to two-dimensional coordinates (x / z, y / z, 1).

[0137] Then comes bilinear interpolation, which maps the pixel values ​​of the projection points onto the target image on the image plane at the target's viewpoint.

[0138] The projected coordinates are normalized to the image plane range [−1,1]. Using bilinear interpolation, the pixel values ​​corresponding to the normalized projected coordinates are interpolated from the image from the target viewpoint. Finally, the depth estimation information after camera pose estimation correction is obtained, which is the final 3D depth estimation map. The calculation expression is as follows:

[0139]

[0140] in, , These represent the horizontal and vertical decimal parts of the normalized projected coordinates, respectively. This represents the four most recent pixel values.

[0141] As described above, the uncorrected depth estimation information used as initial input has different resolutions. Therefore, for each resolution of depth estimation information, it is necessary to first upsample it to the same resolution as the original image, and then reconstruct the final 3D depth estimation map at each scale through the described steps. During inference, only the final 3D depth estimate corresponding to the depth estimation information with a resolution of H×W×1 is used as the result of the final 3D depth estimation of the input original image. The other two resolutions of 3D depth estimation are only used during network training. This can effectively constrain the depth maps at each scale to work towards the same goal, that is, to reconstruct the high-resolution input target image as accurately as possible, thereby improving the convergence speed of the network.

[0142] The depth estimation prediction head learns the model of the target, i.e., minimizes the target image. and synthesized target image Image reconstruction loss between And edge-aware smoothing loss constrained on the predicted depth map. .

[0143] The expression for the depth estimation prediction head loss function is as follows:

[0144]

[0145]

[0146]

[0147]

[0148] in, Indicates the image reconstruction loss. Represents the target image. representing the synthesized target image, representing a fixed threshold value 0.85, representing the inter-pixel structural similarity index, representing the inter-pixel similarity characteristics, used to characterize the similarity of the mapping relationship at the pixel level. representing the minimum photometric loss of the pixels outside the processing field of view and the occluded objects in the source image, representing the previous frame or the next frame image used to represent the source target image. representing the weighted edge-aware smoothing loss, which aims to ensure that the predicted depth map remains smooth in the edge area while maintaining a certain continuity in the non-edge area, representing the average normalized depth. representing the weight information for the weighted edge-aware smoothing loss, which is set to 0.001. representing the sum of the loss functions containing multiple scale features. The final depth information extraction effect diagram obtained by the embodiment is shown in Figure 3 .

[0149] S23, based on the different scale features obtained in the second, third and fourth stages of step S21, i.e., the fusion features with sizes of HxWxC1 ~ , H / 2xW / 2xC2 ~ , H / 4xW / 4xC3 ~ obtained by inputting the upsampling and convolution module in step S23, each size of the feature is sent to a target detection prediction head to realize the classification subtask and the frame regression subtask;

[0150] The target detection prediction head includes a classification prediction head and a frame regression prediction head.

[0151] The classification prediction head includes two layers of convolution layers with C 3x3 convolution kernels, one layer of ReLU activation layer, one layer of convolution layer with 2A 3x3 convolution kernels, and Sigmoid activation function. The frame regression prediction head includes two layers of convolution layers with C 3x3 convolution kernels, one layer of ReLU activation layer, one layer of convolution layer with 4A 3x3 convolution kernels, and Sigmoid activation function. The classification prediction head is used to predict the probability of the class to which the A anchor frames belong at each pixel position, and the classes include background targets and foreground pedestrian targets. The frame regression prediction head is used to predict the offset of each anchor frame (center point x coordinate, center point y coordinate, width w, height ).

[0152] The first layer of the classification prediction head is a 2-layer convolutional layer with C convolutional kernels of size 3x3. The output features are then input to a ReLU activation layer, and then passed through another convolutional layer with 2A convolutional kernels of size 3x3. Finally, the output is passed through a Sigmoid activation function, which outputs 2A binary prediction results for each spatial position, indicating whether it belongs to the background or foreground pedestrian target.

[0153] where A is set to 9, and the size of C is determined according to the actual number of channels of the input features.

[0154] Similarly, the bounding box regression prediction head outputs 4A linear results for each spatial position. For each of the A anchor boxes at each spatial position, the 4 outputs represent the relative offset between the predicted anchor box and the ground truth.

[0155] The bounding box regression prediction head uses a class-independent bounding box regressor, which uses fewer parameters but achieves good results. Although the target classification prediction head and the bounding box regression prediction head share similar structures, they use independent parameters.

[0156] The loss function of the target detection prediction head which adopts the focal Loss for cross-entropy and L1 Loss for regression tasks to complete the training of the prediction head, with the specific expressions as follows:

[0157]

[0158]

[0159]

[0160] where, represents the predicted probability, which is when the sample is a positive sample , represents the direct output probability of the prediction head, represents the balance factor, which is used to balance the influence of positive and negative samples, is used for difficult sample adjustment, when the sample is a difficult sample, the weight of the simple sample is reduced, and the attention to the difficult sample is enhanced. represents the true value, and represents the predicted value, and N represents the number of samples. , represent the weight information of the two loss functions, respectively, and are set to 0.45 and 0.55, respectively.

[0161] In the embodiment, the step S4 is specifically as follows:

[0162] A Mean-shift clustering algorithm for densely occluded targets, OCR Mean-shift algorithm, is realized by introducing a new weight calculation method. The weight coefficient reflects the weight distribution of global sample points, but for an occluded target, the main influence on the calculation of its offset is the target that occludes it, so this global weight coefficient often ignores this more important sample weight.

[0163] The detection frame obtained from the current t time image is subjected to OCR Mean-shift clustering, and the densely occluded and non-densely occluded in the detection frame are classified. Meanwhile, the track frame of the t-1 time image is subjected to OCR Mean-shift clustering, and the densely occluded and non-densely occluded in the track frame are classified.

[0164] Among them, the track information of the t-1 time is known data, that is, each target in the t-1 time image is represented by a bounding box and a unique ID number. The frame with the ID is the track frame, and the three-dimensional depth information of the t-1 time image is known.

[0165] The clustering schematic diagram and clustering effect diagram of the OCR Mean-shift algorithm are shown in Figure 4 、 Figure 5 For all target bounding boxes detected at each time, the center point coordinates are taken as the sample points to be clustered, and the expression is as follows:

[0166]

[0167]

[0168]

[0169] Among them, , represents a calculation factor, which is used to measure the characteristics of whether the target is occluded and the connection between the targets in the occlusion group; represents a certain sample point to be clustered (the center point coordinates of each frame), represents the nearest sample point of the sample point in the Euclidean distance, represents the class probability directly output by the classification prediction head, represents the IOU value between the bboxes corresponding to the sample points and the sample points, represents the area of the bboxes corresponding to the sample points, represents the improved sample point weight, The weight factors of the two factors are fixed as represents a Gaussian kernel function, represents a bandwidth, which controls the action range of the kernel function, for completing the mean-shift clustering and updating the drift point Position.

[0170] , mainly measures the detection effect of the target itself. Generally speaking, the detection difficulty of the occluded target is relatively high, so the classification confidence of the target frame is often low. Although it may be caused by the size of the target itself and the detection challenge of illumination change, occlusion is also a very important reason, so this kind of target is still regarded as a candidate target of the dense occlusion group, and is reflected in the value of the weight coefficient. , mainly measures the weight influence between the target and the target occluding it. The reason is that the size difference of the target in the picture. The coordinates of the sample points cannot fully reflect the occlusion situation, and the area covered by the target frame and the overlap with other targets are the factors directly reflecting the occlusion situation.

[0171] The improvement of the OCR mean-shift algorithm is reflected in the weight calculation of the sample points, and the overall algorithm flow is still implemented according to the algorithm flow of Mean-shift. The specific pseudo code algorithm flow is as shown in Figure 6 .

[0172] In the embodiment, the step S5 is specifically as follows:

[0173] S51, data association of non-dense occlusion targets is realized;

[0174] Based on the non-dense occlusion target obtained in step S4, the basic association method of SORT is directly used, the IOU similarity is calculated through the non-dense occlusion target of the detection frame at t time and the track frame at t-1 time, the association of the non-dense occlusion target is realized by relying on the similarity matrix using the Hungarian algorithm, and the association of this part of the target is relatively accurate, then the associated IOU threshold is set to 0.5, when the IOU similarity is less than 0.5, it is regarded as not associated.

[0175] ​The common multi-target tracking method can realize the identity division of the target by introducing an appearance feature extraction module. On the one hand, the extraction of the appearance feature is relatively difficult, because an additional loss function needs to be designed to train the appearance feature extraction network, and a plurality of Re-ID training sets need to be combined to extract sufficient appearance features. On the other hand, the appearance feature is easily affected by other targets in the case of dense occlusion. The appearance information extracted by the targets may be overlapped and confused due to the occlusion. The embodiment based on the depth information of the MOT only uses the IOU matching to realize the association, which solves the occlusion problem by the advantage of the depth information without the appearance feature.

[0176] The existing method does not additionally consider the processing of dense occlusion, but in the embodiment, the dense occlusion is processed. Because the dense occlusion does not always occur and is not global, it is often short and local, so the mean-shift algorithm based on the confidence of the detection box and the IOU of the same frame to calculate the weight of the sample point of the detection box can well realize the recognition of the dense occlusion, and facilitate the processing of the dense occlusion.

[0177] S52, realizing data association of the dense occlusion target;

[0178] For the processing of the dense occlusion, the dense occlusion is uniformly divided by the depth information, so that the mutually occluded targets are decoupled in different levels, and the targets in different levels are not interfered with each other by using the step-by-step data association, so as to improve the tracking effect of the dense occlusion target and reduce the IDswitch.

[0179] Based on the plurality of dense occlusions obtained in step S4, the dense occlusion centroids of the detection box and the track box are calculated, and the matching of the dense occlusion is completed by the Euclidean distance of the plurality of centroids of the two boxes. That is, the data association in the dense occlusion is realized between the certain dense occlusion of the detection box and the certain dense occlusion of the track box. And the association strategy is based on the three-dimensional depth information at time t obtained in step S3.

[0180] The three-dimensional depth information of the target in the dense occlusion is used to realize the uniform division of the depth in the dense occlusion. That is, according to the depth information of each target in the dense occlusion (the three-dimensional depth corresponding to the pixel of the bottom center point of the target detection box), the deepest depth and the shallowest depth are found. Because in the dense occlusion, the targets are not completely randomized like the global target distribution, but are relatively uniform, the four-level association regions are uniformly divided between the deepest depth and the shallowest depth. For the two dense occlusions of the detection box at time t and the track box at time t-1, only the targets of the same level can realize the data association by calculating the IOU.

[0181] Then, according to the strategy of hierarchical association, the target is associated layer by layer, and the unassociated target is sequentially extended to the next layer for the same IOU association until the four times of cascaded association are completed, and the identity ID is assigned to the matched target.

[0182] In the embodiment, the step S6 is specifically as follows:

[0183] For the target that is not associated in steps S5 and S6, the last data association is performed, a global IOU similarity matrix is established for the target that is not associated in the detection frame and the track frame at t-1 time, and then different from the cascaded matching in step S6, the matching result is obtained by the Hungarian algorithm once, and the calculation threshold of IOU is still set to 0.5. The identity ID of the associated detection frame target is assigned, and the new identity ID is assigned to the unassociated detection frame ID, and the part of the ID of the track frame target that is not associated is discarded.

[0184] Finally, the target trajectory is obtained by frame-by-frame tracking of the target, and the multi-target tracking based on the decoupling of depth information is realized. The final tracking effect diagram of the embodiment is as shown in Figure 7 .

[0185] In summary, the method of the application does not need to solve the problem that the target appearance feature information extraction is not accurate due to dense occlusion, proposes the MOT based on depth information, only uses the IOU matching to realize the association, solves the occlusion problem through the advantage of depth information, does not need the appearance feature, proposes the algorithm based on three-dimensional depth estimation (multi-level data association strategy based on three-dimensional depth information), realizes the hierarchical decoupling of the dense occlusion target based on the difference of depth information, performs data association in different levels, the targets in different levels cannot affect each other, realizes the efficient multi-target tracking, and obtains the robust multi-target tracking effect.

[0186] Those skilled in the art will appreciate that the embodiments described herein are presented for the purpose of helping the reader to understand the principles of the application, and should be understood as not limiting the protection scope of the application to such specific statements and embodiments. Those skilled in the art can make various other specific modifications and combinations according to the technical inspiration disclosed in the application, and these modifications and combinations do not deviate from the essence of the application, and are still within the protection scope of the application.

Claims

1. A multi-target tracking method based on 3D depth information decoupling, the specific steps of which are as follows: S1. Construct a multi-target tracking model based on 3D depth information decoupling; The multi-target tracking model includes: One attention-based feature extraction network, one camera pose estimation network, three depth estimation prediction heads, and three object detection prediction heads; S2, Training the multi-target tracking model constructed in step S1; S3. Based on the trained multi-target tracking model obtained in step S2, input the image at time t to obtain the target detection data of the image at time t and the three-dimensional depth information at time t. Each detected target is represented by a bounding box without ID information; the box without an ID is the detection box. The target detection prediction head outputs multiple pedestrian target boxes for each input image. S4. Based on step S3, the improved Mean-shift clustering algorithm is used to cluster multiple detection boxes at time t and trajectory boxes at time t-1 to identify dense occlusions, and each class is a dense occlusion. In this clustering, if the number of targets in each clustering result exceeds 5, it is considered as dense occlusion; if the number of targets is less than 5, it is considered as a target with mild or no occlusion, i.e., a non-dense occlusion target. S5. Based on step S4, realize the data association between non-densely occluded targets and densely occluded targets; S51. Implement data association for targets with non-dense occlusion. Based on the non-densely occluded targets obtained in step S4, the basic association method of SORT is directly adopted. The IOU similarity is calculated between the detection box at time t and the trajectory box at time t-1 for non-densely occluded targets. The association of non-densely occluded targets is achieved by using the Hungarian algorithm based on the similarity matrix, and the IOU threshold for association is set to 0.

5. When the IOU similarity is less than 0.5, it is considered that no association will occur. S52. Implement data association for densely occluded targets; Based on step S4, multiple dense occlusions are obtained. The centroids of dense occlusions of the detection box and the trajectory box are calculated. The matching of dense occlusions is completed by the Euclidean distance between the multiple centroids of the two boxes. That is, a dense occlusion of the detection box and a dense occlusion of the trajectory box realize the data association within the occlusion. The association strategy is based on the three-dimensional depth information at time t obtained in step S3. By utilizing the 3D depth information of targets within dense occlusion, a uniform depth hierarchy is achieved within the dense occlusion. Specifically, based on the depth information of each target within the dense occlusion, the deepest and shallowest depths are found. The deepest and shallowest depths are uniformly divided into four levels of associated regions. For two densely occluded regions, the detection box at time t and the trajectory box at time t-1, only targets at the same level can be associated by calculating IOU. Then, following the hierarchical association strategy, targets are associated layer by layer. Unassociated targets will be carried over to the next layer for the same IOU association, until the fourth cascade association is completed, and identity IDs are assigned to the matched targets. S6. Process the unmatched detection box targets and trajectory box targets in step S5, track the targets frame by frame to obtain the target trajectory, and realize multi-target tracking based on depth information decoupling; For targets that were not associated in step S5, a final data association is performed. A global IOU similarity matrix is ​​established for targets that are still not associated in the detection box at time t and the trajectory box at time t-1. Then, unlike the cascaded matching in step S6, this association obtains the matching result in one step using the Hungarian algorithm. The IOU calculation threshold is still set to 0.

5. The associated detection frame targets are assigned the identity ID of the associated party, while the unassociated detection frame IDs are assigned new identity IDs. For trajectory frame targets that have not completed association, these IDs are discarded. Finally, the target trajectory is obtained by tracking the target frame by frame, realizing multi-target tracking based on depth information decoupling.

2. The multi-target tracking method based on three-dimensional depth information decoupling according to claim 1, characterized in that, Step S2 is as follows: S21. Extract deep feature information using an attention-based feature extraction network; The feature extraction network is a U-shaped backbone network, including: Conv-stem convolutional module, average pooling layer, stage 2 convolutional layer, stage 3 convolutional layer, stage 4 convolutional layer, and 3 upsampling and convolutional modules; In this configuration, the kernel size of the convolutional layers in stages 1-4 is 3×3, and the stride is 2. The convolutional layers in stages 2 and 3 each include: one downsampling layer, three continuously dilated convolutional modules (CDC), and one local-global feature interaction module (LGFI). The convolutional layer in stage 4 includes: one downsampling layer, six continuously dilated convolutional modules (CDC), and one local-global feature interaction module (LGFI). The upsampling and convolutional modules include: one upsampling layer and one convolutional layer. The feature extraction network is used to extract deep feature information, specifically using the encoder of like-UNet. This process involves four stages, as detailed below: 1) First stage: The original image of size H×W×3 is used as the input feature and input into the Conv-stem convolutional module for downsampling to obtain a feature of size H / 2×W / 2×C1. tmp1 The output feature map; Where H represents the height of the image, W represents the width of the image, and C represents the number of channels of the image; the Conv-stem module consists of three convolutional layers. The first layer uses a 3×3 kernel with a stride of 2 to achieve convolution and downsampling. The latter two layers use a 3×3 kernel with a stride of 1 to achieve convolution and local feature extraction. 2) Second stage: The original image is processed by an average pooling layer to obtain a feature map of size H / 2×W / 2×3, which is compared with the feature map of size H / 2×W / 2×C1 obtained in the first stage. tmp1 The feature maps are concatenated to obtain a feature map of size H / 2×W / 2×C1, which is then used as the input feature map. This input feature map is then fed into a convolutional layer with a kernel size of 3×3 and a stride of 2 for downsampling, resulting in a feature map of size H / 4×W / 4×C2. tmp1 The downsampled features are then processed through three consecutively dilated convolutional modules (CDC) and one local-global feature interaction module (LGFI) to obtain a sample of size H / 4×W / 4×C2. tmp2 The output feature map; In this stage, the window size of the average pooling layer is set to H / 2×H / 2, and the stride is set to H / 2. That is, the original image with size H×W×3 passes through this average pooling layer and outputs an image with size H / 2×W / 2×3. 3) Third stage: The size obtained in the second stage is H / 4×W / 4×C2. tmp1 The initial downsampling feature map and the final output size of the stage are H / 4×W / 4×C2. tmp2 The feature map of the original image and the feature map of the original image after average pooling are concatenated to obtain a feature map of size H / 4×W / 4×3 as the input feature. This feature map is then input into a convolutional layer with a kernel size of 3×3 and a stride of 2 for downsampling to obtain a feature map of size H / 8×W / 8×C3. tmp1 The downsampled feature map is then processed through 3 CDC modules and 1 LGFI module to obtain a size of H / 8×W / 8×C3. tmp2 The output feature map; In this stage, the average pooling window size is set to H / 4×H / 4, the step size is set to H / 4, and the output size of the average pooling layer is H / 4×W / 4×3. 4) Fourth stage: The size obtained in the third stage is H / 8×W / 8×C3. tmp1 The initial downsampling feature map and the final output size of the stage are H / 8×W / 8×C3. tmp2 The feature map of the original image and the feature map of the original image after the average pooling layer are concatenated to obtain a feature map of size H / 8×W / 8×C3 as input features. Then, it is input into a convolutional layer with a kernel size of 3×3 and a stride of 2 for downsampling. The sampling result is input into 6 CDC modules and 1 LGFI module to obtain an output feature map of size H / 16×W / 16×C4. In this stage, the average pooling window size is set to H / 8×H / 8, the step size is set to H / 8, and the output size of the average pooling layer is H / 8×W / 8×3. S22. The depth estimation information at different scales is obtained by fusing the multi-scale feature information extracted in the depth estimation prediction head fusion step S21. Then, the camera pose estimation network Pose Net is used to fuse and reconstruct the multi-scale depth estimation information to obtain the final three-dimensional depth information, thereby realizing the reconstruction of three-dimensional depth feature information. S221. By fusing the features of different stages obtained in step S21, and then obtaining depth estimation information of three different scales through the depth estimation prediction head, i.e. multi-scale depth estimation information. (1) Input the features of size H / 16×W / 16×C4 obtained in the fourth stage into the first upsampling and convolution module for upsampling and residual connection to the third stage with size H / 8×W / 8×C3. tmp2 The features are then processed through a convolutional layer to obtain a value of H / 8×W / 8×C3. ~ The fusion features are input into the depth estimation prediction head, and finally the depth estimation information with a size of H / 4×W / 4×1 is obtained. The depth estimation prediction head includes: a convolutional layer (Conv), an upsampling layer, and an activation layer (Sigmoid). (2) The size is H / 8×W / 8×C3 ~ The fused features are input to the second upsampling and convolutional module for upsampling, and residual connections are made in the second stage with a size of H / 4×W / 4×C2. tmp2 The features are then processed through a convolutional layer to obtain a value of H / 4×W / 4×C2. ~ The fusion features are input into another depth estimation prediction head with the same structure, and finally the depth estimation information with a size of H / 2×W / 2×1 is obtained. (3) The size is H / 4×W / 4×C2 ~ The fused features are input to the third upsampling and convolutional module for upsampling, and then directly passed through the convolutional layer to obtain the features H / 2×W / 2×C1. ~ The data is then input into a depth estimation prediction head with the same structure, ultimately yielding depth estimation information of size H×W×1. S222. Obtain camera pose estimation information for two adjacent frames, that is, complete camera pose estimation through the Pose Net camera pose estimation network. For monocular camera training, the camera pose estimation network is formed by ResNet18, with input being a pair of color images or six channels. A four-layer convolutional pose decoder is used to estimate the corresponding six degrees of freedom relative pose between two adjacent frames. Simultaneously, horizontal flipping and random brightness, contrast, saturation, and hue jitter within the ranges of ±0.2, ±0.2, ±0.2, and ±0.1 are performed during training. The six degrees of freedom are the translational and rotational degrees of freedom of the x-axis, y-axis, and z-axis in the three-dimensional coordinate system. The camera's pose is represented by a 4x4 homogeneous transformation matrix T, which is a 3x3 rotation matrix R and a 3x1 translation vector z. S223. Correct the original coarse 3D depth information obtained in step S21. That is, based on step S222, fuse and reconstruct the multi-scale depth estimation information obtained in step S221 to obtain the final 3D depth information. The 3D depth feature information is reconstructed through projection transformation and linear interpolation. First, a projection transformation is performed, which projects each pixel in the depth map onto the world coordinate system. Then, based on the camera pose transformation matrix, the pixel is projected onto the image plane of the target view. For each pixel (u,v), a pixel coordinate grid is created, the pixel coordinates are converted to normalized coordinates, and then the normalized coordinates are multiplied by the depth estimation information d of the corresponding pixel to obtain the 3D point in the world coordinate system. Then, the camera's pose transformation matrix T is used to project the 3D point from the current viewpoint to the target viewpoint. The camera intrinsic parameter matrix K of the target viewpoint is used to project the 3D point back to the image plane, and the projected coordinates are normalized to the image plane, that is, from three-dimensional coordinates (x, y, z) to two-dimensional coordinates (x / z, y / z, 1). Then comes bilinear interpolation, which maps the pixel values ​​of the projection points onto the target image on the image plane of the target viewpoint; The projected coordinates are normalized to the image plane range [−1,1]. Using bilinear interpolation, the pixel values ​​corresponding to the normalized projected coordinates are interpolated from the image from the target viewpoint. Finally, the depth estimation information after camera pose estimation correction is obtained, which is the final 3D depth estimation map. The calculation expression is as follows: ； in, , These represent the horizontal and vertical decimal parts of the normalized projected coordinates, respectively. Represents the four most recent pixel values; The depth estimation prediction head learns the model of the target, i.e., minimizes the target image. and synthesized target image Image reconstruction loss between And edge-aware smoothing loss constrained on the predicted depth map. ; The expression for the depth estimation prediction head loss function is as follows: ； ； ； ； in, Indicates the image reconstruction loss. Represents the target image. Indicates the synthesized target image. This indicates a fixed threshold of 0.

85. It represents the structural similarity index between pixels. Indicates the distance between pixels Similarity characteristics are used to characterize the similarity of pixel-level mapping relationships; This represents the minimum photometric loss in the source image when processing out-of-view pixels and occluded objects. This indicates the previous or next frame of the source / target image; This represents the weighted edge-aware smoothing loss, which aims to ensure that the predicted depth map remains smooth in edge regions while maintaining a certain degree of continuity in non-edge regions. Indicates the average normalized depth; This represents the weight information used in the weighted edge-aware smoothing loss; This represents the sum of loss functions that include features at multiple scales; S23. Based on the features of different scales obtained from the fusion of the second, third, and fourth stages in step S21, i.e., the size obtained by the input upsampling and convolution module fusion in step S23 is H×W×C1 ~ H / 2×W / 2×C2 ~ H / 4×W / 4×C3 ~ The fusion features are used to feed the features of each size into an object detection prediction head to achieve the classification subtask and the bounding box regression subtask; The target detection prediction head includes: a classification prediction head and a bounding box regression prediction head; The classification prediction head includes: two convolutional layers with C 3×3 kernels, one ReLU activation layer, one convolutional layer with 2A 3×3 kernels, and a Sigmoid activation function; the bounding box regression prediction head includes: two convolutional layers with C 3×3 kernels, one ReLU activation layer, one convolutional layer with 4A 3×3 kernels, and a Sigmoid activation function; the classification prediction head predicts the probability of the category of A anchor boxes at each pixel location, with categories including background targets and foreground pedestrian targets; while the bounding box regression prediction head predicts the x-coordinate of the center point, the y-coordinate of the center point, the width w, and the height of each anchor box. The offset; The classification prediction head first uses two convolutional layers with C 3×3 kernels each. Then, its output features are input into a ReLU activation layer. The output is then passed through a convolutional layer with 2A 3×3 kernels each. Finally, the output is passed through a Sigmoid activation function to output 2A binary prediction results for each spatial location, i.e., whether it belongs to the background target or the foreground pedestrian target. Where A is set to 9, and the value of C is determined according to the actual number of channels of the input feature; Similarly, the box regression prediction head outputs 4A linear results at each spatial location; for each spatial location with A anchor boxes, these 4 outputs represent the relative offset between the predicted anchor box and the ground truth box. The loss function of the target detection prediction head That is, using focal loss for cross-entropy. and L1 Loss for regression tasks To complete the training of the prediction head, the expression is as follows: ； ； ； in, This represents the predicted probability when the sample is a positive sample. When it is a negative sample , This represents the direct output probability of the prediction head. This represents a balancing factor used to balance the influence of positive and negative samples. Used for adjusting between easy and difficult samples At times, the weight of simple samples is reduced, while the focus on difficult samples is increased; Represents the true value, while N represents the predicted value, and N represents the sample size. , These represent the weight information for the two loss functions, set to 0.45 and 0.55 respectively.

3. The multi-target tracking method based on three-dimensional depth information decoupling according to claim 1, characterized in that, Step S4 is as follows: For the detection boxes obtained in the image at time t, OCR Mean-shift clustering is performed to classify dense occlusion and non-dense occlusion in the detection boxes. At the same time, OCR Mean-shift clustering is performed on the trajectory boxes in the image at time t-1 to classify dense occlusion and non-dense occlusion in the trajectory boxes. Among them, the trajectory information at time t-1 is known data, that is, each target in the image at time t-1 is represented by a bounding box and a unique ID number. The box with the ID is the trajectory box, and the three-dimensional depth information of the image at time t-1 is known. For all target bounding boxes detected at each time step, the coordinates of their center points are taken as sample points to be clustered, and the following expression is used to calculate the expression: ； ； ； in, , This represents a calculation factor used to measure whether a target itself is occluded and the relationships between targets within an occlusion clump. This represents a sample point to be clustered. Represents the Euclidean distance of the sample points The most recent sample point, This represents the class probability directly output by the classification prediction head. express Sample points and The IOU value between the sample points and the corresponding bounding boxes. express The area of ​​the bounding box corresponding to the sample point. This represents the improved sample point weights. The weighting factors for the two factors are fixed at [value]. ,and Represents the Gaussian kernel function. Indicates bandwidth, controls the scope of the kernel function. Used to perform mean-shift clustering and update drift points. Location.

Images