Multi-target tracking method based on neural Kalman filter and depth estimation network

Abstract

The invention discloses a multi-target tracking method based on a neural Kalman filter and a depth estimation network, on one hand, the neural Kalman filter is used for replacing a visual Transform network, the problem of overlarge calculation amount is improved, the reasoning speed is increased, on the other hand, depth information for describing a tracking target is added, and the tracking accuracy is improved. The performance of the tracking model is obviously improved; according to the method, the model calculation amount is reduced, the model reasoning speed is increased, meanwhile, continuous and excellent performance of the model is guaranteed, good balance of the tracking model in the aspects of speed and precision is achieved, and the deployment possibility of the high-performance multi-target tracking model on the edge terminal is further expanded.

Description

Technical Field

[0001] The invention relates to a multi-target tracking method based on a neural Kalman filter and a depth estimation network, and belongs to the technical field of computer vision. Background Art

[0002] Multi-target tracking is an important research area in computer vision, which involves detecting and tracking multiple moving objects simultaneously in a video. This process requires not only identifying the targets in the video frames, but also maintaining the identity consistency of these targets between consecutive frames, even in the case of occlusion between targets, changes in target appearance, or changes in camera perspective. Multi-target tracking technology plays an important role in many practical applications, such as intelligent monitoring systems, self-driving cars, sports analysis, human-computer interaction, etc. With the development of artificial intelligence technology, especially the advancement of deep learning, the performance of multi-target tracking algorithms has been significantly improved, which can better cope with the challenges in complex scenarios.

[0003] However, most of the current multi-target tracking methods have difficulty in achieving a balance between performance and speed due to the large amount of model calculations. These methods often require very high hardware conditions in practical applications and cannot be deployed on most terminal devices, which greatly limits the implementation of these multi-target tracking methods. Although some trackers with smaller models have fast inference speed, they have poor performance and cannot achieve a balance between performance and inference speed. Summary of the invention

[0004] The purpose of the present invention is to overcome the shortcomings of the prior art and provide a multi-target tracking method based on a neural Kalman filter and a depth estimation network to solve the problem in the prior art that it is difficult to balance the inference speed and performance of the tracker, including the fact that most of the existing multi-target tracking methods cannot be implemented on terminal devices due to the large number of parameters, which greatly limits the practical application of these algorithms, and some smaller trackers have fast inference speed but poor performance.

[0005] In order to solve the above technical problems, the present invention is implemented by adopting the following technical solutions:

[0006] In a first aspect, the present invention provides a multi-target tracking method based on a neural Kalman filter and a depth estimation network, which performs target tracking based on a given video, comprising the following steps:

[0007] Step 1: Use YOLOX to detect the target in the video frame by frame, obtain the target detection position, and obtain 2D clues;

[0008] Step 2: Use a deep neural regressor to estimate the distance between each target and the camera and obtain 3D positioning information;

[0009] Step 3: Use a neural network to dynamically adjust the process noise and observation noise of the Kalman filter according to the motion state of the target;

[0010] Step 4: Integrate the target motion state information obtained in steps 2 and 3 to obtain the target predicted position;

[0011] Step 5: Calculate the cost matrix based on the target detection position obtained in step 1 and the target prediction position obtained in step 4;

[0012] Step 6: Use the depth density estimator to combine the 2D clues obtained in step 1 with the 3D positioning information obtained in step 2, and correct and normalize the cost matrix obtained in step 5;

[0013] Step 7: Based on the normalized cost matrix, the Hungarian algorithm is used to complete the matching;

[0014] Step 8: Repeat steps 1-7 until the target tracking of the video is completed.

[0015] Furthermore, the step 1 specifically includes:

[0016] Use the trained YOLOX model to obtain the x-coordinate, y-coordinate, height h, and aspect ratio a of the target;

[0017] The YOLOX-S model is used for devices with average hardware performance, and the YOLOX-X model is used for devices with better hardware performance.

[0018] Furthermore, the step 2 specifically includes:

[0019] Train a deep neural regressor on the MOTSynth training set;

[0020] Use LSTM long short-term memory network to encode the historical information of all past frames, and its output is a single-frame feature map;

[0021] Using the Feature Pyramid Network (FPN), an additional branch is added to the deep network to restore the original resolution and avoid the loss of spatial information caused by downsampling;

[0022] The predicted distance is output through a multi-layer perceptron, and instead of directly outputting an accurate estimate, a Gaussian distribution is output to obtain the uncertainty of the model.

[0023] Furthermore, the step 3 specifically includes:

[0024] The observation noise R in the traditional Kalman filter k and process noise Q kInstead of fixed preset parameters, they are dynamically updated through two independent neural networks that adjust the noise parameters based on the current motion state.

[0025] R k =R-Net(z k )

[0026] Q k =Q-Net(x k-1 )

[0027] Z k Indicates the detection result of the current frame, x k-1 represents the state mean within the k-1 time length, R represents the covariance matrix of observation noise, which describes the influence of random errors in the observation model on the estimated value, Q represents the covariance matrix of process noise, which describes the influence of random errors in the state transfer model on the predicted value, and Net() represents the use of neural network to process Z k With x k-1 parameter;

[0028] These two networks are designed as fully connected layers.

[0029] Furthermore, the step 6 specifically includes:

[0030] Receive multiple inputs, including 2D cues and 3D positioning information;

[0031] Construct a conditional probability distribution P(D∈T|T) that estimates the probability of detection D belonging to trajectory T based on the input cost matrix and outputs the corrected cost matrix;

[0032] This cost matrix is ​​further normalized and smoothed by the softmax function.

[0033] Furthermore, the step 7 specifically includes:

[0034] First, match the high-score detection frame, calculate the IOU of the high-score frame and the predicted frame, and use the Hungarian algorithm to match the IOU to obtain three results: the matched trajectory and the high-score frame, the unsuccessfully matched trajectory and the unsuccessfully matched high-score frame. After the match is successful, update the frame in the tracking trajectory to the high-score detection frame;

[0035] Then, for the low-resolution frame, the IOU between the low-resolution frame and the prediction frame that was not matched in the previous step is calculated, and the Hungarian algorithm is used for matching to obtain three results: the matched trajectory and the low-resolution frame, the unsuccessfully matched trajectory, and the unsuccessfully matched low-resolution frame. After the match is successful, the frame in the tracking trajectory is updated to the detection frame;

[0036] Finally, for the unmatched high-scoring detection frames, they are matched with the inactive tracks to obtain three results: matching, unmatched tracks, and unmatched detection frames. Unmatched tracks are marked for deletion. For unmatched detection frames, if the confidence is higher than the threshold, they are updated to new tracking tracks, otherwise they are discarded.

[0037] In a second aspect, the present invention provides a general multi-target tracking device, comprising:

[0038] Target detection module: Two target detection models, YOLOX-X and YOLOX-S, are used. The former has a large number of parameters and better performance, and is suitable for deployment on devices with higher hardware levels. The latter has a small number of parameters and slightly worse performance, and is suitable for deployment on devices with average hardware levels.

[0039] Motion information encoding module: Use LSTM long short-term memory network to encode the historical information of all past frames, and its output is a single-frame feature map;

[0040] Feature recovery module: Using the feature pyramid network FPN, an additional branch is added to the deep network to restore the original resolution and avoid the loss of spatial information caused by downsampling;

[0041] Deep computing module: Integrate the motion information encoding module and feature recovery module into the same neural network, and add a multi-layer perceptron to the last layer of the network to output the predicted distance;

[0042] Motion estimation module: Dynamically updates the process noise and observation noise in the Kalman filter process through two independent neural networks, and predicts the current motion state based on the historical trajectory information of the target;

[0043] Cost matrix calculation module: calculates the cost matrix of the target detection module and the motion estimation module based on the detection information obtained by the target detection module and the prediction information obtained by the motion estimation module;

[0044] Depth density estimation module: integrates the 2D clues obtained by the motion estimation module with the 3D positioning information obtained by the depth calculation module, corrects and normalizes the cost matrix, and smoothes it through the softmax function;

[0045] Association matching module: According to the cost matrix, the detection frame is matched with the trajectory. By matching the high-score detection frame first and then the low-score detection frame, the matching accuracy is effectively improved.

[0046] In a third aspect, the present invention provides a terminal device, the terminal including a processor and a storage medium;

[0047] The storage medium is used to store instructions;

[0048] The processor executes the steps corresponding to the method described in the first aspect according to these instructions.

[0049] In a fourth aspect, the present invention further relates to a computer-readable storage medium storing a computer program, wherein the computer program is characterized in that when the program is executed by a processor, the steps of the method described in the first aspect can be implemented.

[0050] Compared with the prior art, the present invention has the following beneficial effects:

[0051] 1. While reducing the number of model parameters and accelerating the model inference speed, the present invention ensures the excellent level of model performance, achieves a good balance between the performance and inference speed of the multi-target tracking algorithm, and further expands the possibility of deploying the multi-target tracking model on the terminal;

[0052] 2. The present invention designs a neural Kalman filter and a depth estimation network. The neural Kalman filter uses a neural network to replace the traditional Kalman filter as a motion model, which improves the tracking accuracy without sacrificing the reasoning speed. The depth estimation network provides a more reliable cost matrix for the matching stage by calculating the depth information of the target, which greatly improves the tracking accuracy. The present invention does not adopt the traditional Transformer-based multi-target tracking method, but uses a simple and efficient YOLOX model as a detector, in conjunction with a neural Kalman filter and a depth estimation network, to achieve a real-time and accurate multi-target tracking method at the terminal. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] Figure 1 It is a flowchart of a multi-target tracking method based on a neural Kalman filter and a depth estimation network provided according to an embodiment of the present invention;

[0054] Figure 2 is a schematic diagram of a network framework of a multi-target tracking method based on a neural Kalman filter and a depth estimation network provided according to an embodiment of the present invention;

[0055] Figure 3 is a schematic diagram of the structure of a deep neural regressor provided according to an embodiment of the present invention;

[0056] Figure 4 is a schematic diagram of the structure of a depth density estimator provided according to an embodiment of the present invention. DETAILED DESCRIPTION

[0057] The technical solution of the present invention is described in detail below through the accompanying drawings and specific embodiments. It should be understood that the embodiments of the present application and the specific features in the embodiments are detailed descriptions of the technical solution of the present application, rather than limitations on the technical solution of the present application. In the absence of conflict, the embodiments of the present application and the technical features in the embodiments can be combined with each other.

[0058] The term "and / or" in this article is only a description of the association relationship of the associated objects, indicating that there can be three relationships. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship.

[0059] Embodiment 1:

[0060] like Figure 1-4 As shown, the present invention provides a target tracking method based on multiple background noise filtering, and performs target tracking based on a given video, comprising the following steps:

[0061] Step 1: Use YOLOX to detect the target in the video frame by frame, obtain the target detection position, and obtain 2D clues;

[0062] Step 2: Use a deep neural regressor to estimate the distance between each target and the camera and obtain 3D positioning information;

[0063] Step 3: Use a neural network to dynamically adjust the process noise and observation noise of the Kalman filter according to the motion state of the target;

[0064] Step 4: Integrate the target motion state information obtained in steps 2 and 3 to obtain the target predicted position;

[0065] Step 5: Calculate the cost matrix based on the target detection position obtained in step 1 and the target prediction position obtained in step 4;

[0066] Step 6: Use the depth density estimator to combine the 2D clues obtained in step 1 with the 3D positioning information obtained in step 2, and correct and normalize the cost matrix obtained in step 5;

[0067] Step 7: Based on the normalized cost matrix, the Hungarian algorithm is used to complete the matching;

[0068] Step 8: Repeat steps 1-7 until the target tracking of the video is completed.

[0069] Specifically, first use the YOLOX (two models, X and S, are trained. The former is suitable for terminal devices with high hardware performance, and the latter is suitable for terminal devices with general performance) target detector to extract all targets in the current frame of the input video. According to the pre-set threshold θ, the detection results are divided into low-score and high-score. The high-score detection frame will be combined with the prediction result of the motion trajectory of the current frame output by the neural Kalman filter to generate a cost matrix based on IOU (intersection-over-union). The low-score detection frame will be used for the subsequent second matching. The function of the deep neural regressor is to estimate the depth information of the target in the video. The input received is the undetected video image. First, the video image is encoded by the ResNet neural network (a widely used neural network), and then the LSTM long short-term memory network (a basic neural network, usually used to process data with long-term relationships in time) is used to extract the long-term motion information of the target, and finally the depth information of each target from the camera in the video is obtained. The cost matrix and the depth information are simultaneously input into the depth density estimator, and a more reliable normalized cost matrix is ​​obtained through normalization, standardization, softmax (a common activation function in the field of deep learning) smoothing and other operations. Based on this normalized cost matrix, the Hungarian algorithm is used to start the first track matching. The first matching high-score detection will obtain three results: the matched track and the high-score frame, the unsuccessfully matched track and the unsuccessfully matched high-score frame. After the match is successful, the frame in the track is updated to the high-score detection frame; the second matching is for the low-score frame, and the IOU of the low-score frame and the prediction frame that was not matched in the previous step is calculated to obtain three results: the matched track and the low-score frame, the unsuccessfully matched track and the unsuccessfully matched low-score frame. After the match is successful, the frame in the track is updated to the detection frame; finally, for the unmatched high-score detection frame, it is matched with the track in the inactive state, and three results are obtained: matching, unmatched track and unmatched detection frame. The unmatched track is marked for deletion. For the unmatched detection frame, if the confidence is higher than the threshold, it is updated to a new tracking track. If it is less than the threshold, it is considered an invalid track and will be discarded. At this point, the tracking process of all targets in the current frame is completed.

[0070] Compared with the prior art, the present invention reduces the number of model parameters and speeds up the model inference while ensuring the excellent level of model performance, achieving a good balance between the performance and inference speed of the multi-target tracking algorithm, and further expanding the possibility of deploying the multi-target tracking model on the terminal;

[0071] The present invention designs a neural Kalman filter and a depth estimation network. The neural Kalman filter uses a neural network to replace the traditional Kalman filter as a motion model, which improves the tracking accuracy without sacrificing the reasoning speed. The depth estimation network provides a more reliable cost matrix for the matching stage by calculating the depth information of the target, which greatly improves the tracking accuracy. The present invention does not adopt the traditional Transformer-based multi-target tracking method, but uses a simple and efficient YOLOX model as a detector, in conjunction with a neural Kalman filter and a depth estimation network, to achieve a real-time and accurate multi-target tracking method at the terminal.

[0072] In one embodiment, YOLOX is used to detect the target in the video frame by frame, and obtaining the target position includes:

[0073] Use the trained YOLOX model to obtain the x-coordinate, y-coordinate, height h, and aspect ratio a of the target;

[0074] The YOLOX-S model is used for devices with average hardware performance, and the YOLOX-X model is used for devices with better hardware performance.

[0075] Specifically, the YOLOX target detector will re-crop the input image to a resolution of 640*640, and after passing through its own Backbone, Neck, and Prediction modules, it will output a detection result, which is represented by the target's x-coordinate, y-coordinate, height h, and aspect ratio a. Although the YOLOX-S model requires slightly lower hardware performance, its tracking effect is not significantly reduced compared to the YOLOX-X model.

[0076] like Figure 3 As shown in an embodiment, the use of a deep neural regressor to estimate the distance between each pedestrian and the camera based on a given video includes:

[0077] Train a deep neural regressor on the MOTSynth training set;

[0078] Use LSTM long short-term memory network to encode the historical information of all past frames, and its output is a single-frame feature map;

[0079] Using the Feature Pyramid Network (FPN), an additional branch is added to the deep network to restore the original resolution and avoid the loss of spatial information caused by downsampling;

[0080] The predicted distance is output through a multi-layer perceptron, and instead of directly outputting an accurate estimate, a Gaussian distribution is output to obtain the uncertainty of the model.

[0081] The process noise and observation noise of the Kalman filter are dynamically adjusted according to the motion state of the target using a neural network, including:

[0082] The observation noise R in the traditional Kalman filter k and process noise Q k Instead of fixed preset parameters, they are dynamically updated through two independent neural networks that adjust the noise parameters based on the current motion state.

[0083] R k =R-Net(z k )

[0084] Q k =Q-Net(x k-1 )

[0085] Z k Indicates the detection result of the current frame, x k-1 represents the state mean within the k-1 time length, R represents the covariance matrix of observation noise, which describes the influence of random errors in the observation model on the estimated value, Q represents the covariance matrix of process noise, which describes the influence of random errors in the state transfer model on the predicted value, and Net() represents the use of neural network to process Z k With x k-1 parameter;

[0086] These two networks are designed as fully connected layers.

[0087] Specifically, the traditional Kalman filter is a two-stage process: state prediction and state update. The Kalman gain K k It is designed to balance the weights of estimates and observations. Specifically, the Kalman gain at the kth moment is calculated as follows:

[0088]

[0089] Here H k is the observation model, R k is the observation noise covariance matrix, P k ′ is the state covariance predicted by the following formula:

[0090]

[0091] where F k is the state transition model, Q k is the process noise covariance matrix, and P k-1 is the state covariance of the previous moment. In traditional multi-target trackers, R k and Q kThey are usually determined based on preset parameters, which makes them difficult to adapt to various scenarios. To overcome this limitation, the neural Kalman filter introduces two neural networks to dynamically update R based on the current motion state. k and Q k :

[0092] R k =R-Net(z k )

[0093] Q k =Q-Net(x k-1 )

[0094] In this way, the Neural Kalman Filter is able to dynamically adjust its parameters in different tracking situations, thus providing more accurate tracking results.

[0095] like Figure 4 As shown, in one embodiment, the use of a depth density estimator to correct and normalize the cost matrix obtained in step f by using the 2D clues obtained in step a and the 3D positioning information obtained in step b includes:

[0096] Receive multiple inputs, including 2D cues and 3D positioning information;

[0097] Construct a conditional probability distribution P(D∈T|T) that estimates the probability of detection D belonging to trajectory T based on the input cost matrix and outputs the corrected cost matrix;

[0098] This cost matrix is ​​further normalized and smoothed by the softmax function.

[0099] Specifically, the cost matrix and depth information are simultaneously input into the depth density estimator, and a more reliable normalized cost matrix is ​​obtained through normalization, standardization, softmax smoothing and other operations.

[0100] The present invention divides the process of association matching into multiple stages, and combines the 2D information obtained by the target detector with the 3D depth information obtained by the deep neural regressor to obtain a more accurate matching effect. Using the Hungarian algorithm, first match the high-score detection frame, and you will get three results: the matched trajectory and the high-score frame, the unsuccessfully matched trajectory, and the unsuccessfully matched high-score frame. After the match is successful, the frame in the tracking trajectory is updated to the high-score detection frame; the second match is for the low-score frame, and the IOU of the low-score frame and the prediction frame that was not matched in the previous step is calculated to obtain three results: the matched trajectory and the low-score frame, the unsuccessfully matched trajectory, and the unsuccessfully matched low-score frame. After the match is successful, the frame in the tracking trajectory is updated to the detection frame; finally, for the unmatched high-score detection frame, it is matched with the trajectory that is not activated, and three results are also obtained: matching, unmatched trajectory, and unmatched detection frame. The unmatched trajectory is marked as deleted. For the unmatched detection frame, if the confidence is higher than the threshold, it is updated to a new tracking trajectory. If it is less than the threshold, it is regarded as an invalid trajectory and will be discarded. At this point, the tracking process of all targets in the current frame is completed.

[0101] Embodiment 2:

[0102] The present invention provides a universal multi-target tracking device, which can be applied to different multi-target tracking methods separately, and the device comprises:

[0103] Target detection module: Two target detection models, YOLOX-X and YOLOX-S, are used. The former has a large number of parameters and better performance, and is suitable for deployment on devices with higher hardware levels. The latter has a small number of parameters and slightly worse performance, and is suitable for deployment on devices with average hardware levels.

[0104] Motion information encoding module: Use LSTM long short-term memory network to encode the historical information of all past frames, and its output is a single-frame feature map

[0105] Feature recovery module: Using the feature pyramid network FPN, an additional branch is added to the deep network to restore the original resolution and avoid the loss of spatial information caused by downsampling;

[0106] Deep computing module: Integrate the motion information encoding module and feature recovery module into the same neural network, and add a multi-layer perceptron to the last layer of the network to output the predicted distance;

[0107] Motion estimation module: Dynamically updates the process noise and observation noise in the Kalman filter process through two independent neural networks, and predicts the current motion state based on the historical trajectory information of the target;

[0108] Cost matrix calculation module: calculates the cost matrix of the target detection module and the motion estimation module based on the detection information obtained by the target detection module and the prediction information obtained by the motion estimation module;

[0109] Depth density estimation module: integrates the 2D clues obtained by the motion estimation module with the 3D positioning information obtained by the depth calculation module, corrects and normalizes the cost matrix, and smoothes it through the softmax function;

[0110] Association matching module: According to the cost matrix, the detection frame is matched with the trajectory. By matching the high-score detection frame first and then the low-score detection frame, the matching accuracy is effectively improved;

[0111] Embodiment three:

[0112] An embodiment of the present invention further provides a terminal device, the device comprising a processor and a storage medium;

[0113] The storage medium is used to store instructions;

[0114] The processor operates according to these instructions to implement the various steps of the method described in the first embodiment.

[0115] Embodiment 4:

[0116] The embodiments of the present invention further provide a computer-readable storage medium, in which computer programs are stored, and when these programs are executed by a processor, the method steps described in the above embodiments can be implemented. Because the storage medium can run the method of the above embodiments, it also has the functional modules and positive effects brought by executing the method.

[0117] Those skilled in the art will appreciate that the present application may be provided as a method, system, or computer program product. Therefore, the present application may be provided in the form of complete hardware, complete software, or a combination of hardware and software. In addition, the present application may also be implemented as a computer program product on a computer-readable storage medium containing computer program code, such storage medium may include but is not limited to a disk storage, a CD-ROM, an optical storage, etc.

[0118] The present application is described in accordance with the flowcharts and / or block diagrams of the methods, devices (systems) and computer program products of the embodiments. It should be understood that each process and / or block diagram in the flowchart and / or block diagram, as well as the combination of the processes and block diagrams in the flowchart and / or block diagram, can be implemented by computer program instructions. These instructions can be provided to a processor of a general-purpose computer, a special-purpose computer or other programmable data processing device, thereby creating a machine that, by executing these instructions, forms a device that implements the functions specified in one or more processes in the flowchart and / or one or more block diagrams in the block diagram.

[0119] These instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the memory constitute a product that includes instruction means for implementing the above functions.

[0120] These instructions can also be loaded onto a computer or other programmable data processing device to perform processing through a series of operating steps, so that the instructions executed on the computer or other programmable device provide steps to implement the functions specified in one or more processes in the flowchart and / or one or more block diagrams in the block diagram.

[0121] The above contents are only preferred embodiments of the present invention. It should be pointed out that, for ordinary technicians in this technical field, various improvements and changes can be made to it without departing from the technical principles of the present invention. These improvements and changes should also be regarded as the protection scope of the present invention.

Claims

1. A multi-target tracking method based on neural Kalman filter and depth estimation network, characterized in that: Target tracking based on a given video includes the following steps: Step 1: Use YOLOX to detect the target in the video frame by frame, obtain the target detection position, and obtain 2D clues; Step 2: Use a deep neural regressor to estimate the distance between each target and the camera and obtain 3D positioning information; Step 3: Use a neural network to dynamically adjust the process noise and observation noise of the Kalman filter according to the motion state of the target; Step 4: Integrate the target motion state information obtained in steps 2 and 3 to obtain the target predicted position; Step 5: Calculate the cost matrix based on the target detection position obtained in step 1 and the target prediction position obtained in step 4; Step 6: Use the depth density estimator to combine the 2D clues obtained in step 1 with the 3D positioning information obtained in step 2, and correct and normalize the cost matrix obtained in step 5; Step 7: Based on the normalized cost matrix, the Hungarian algorithm is used to complete the matching; Step 8: Repeat steps 1-7 until the target tracking of the video is completed.

2. The multi-target tracking method based on neural Kalman filter and depth estimation network according to claim 1 is characterized in that: The step 1 specifically includes: Use the trained YOLOX model to obtain the x-coordinate, y-coordinate, height h, and aspect ratio a of the target; The YOLOX-S model is used for devices with average hardware performance, and the YOLOX-X model is used for devices with better hardware performance.

3. The multi-target tracking method based on neural Kalman filter and depth estimation network according to claim 1 is characterized in that: The step 2 specifically includes: Train a deep neural regressor on the MOTSynth training set; Use LSTM network to encode the historical information of all past frames, and its output is a single-frame feature map; Using the Feature Pyramid Network (FPN), an additional branch is added to the deep network to restore the original resolution and avoid the loss of spatial information caused by downsampling; The predicted distance is output by a multi-layer perceptron, and the output is a Gaussian distribution, so as to obtain the uncertainty of the model.

4. The multi-target tracking method based on neural Kalman filter and depth estimation network according to claim 1 is characterized in that: The step 3 specifically includes: Observation Noise R in Kalman Filter k and process noise Q k It is dynamically updated by two independent neural networks, which adjust the noise parameters based on the current motion state. R k =R-Net(z k ) Q k =Q-Net(x k-1 ) Z k Indicates the detection result of the current frame, x k-1 represents the state mean within the k-1 time length, R represents the covariance matrix of observation noise, which describes the influence of random errors in the observation model on the estimated value, Q represents the covariance matrix of process noise, which describes the influence of random errors in the state transfer model on the predicted value, and Net() represents the use of neural network to process Z k With x k-1 parameter; These two networks are designed as fully connected layers.

5. The multi-target tracking method based on neural Kalman filter and depth estimation network according to claim 1 is characterized in that: The step 6 specifically includes: Receive multiple inputs, including 2D cues and 3D positioning information; Construct a conditional probability distribution P(D∈T|T) based on the input cost matrix, estimate the probability of detection D belonging to trajectory T, and output the corrected cost matrix; This cost matrix is ​​further normalized and smoothed by the softmax function.

6. The multi-target tracking method based on neural Kalman filter and depth estimation network according to claim 1 is characterized in that: The step 7 specifically includes: First, match the high-score detection frame, calculate the IOU of the high-score frame and the predicted frame, and use the Hungarian algorithm to match the IOU to obtain three results: the matched trajectory and the high-score frame, the unsuccessfully matched trajectory and the unsuccessfully matched high-score frame. After the match is successful, update the frame in the tracking trajectory to the high-score detection frame; Then, for the low-resolution frame, the IOU between the low-resolution frame and the prediction frame that was not matched in the previous step is calculated, and the Hungarian algorithm is used for matching to obtain three results: the matched trajectory and the low-resolution frame, the unsuccessfully matched trajectory and the unsuccessfully matched low-resolution frame. After the match is successful, the frame in the tracking trajectory is updated to the detection frame; Finally, for the unmatched high-scoring detection frames, they are matched with the inactive trajectories to obtain three results: matching, unmatched trajectories, and unmatched detection frames. The unmatched trajectories are marked for deletion. For the unmatched detection frames, if the confidence is higher than the threshold, they are updated to the new tracking trajectory, otherwise they are discarded.

7. A multi-target tracking device based on a neural Kalman filter and a depth estimation network, characterized in that: include: Target detection module: Two target detection models, YOLOX-X and YOLOX-S, are used. The former has a large number of parameters and better performance, and is suitable for deployment on devices with higher hardware levels. The latter has a small number of parameters and slightly worse performance, and is suitable for deployment on devices with average hardware levels. Motion information encoding module: Use LSTM long short-term memory network to encode the historical information of all past frames, and its output is a single-frame feature map; Feature recovery module: Using the Feature Pyramid Network (FPN), an additional branch is added to the deep network to restore the original resolution and avoid the loss of spatial information caused by downsampling; Deep computing module: Integrate the motion information encoding module and feature recovery module into the same neural network, and add a multi-layer perceptron to the last layer of the network to output the predicted distance; Motion estimation module: Dynamically updates the process noise and observation noise in the Kalman filter process through two independent neural networks, and predicts the current motion state based on the historical trajectory information of the target; Cost matrix calculation module: calculates the cost matrix of the target detection module and the motion estimation module based on the detection information obtained by the target detection module and the prediction information obtained by the motion estimation module; Depth density estimation module: integrates the 2D clues obtained by the motion estimation module with the 3D positioning information obtained by the depth calculation module, corrects and normalizes the cost matrix, and smoothes it through the softmax function; Association matching module: According to the cost matrix, the detection frame is matched with the trajectory. By matching the high-score detection frame first and then the low-score detection frame, the matching accuracy is effectively improved.

8. A terminal, characterized in that: including processor and storage medium; The storage medium is used to store instructions; The processor is used to operate according to the instructions to execute the steps of the method according to any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images