Multi-target tracking method based on binocular camera combined with pseudo depth estimation

Abstract

The invention relates to a multi-target tracking method based on a binocular camera combined with pseudo depth estimation, and belongs to the technical field of computer vision. Video images are obtained through a binocular camera and preprocessed, and grouped common images and depth image sequences are obtained. Obtaining a target bounding box of the common image by using the target detection model, and dividing the target bounding box into a high-score bounding box and a low-score bounding box according to the confidence coefficient; and a complete target detection bounding box is obtained through a depth fusion strategy in combination with pseudo depth estimation of common image information and depth image information. Secondly, predicting a trajectory bounding box by adopting a motion predictor based on a diffusion model; in the matching process, through a hierarchical cascade matching strategy, an IoU distance and a Hungary algorithm are used for correlation matching, and a target track is updated. According to the invention, through depth information fusion and hierarchical cascade matching, the tracking accuracy under the conditions of target shielding and congestion is improved, the complexity of the method is simplified, and actual deployment is easier.

Description

Technical Field

[0001] This invention belongs to the field of computer vision technology and relates to a multi-object tracking method based on binocular cameras combined with pseudo-depth estimation. Background Technology

[0002] Multi-object tracking is a classic subtask in computer vision, with wide applications in autonomous driving, intelligent security, and sports event analysis. Its purpose is to locate and track multiple targets in consecutive video frames, providing accurate trajectories for subsequent tasks. Mainstream multi-object tracking methods follow a detection-based tracking framework. First, a target detection module identifies all targets in the video frame and obtains their bounding boxes. Then, the problem is transformed into solving the object association problem between consecutive frames. However, the many complex characteristics of real-world applications present numerous challenges to existing multi-object tracking methods.

[0003] In practical applications, especially in complex scenarios such as sports, targets often have similar appearances, complex movement patterns, and serious occlusion problems. In addition, the shooting equipment and conditions (such as shooting distance and lighting conditions) also affect the accuracy of the detector. These problems can significantly affect the overall performance of the tracking method.

[0004] With the rapid improvement of deep learning detector performance, research in multi-object tracking mainly focuses on achieving high accuracy and robustness in association. Depth information, as an effective parameter for determining the positional differences of 2D objects in 3D reality, has been introduced into multi-object tracking methods in recent years. SparseTrack and PD-SORT calculate pseudo-depth values ​​of targets based on the bounding box information output by the detection module and trajectory prediction. During the target association stage, different depth levels are divided by the range of pseudo-depth values, and association matching is performed separately at each depth level. This effectively reduces problems such as ID switching and trajectory fragmentation caused by target congestion and occlusion during tracking in simple scenarios (monitoring pedestrians, vehicles, etc.), thus improving tracking accuracy. However, these methods are all based on the assumption that the image is taken from a top-down angle and the target is on a flat surface. The pseudo-depth value is obtained by calculating the difference between the image height and the bottom ordinate of the target, which is not the true depth information in 3D space, resulting in insufficient robustness. In scenarios with severe occlusion issues, such as sports events, the crowding and occlusion of targets can lead to multiple targets with different front and back positions having the same pseudo-depth value. This significantly increases the number of ID switching during association matching and the occurrence of trajectory mismatch, reducing the overall tracking accuracy. It cannot cope with complex motion patterns and frequent occlusion.

[0005] In summary, existing multi-target tracking technologies exhibit poor robustness and low accuracy when dealing with scenarios involving complex target motion patterns and severe occlusion, posing challenges to the application of multi-target tracking technologies in real-world scenarios. Summary of the Invention

[0006] In view of this, the purpose of this invention is to provide a multi-target tracking method based on binocular camera combined with pseudo-depth estimation, which fuses binocular depth information and pseudo-depth information and introduces it into the trajectory prediction and association stage of multi-target tracking tasks, so as to solve the problems existing in the prior art mentioned in the background art above.

[0007] To achieve the above objectives, the present invention provides the following technical solution:

[0008] A multi-object tracking method based on binocular cameras and pseudo-depth estimation includes the following steps:

[0009] Video images are acquired using a binocular camera, and after preprocessing, a sequence of images is obtained from groups of all video frames. Each group of images includes a normal image and a depth image corresponding to the video frame.

[0010] Input ordinary images from each image sequence into the object detection model to obtain bounding boxes (x, y, w, h, s) of all objects in the ordinary images; classify the object bounding boxes into high-scoring bounding boxes and low-scoring bounding boxes based on the confidence s of the bounding boxes.

[0011] The pseudo-depth information value d of the target is obtained based on the target bounding box information and the height of the corresponding ordinary image. p And obtain the target stereo depth information value d based on its depth image. s A deep fusion strategy is used to perform deep information fusion to obtain the fused target depth information value d, and finally obtain the complete target detection bounding box (x,y,w,h,d). detect ;

[0012] A motion predictor based on a diffusion model is used to predict the bounding box (x, y, w, h, d) of the current frame trajectory based on the target detection bounding boxes of each historical frame for each tracking trajectory. track ;

[0013] Define the target detection bounding box (x, y, w, h, d). detect With the predicted trajectory bounding box (x,y,w,h,d) track The target trajectory is updated by hierarchical cascaded matching to obtain the tracking result of the current frame. When the association matching of all frames is completed, the complete multi-target tracking result is obtained.

[0014] Furthermore, during the preprocessing of video images acquired by the stereo camera, the acquired video frames are filtered and depth normalized to obtain a set of image sequences, which are arranged in chronological order.

[0015] Filtering refers to the process of using weighted median filtering combined with bilateral filtering to address depth anomalies caused by motion blur and local texture loss, thereby preserving edge details while suppressing noise.

[0016] Depth normalization is a process that linearly maps pixel values ​​in a depth map to the same numerical range.

[0017] Furthermore, the detection bounding boxes (x, y, w, h, s) of all targets in a normal image are obtained through the target detection model. The target detection model is pre-trained on a targeted scene dataset, where x and y represent the coordinates of the center point of the bounding box, the origin of the coordinates is the top left corner vertex of the image, w and h represent the width and height of the bounding box, and s represents the detection confidence value of the bounding box.

[0018] Then, the confidence value s of the bounding box is compared with the preset confidence threshold τ. det The bounding boxes are compared, and those with scores greater than a preset threshold are considered high-scoring bounding boxes, while those with scores less than the preset threshold are considered low-scoring bounding boxes.

[0019] Furthermore, obtain the complete object detection bounding box (x, y, w, h, d). detect The process is as follows:

[0020] The target pseudo-depth information value d is calculated based on the obtained target bounding box information (y, h) and the height of the ordinary image. p The calculation method is as follows:

[0021] d p =Hy b

[0022] Where H represents the height of the current image, y b The vertical coordinate representing the bottom of the current target bounding box is obtained from the bounding box position coordinates mentioned above, and its calculation method is as follows:

[0023]

[0024] Then, based on the acquired target bounding box information (x, y, w, h), the depth image is cropped to obtain a local depth map of the target region. The depth pixel value at the bottom center of the target is the depth information value d. s ;

[0025] Deep information fusion is performed using a deep fusion strategy to obtain the fused target depth information value d:

[0026]

[0027] Among them, v i τ represents the standard deviation of depth pixel values ​​within the bottom region of the target bounding box in the depth map. noise This represents the noise threshold, used to determine the depth confidence level of the region. i Greater than τ noise This indicates that the depth confidence in this area is low, and it needs to be combined with a pseudo-depth value d. p ;r valid τ represents the percentage of valid pixel values ​​in the entire region corresponding to the target bounding box in the depth map. valid This represents the effective threshold, used to determine whether the depth of the target region is missing. When r valid Less than τ valid This indicates that the target region lacks depth and needs to be combined with a pseudo-depth value d. p α∈[0,1], representing dynamic weights, determined by σ i and r valid Dynamic adjustment, the calculation method is as follows:

[0028]

[0029] The obtained deep fusion information d is combined with the previous target bounding box to obtain the complete detection bounding box (x, y, w, h, d). detect .

[0030] Furthermore, the motion predictor based on the diffusion model predicts the bounding box (x,y,w,h,d) of the predicted trajectory for the current frame. track The process is as follows:

[0031] C contains the bounding box information of the historical frames of each tracking trajectory. f The input is fed into a motion predictor based on a diffusion model; where the trajectory of a target is represented as T = {B1,…,B}. f ,…,B N The initial trajectory is the bounding box information of the target in frame 0, f is the frame index, N is the total number of frames, and B... f =(x f ,y f ,w f ,h f ,d f This represents the complete bounding box information of the target, showing the center point coordinates, the height and width of the bounding box, and the depth fusion information, which will show the target's motion M. f Defined as the difference in bounding box information between two consecutive frames, it is expressed as:

[0032] M f =B f -B f-1 =(Δx) f ,Δyf ,Δw f ,Δh f ,Δd f )

[0033] Complete target frame information I f Represented as a 10-dimensional vector:

[0034] I f = (B f M f )=(x f ,y f ,w f ,h f ,d f ,Δx f ,Δy f ,Δw f ,Δh f ,Δd f )

[0035] The target's historical frame information C is input into the diffusion-based motion predictor. f Represented as a vector matrix:

[0036]

[0037] Where n represents the number of historical frames of the target trajectory, and the above historical frame information C f The data is fed into the motion predictor for diffusion prediction to obtain the bounding box (x, y, w, h, d) of the current frame trajectory. track .

[0038] Furthermore, the hierarchical cascade matching process includes:

[0039] Based on the deep fusion information values ​​in the detected bounding boxes, all targets in each frame of the image are hierarchically divided to distinguish different targets in occluded and crowded scenes;

[0040] Based on the k different depth levels, the detection bounding box sets D at different depth levels are obtained. sub and trajectory bounding box set T sub The value of k will be set separately depending on the different situations;

[0041] Based on the obtained detection sets D at different depth levels sub and trajectory set T sub Based on IoU distance and Hungarian algorithm, association matching is performed at the same depth level. If the target detection bounding box and the predicted trajectory bounding box fail to match at the current depth level, they will enter the association process at the next depth level.

[0042] After all targets in the current frame have been matched, all unmatched high-resolution bounding boxes will be matched with the bounding boxes of the newly appearing trajectories in the previous frame; high-resolution bounding boxes that are still unmatched will be initialized as new target tracking trajectories.

[0043] All unmatched low-resolution bounding boxes will be associated with unmatched trajectory bounding boxes from the previous frame for matching; low-resolution bounding boxes that still do not match will be treated as background deletion.

[0044] Unmatched trajectory bounding boxes are retained and participate in the association matching of the next frame, exceeding the retention threshold τ. lost If the matching is still not completed, the corresponding trajectory will be deleted.

[0045] Finally, the high and low bounding boxes and the low bounding box matched in the previous steps are added to the target trajectory corresponding to the trajectory bounding box that matches them, the target tracking trajectory is updated, and the tracking result of the current frame is obtained. When the association matching of all frames is completed, the complete multi-target tracking result is obtained.

[0046] Furthermore, the specific method of hierarchical division is as follows:

[0047] Detect the bounding box (x, y, w, h, d) based on the target. detect The depth fusion information d in the set obtains the minimum depth value MIN. d and the maximum depth value MAX d According to MIN d and MAX d The interval between them is evenly divided into k different depth intervals, which serve as k different depth levels;

[0048] Meanwhile, based on the predicted trajectory bounding box (x,y,w,h,d) track The depth fusion information in the set obtains the minimum depth value MIN. t and the maximum depth value MAX t According to MIN t and MAX t The distance between the intervals is evenly divided into k different depth intervals, which serve as k different depth levels.

[0049] Furthermore, the process of association matching based on IoU distance and the Hungarian algorithm at the same depth level is as follows:

[0050] D sub High-resolution bounding boxes and T based on detection confidence partitioning sub During association matching, the IoU distance between bounding boxes is calculated to obtain a similarity matrix. Then, the Hungarian algorithm is used to complete the matching based on the similarity matrix. Unmatched high-scoring detection bounding boxes and trajectory bounding boxes will be retained.

[0051] D sub The low-scoring bounding boxes based on detection confidence are associated with the unmatched trajectory bounding boxes from the previous step. Similarly, during the association process, the IoU distance between the bounding boxes is calculated to obtain the similarity matrix. Then, the Hungarian algorithm is used to complete the matching based on the similarity matrix. The unmatched low-scoring bounding boxes and trajectory bounding boxes will be retained.

[0052] The IoU distance measures the similarity between two bounding boxes by calculating the ratio of their overlapping region to their union region. The calculation method is as follows:

[0053]

[0054] s union =w d h d +w t h t -s overlap

[0055] Among them, s overlap s represents the area of ​​the overlapping region of the two bounding boxes. union Represents the area of ​​the union of two bounding boxes; (x d ,y d ,w d ,h d (x) represents the center point coordinates, width, and height of the detection bounding box. t ,y t ,w t ,h t () represents the center point coordinates, width, and height of the trajectory bounding box;

[0056] High-resolution bounding boxes, low-resolution bounding boxes, and trajectory bounding boxes that are not matched at the current depth level participate in the association matching at the next depth level.

[0057] On the other hand, a computer-readable storage medium is also proposed, on which a computer program is stored. When the computer program is executed by a processor, it implements the aforementioned multi-target tracking method based on binocular cameras combined with pseudo-depth estimation.

[0058] On the other hand, a computer program product is also proposed, including a computer program that, when executed by a processor, implements the aforementioned multi-target tracking method based on binocular cameras combined with pseudo-depth estimation.

[0059] The beneficial effects of this invention are as follows:

[0060] First, this invention organically fuses binocular depth information with pseudo-depth information obtained through pseudo-depth estimation, and combines this with a depth hierarchical cascade matching strategy to significantly enhance the robustness of target trajectory association. This innovation effectively improves tracking accuracy under target occlusion and crowded conditions, and performs particularly well in multi-target tracking in complex scenes.

[0061] Secondly, this invention cleverly integrates deep fusion information into a diffusion-based predictor for motion modeling and trajectory prediction. This approach enables the system to predict the trajectory state of targets more effectively, especially for targets with complex motion patterns, significantly improving prediction accuracy.

[0062] Finally, in the trajectory association matching process, this invention only uses an IoU-based method to calculate bounding box similarity, avoiding the introduction of additional appearance features, thereby simplifying the method's complexity and making the system easier to deploy and apply in practice. These innovations collectively drive the development of multi-target tracking technology.

[0063] Other advantages, objectives, and features of the invention will be set forth in part in the description which follows, and in part will be apparent to those skilled in the art from the following examination, or may be learned from practice of the invention. The objectives and other advantages of the invention can be realized and obtained through the following description. Attached Figure Description

[0064] To make the objectives, technical solutions, and advantages of the present invention clearer, the preferred embodiments of the present invention will be described in detail below with reference to the accompanying drawings, wherein:

[0065] Figure 1 This is a flowchart illustrating the multi-target tracking method based on binocular camera combined with pseudo-depth estimation according to an embodiment of the present invention.

[0066] Figure 2 This is a flowchart illustrating the cascading matching strategy according to an embodiment of the present invention;

[0067] Figure 3 This is a schematic diagram of the IoU distance in an embodiment of the present invention. Detailed Implementation

[0068] The following specific embodiments illustrate the implementation of the present invention. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and various details in this specification can be modified or changed based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the illustrations provided in the following embodiments are only schematic representations of the basic concept of the present invention. Unless otherwise specified, the following embodiments and features can be combined with each other.

[0069] The accompanying drawings are for illustrative purposes only and are schematic diagrams, not actual pictures. They should not be construed as limiting the invention. To better illustrate the embodiments of the invention, some parts in the drawings may be omitted, enlarged, or reduced, and do not represent the actual product dimensions. It is understandable to those skilled in the art that some well-known structures and their descriptions may be omitted in the drawings.

[0070] In the accompanying drawings of the embodiments of the present invention, the same or similar reference numerals correspond to the same or similar components. In the description of the present invention, it should be understood that if terms such as "upper," "lower," "left," "right," "front," and "rear" indicate the orientation or positional relationship based on the orientation or positional relationship shown in the drawings, they are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, the terms used to describe positional relationships in the drawings are only for illustrative purposes and should not be construed as limiting the present invention. For those skilled in the art, the specific meaning of the above terms can be understood according to the specific circumstances.

[0071] Please see Figures 1-3 This is a multi-target tracking method based on binocular cameras combined with pseudo-depth estimation.

[0072] In this embodiment, Figure 1 This diagram illustrates the flowchart of the multi-target tracking method based on binocular cameras and pseudo-depth estimation provided by the present invention. Figure 1 As shown, the method includes:

[0073] S1. Acquiring the image to be processed: Video is captured using a stereo camera, filtered, and normalized to obtain a sequence of images grouped from all video frames. Each group includes a normal image and a depth image corresponding to the video frame. Specifically, S1 includes:

[0074] This embodiment uses a stereo camera to capture complex sports scenes, specifically a basketball game. The acquired video frames are filtered and depth-normalized to obtain a series of images, each consisting of a normal image and a depth image. Filtering addresses depth anomalies caused by motion blur and missing textures by using a weighted median filter combined with a bilateral filter, preserving edge details while suppressing noise. Depth normalization linearly maps pixel values ​​in the depth map to the same numerical range. The image sequence is arranged chronologically.

[0075] S2. Perform object detection: Input the ordinary image of the video frame obtained in the previous step into the YOLOX object detection model, and output the detection bounding boxes of all objects in the image. The bounding boxes include the position coordinates and confidence scores. Based on the confidence scores, the object bounding boxes are divided into high-scoring bounding boxes and low-scoring bounding boxes. Specifically, S2 includes:

[0076] S201. Input the ordinary images in the image sequence obtained in S1 into the YOLOX object detection model to obtain the bounding boxes of all objects. The bounding box information includes the position coordinates and confidence score, which can be represented as (x,y,w,h,s). x and y represent the coordinates of the center point of the bounding box, the origin of the coordinates is the top left corner of the image, w and h represent the width and height of the bounding box, and s represents the detection confidence score of the bounding box.

[0077] S202. Based on the confidence scores, the target bounding boxes are divided into high-scoring bounding boxes and low-scoring bounding boxes. This requires comparing the confidence scores s of the bounding boxes with a preset confidence threshold τ. det The bounding boxes are compared; those with scores greater than a preset threshold are considered high-scoring bounding boxes, and those with scores less than a preset threshold are considered low-scoring bounding boxes. In this embodiment, τ... det Set it to 0.6.

[0078] In this embodiment, the YOLOX model uses pre-trained weights trained on the COCO dataset, and then selects the SportsMOT sports scene dataset for training.

[0079] S3. Obtaining Depth Fusion Information: Based on the target bounding box output in the previous step, obtain the target pseudo-depth information value d using the pseudo-depth estimation method. p Then, the target stereo depth information value d is obtained by combining the depth image. s Through a deep information fusion strategy, the fused depth information value d is obtained, and it is combined with the bounding box information output by the target detection to obtain the complete detection bounding box of each target; wherein, S3 specifically includes:

[0080] S301. Calculate the target pseudo-depth information value d based on the target bounding box information (y, h) obtained in S2 and the height of the ordinary image. p The calculation method is as follows:

[0081] d p =Hy b

[0082] Where H represents the height of the current image, and in this embodiment, the image height H is 1080, y b The vertical coordinate representing the bottom of the current target bounding box is obtained from the bounding box position coordinates mentioned above, and its calculation method is as follows:

[0083]

[0084] S302. Based on the target bounding box information (x, y, w, h) obtained in S2, crop the depth image to obtain a local depth map of the target region, d s This is the depth pixel value at the bottom center of the target;

[0085] S303. Deep information fusion is performed using a deep fusion strategy to obtain the fused target depth information value d. The fusion method is as follows:

[0086]

[0087] Where, σ i This represents the standard deviation of depth pixel values ​​within the bottom region of the target bounding box in the depth map. In this embodiment, the bottom region is the area occupying 1 / 4 of the bounding box height; τ noise This represents the noise threshold, used to determine the depth confidence level of the region. i A value greater than this threshold indicates low depth confidence in the region, d s The value may be affected by noise and needs to be combined with the pseudo-depth value d. p In this embodiment, τ noise Set at 50; r valid τ represents the percentage of valid pixel values ​​in the entire region corresponding to the target bounding box in the depth map. valid This represents the effective threshold, used to determine whether the depth of the target region is missing. When r valid If the value is less than this threshold, it indicates that the target region is missing at a certain depth, d s The value is affected by the background of the target area and should be combined with the pseudo-depth value d. p In this embodiment, τ valid Set to 0.7; α∈[0,1], representing dynamic weights, determined by σ. i and r valid Dynamic adjustment, the calculation method is as follows:

[0088]

[0089] S304. Combine the obtained deep fusion information d with the previous target bounding box to obtain the complete detection bounding box (x,y,w,h,d). detectThe depth information d obtained through the deep fusion strategy can solve the problems of insufficient robustness of pseudo depth information under severe occlusion and the influence of target distance and background noise on binocular depth information.

[0090] S4. Perform motion state modeling and prediction: Based on the detection bounding boxes obtained in the previous step, input the historical frame bounding box information of each tracking trajectory into the motion predictor based on the diffusion model to perform motion modeling and diffusion prediction, and obtain the bounding box (x,y,w,h,d) of the current frame trajectory. track Specifically, S4 includes:

[0091] Based on the detection bounding boxes containing deep fusion information obtained from S3, the bounding box information C of the historical frames of each tracking trajectory is... f The input is fed into a motion predictor based on a diffusion model;

[0092] The trajectory of a target can be represented as T = {B1,…,B}. f ,…,B N The initial trajectory is the bounding box information of the target in frame 0, f is the frame index, N is the total number of frames, and B... f =(x f ,y f ,w f ,h f ,d f This represents the complete bounding box information of the target, showing the center point coordinates, the height and width of the bounding box, and the depth fusion information, which will show the target's motion M. f Defined as the difference in bounding box information between two consecutive frames, it is expressed as:

[0093] M f =B f -B f-1 =(Δx) f ,Δy f ,Δw f ,Δh f ,Δd f )

[0094] Complete target frame information I f Represented as a 10-dimensional vector:

[0095] I f = (B f M f )=(x f ,y f ,w f ,h f ,d f ,Δx f ,Δy f ,Δw f ,Δhf ,Δd f )

[0096] The target's historical frame information C is input into the diffusion-based motion predictor. f Represented as a vector matrix:

[0097]

[0098] Where n represents the number of historical frames of the target trajectory, and the above historical frame information C f The data is fed into the motion predictor for diffusion prediction to obtain the bounding box (x, y, w, h, d) of the current frame trajectory. track .

[0099] S5. Perform hierarchical cascade matching: Based on the target detection bounding boxes and motion prediction trajectory bounding boxes obtained in the previous steps, and combined with the cascade matching strategy, update the target trajectory, such as... Figure 2 As shown, Figure 2 This is a schematic diagram of the deep cascading matching strategy provided in this embodiment. Specifically, S5 includes:

[0100] S501. Based on the depth fusion information values ​​in the detected bounding boxes, all targets in each frame of the image are hierarchically divided to distinguish different targets in occluded and crowded scenes, including:

[0101] Based on the detection bounding box (x, y, w, h, d) obtained by the detection module detect The depth fusion information d in the set obtains the minimum depth value MIN. d and the maximum depth value MAX d According to MIN d and MAX d The interval between them is evenly divided into k different depth intervals, which serve as k different depth levels;

[0102] Similarly, the trajectory bounding box (x,y,w,h,d) obtained from the motion predictor... track The depth fusion information in the set obtains the minimum depth value MIN. t and the maximum depth value MAX t According to MIN t and MAX t The interval between them is evenly divided into k different depth intervals, which serve as k different depth levels;

[0103] Based on the k different depth levels, the detection bounding box sets D at different depth levels are obtained. sub and trajectory bounding box set T subThe value of k will be set according to different situations. In this embodiment, for the basketball game scenario, when the high-scoring bounding box matching indicates that the target occlusion and crowding are good, the value of k is set to 1. When the low-scoring bounding box matching indicates that the target occlusion and crowding are serious, the value of k is set to 8.

[0104] S502, Based on the obtained detection sets D at different depth levels sub and trajectory set T sub Based on IoU distance and the Hungarian algorithm, association matching is performed at the same depth level. Detected bounding boxes and trajectory bounding boxes that do not match at this depth level will proceed to the association process at the next depth level, including:

[0105] D sub High-resolution bounding boxes and T based on detection confidence partitioning sub During association matching, the IoU distance between bounding boxes is calculated to obtain a similarity matrix. Then, the Hungarian algorithm is used to complete the matching based on the similarity matrix. Unmatched high-scoring detection bounding boxes and trajectory bounding boxes will be retained.

[0106] D sub The low-scoring bounding boxes based on detection confidence are associated with the unmatched trajectory bounding boxes from the previous step. Similarly, during the association process, the IoU distance between the bounding boxes is calculated to obtain the similarity matrix. Then, the Hungarian algorithm is used to complete the matching based on the similarity matrix. The unmatched low-scoring bounding boxes and trajectory bounding boxes will be retained.

[0107] The Intersection over Union (IoU) distance is a metric used to assess the degree of overlap between two bounding boxes. It is a simple and effective measure that intuitively reflects the similarity and differences between bounding boxes. It measures similarity by calculating the ratio of the overlapping region to the union region of the two bounding boxes. Figure 3 As shown, Figure 3 The IoU distance diagram provided in this embodiment is calculated as follows:

[0108]

[0109]

[0110] s union =w d h d +w t h t -s overlap

[0111] Among them, s overlap s represents the area of ​​the overlapping region of the two bounding boxes. union Represents the area of ​​the union of two bounding boxes; (x d ,yd ,w d ,h d (x) represents the center point coordinates, width, and height of the detection bounding box. t ,y t ,w t ,h t () represents the center point coordinates, width, and height of the trajectory bounding box;

[0112] High-resolution bounding boxes, low-resolution bounding boxes, and trajectory bounding boxes that are not matched at the current depth level participate in the association matching at the next depth level.

[0113] S503. After the association matching of all targets in the current frame is completed, all unmatched high-resolution bounding boxes will be associated and matched with the trajectory bounding boxes corresponding to the newly appearing trajectory in the previous frame. After this step, the remaining unmatched high-resolution bounding boxes will be initialized as new target tracking trajectories.

[0114] S504. All unmatched low-resolution bounding boxes will be associated with unmatched trajectory bounding boxes from the previous frame. After this step, the remaining low-resolution bounding boxes will be treated as background and deleted. Unmatched trajectory bounding boxes will still be retained and participate in the association matching of the next frame. This process continues until the retention threshold τ is exceeded. lost If the matching is still not completed, the corresponding trajectory will be deleted. In this embodiment, τ lost Set to 60;

[0115] S505. Finally, add the high and low bounding boxes and the low bounding box matched in the previous steps to the target trajectory corresponding to the trajectory bounding box that matches them, update the target tracking trajectory, and obtain the tracking result of the current frame. When the association matching of all frames is completed, the complete multi-target tracking result is obtained.

[0116] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. A multi-target tracking method based on binocular cameras combined with pseudo-depth estimation, characterized in that: The method includes the following steps: Video images are acquired using a binocular camera, and after preprocessing, a sequence of images is obtained from groups of all video frames. Each group of images includes a normal image and a depth image corresponding to the video frame. Input ordinary images from each image sequence into the object detection model to obtain bounding boxes (x, y, w, h, s) of all objects in the ordinary images; classify the object bounding boxes into high-scoring bounding boxes and low-scoring bounding boxes based on the confidence s of the bounding boxes. The pseudo-depth information value d of the target is obtained based on the target bounding box information and the height of the corresponding ordinary image. p And obtain the target stereo depth information value d based on its depth image. s A deep fusion strategy is used to perform deep information fusion to obtain the fused target depth information value d, and finally obtain the complete target detection bounding box (x,y,w,h,d). detect ; A motion predictor based on a diffusion model is used to predict the bounding box (x, y, w, h, d) of the current frame trajectory based on the target detection bounding boxes of each historical frame for each tracking trajectory. track ; Define the target detection bounding box (x, y, w, h, d). detect With the predicted trajectory bounding box (x,y,w,h,d) track The target trajectory is updated by hierarchical cascaded matching to obtain the tracking result of the current frame. When the association matching of all frames is completed, the complete multi-target tracking result is obtained.

2. The multi-target tracking method based on binocular camera combined with pseudo-depth estimation according to claim 1, characterized in that: In the preprocessing of video images acquired by the stereo camera, the acquired video frames are filtered and depth normalized to obtain a set of image sequences. The image sequences are arranged in chronological order. Filtering refers to the process of using weighted median filtering combined with bilateral filtering to address depth anomalies caused by motion blur and local texture loss, thereby preserving edge details while suppressing noise. Depth normalization is a process that linearly maps pixel values ​​in a depth map to the same numerical range.

3. The multi-target tracking method based on binocular camera combined with pseudo-depth estimation according to claim 1, characterized in that: The object detection model obtains the detection bounding boxes (x, y, w, h, s) of all objects in a normal image. The object detection model is pre-trained on a targeted scene dataset. Here, x and y represent the coordinates of the center point of the bounding box, with the origin being the top left corner of the image. w and h represent the width and height of the bounding box, and s represents the detection confidence value of the bounding box. Then, the confidence value s of the bounding box is compared with the preset confidence threshold τ. det The bounding boxes are compared, and those with scores greater than a preset threshold are considered high-scoring bounding boxes, while those with scores less than the preset threshold are considered low-scoring bounding boxes.

4. The multi-target tracking method based on binocular camera combined with pseudo-depth estimation according to claim 1, characterized in that: Obtain the complete object detection bounding box (x, y, w, h, d). detect The process is as follows: The target pseudo-depth information value d is calculated based on the obtained target bounding box information (y, h) and the height of the ordinary image. p The calculation method is as follows: d p =h-y b Where h represents the height of the current image, and y b The vertical coordinate representing the bottom of the current target bounding box is obtained from the bounding box position coordinates mentioned above, and its calculation method is as follows: Then, based on the acquired target bounding box information (x, y, w, h), the depth image is cropped to obtain a local depth map of the target region. The depth pixel value at the bottom center of the target is the depth information value d. s ; Deep information fusion is performed using a deep fusion strategy to obtain the fused target depth information value d: Where, σ i τ represents the standard deviation of depth pixel values ​​within the bottom region of the target bounding box in the depth map. noise This represents the noise threshold, used to determine the depth confidence level of the region. i Greater than τ noise This indicates that the depth confidence in this area is low, and it needs to be combined with a pseudo-depth value d. p ;r valid τ represents the percentage of valid pixel values ​​in the entire region corresponding to the target bounding box in the depth map. valid This represents the effective threshold, used to determine whether the depth of the target region is missing. When r valid Less than τ valid This indicates that the target region lacks depth and needs to be combined with a pseudo-depth value d. p α∈[0,1], representing dynamic weights, determined by σ i and r valid The dynamic adjustment is calculated as follows: The obtained deep fusion information d is combined with the previous target bounding box to obtain the complete detection bounding box (x, y, w, h, d). detect .

5. The multi-target tracking method based on binocular camera combined with pseudo-depth estimation according to claim 1, characterized in that: A motion predictor based on a diffusion model predicts the bounding box (x, y, w, h, d) of the predicted trajectory in the current frame. track The process is as follows: C contains the bounding box information of the historical frames of each tracking trajectory. f The input is fed into a motion predictor based on a diffusion model; where the trajectory of a target is represented as T = {B1,…,B}. f ,…,B N The initial trajectory is the bounding box information of the target in frame 0, f is the frame index, N is the total number of frames, and B... f =(x f ,y f ,w f ,h f ,d f This represents the complete bounding box information of the target, showing the center point coordinates, the height and width of the bounding box, and the depth fusion information, which will show the target's motion M. f Defined as the difference in bounding box information between two consecutive frames, it is expressed as: M f =B f -B f-1 =(Δx f ,Δy f ,Δw f ,Δh f ,Δd f ) Complete target frame information I f Represented as a 10-dimensional vector: I f =(B f ,M f )=(x f ,y f ,w f ,h f ,d f ,Δx f ,Δy f ,Δw f ,Δh f, Δd f ) The target's historical frame information C is input into the diffusion-based motion predictor. f Represented as a vector matrix: Where n represents the number of historical frames of the target trajectory, and the above historical frame information C f The data is fed into the motion predictor for diffusion prediction to obtain the bounding box (x, y, w, h, d) of the current frame trajectory. track .

6. The multi-target tracking method based on binocular camera combined with pseudo-depth estimation according to claim 1, characterized in that: The hierarchical cascade matching process includes: Based on the deep fusion information values ​​in the detected bounding boxes, all targets in each frame of the image are hierarchically divided to distinguish different targets in occluded and crowded scenes; Based on the k different depth levels, the detection bounding box sets D at different depth levels are obtained. sub and trajectory bounding box set T sub The value of k will be set separately depending on the different situations; Based on the obtained detection sets D at different depth levels sub and trajectory set T sub Based on IoU distance and Hungarian algorithm, association matching is performed at the same depth level. If the target detection bounding box and the predicted trajectory bounding box fail to match at the current depth level, they will enter the association process at the next depth level. After all targets in the current frame have been matched, all unmatched high-resolution bounding boxes will be matched with the bounding boxes of the newly appearing trajectories in the previous frame; high-resolution bounding boxes that are still unmatched will be initialized as new target tracking trajectories. All unmatched low-resolution bounding boxes will be associated with unmatched trajectory bounding boxes from the previous frame for matching; low-resolution bounding boxes that still do not match will be treated as background deletion. Unmatched trajectory bounding boxes are retained and participate in the association matching of the next frame, exceeding the retention threshold τ. lost If the matching is still not completed, the corresponding trajectory will be deleted. Finally, the high and low bounding boxes and the low bounding box matched in the previous steps are added to the target trajectory corresponding to the trajectory bounding box that matches them, the target tracking trajectory is updated, and the tracking result of the current frame is obtained. When the association matching of all frames is completed, the complete multi-target tracking result is obtained.

7. A multi-target tracking method based on binocular camera combined with pseudo-depth estimation according to claim 6, characterized in that: The specific method for hierarchical division is as follows: Detect the bounding box (x, y, w, h, d) based on the target. detect The depth fusion information d in the set obtains the minimum depth value MIN. d and the maximum depth value MAX d According to MIN d and MAX d The interval between them is evenly divided into k different depth intervals, which serve as k different depth levels; Meanwhile, based on the predicted trajectory bounding box (x,y,w,h,d) track The depth fusion information in the set obtains the minimum depth value MIN. t and the maximum depth value MAX t According to MIN t and MAX t The distance between the intervals is evenly divided into k different depth intervals, which serve as k different depth levels.

8. A multi-target tracking method based on binocular camera combined with pseudo-depth estimation according to claim 6, characterized in that: The process of association matching based on IoU distance and the Hungarian algorithm at the same depth level is as follows: D sub High-resolution bounding boxes and T based on detection confidence partitioning sub During association matching, the IoU distance between bounding boxes is calculated to obtain a similarity matrix. Then, the Hungarian algorithm is used to complete the matching based on the similarity matrix. Unmatched high-scoring detection bounding boxes and trajectory bounding boxes will be retained. D sub The low-scoring bounding boxes based on detection confidence are associated with the unmatched trajectory bounding boxes from the previous step. Similarly, during the association process, the IoU distance between the bounding boxes is calculated to obtain the similarity matrix. Then, the Hungarian algorithm is used to complete the matching based on the similarity matrix. The unmatched low-scoring bounding boxes and trajectory bounding boxes will be retained. The IoU distance measures the similarity between two bounding boxes by calculating the ratio of their overlapping region to their union region. The calculation method is as follows: s union =w d h d +w k h t -s overlap Among them, s overlap s represents the area of ​​the overlapping region of the two bounding boxes. union Represents the area of ​​the union of two bounding boxes; (x d ,y d ,w d ,h d (x) represents the center point coordinates, width, and height of the detection bounding box. t ,y t ,w t ,h t () represents the center point coordinates, width, and height of the trajectory bounding box; High-resolution bounding boxes, low-resolution bounding boxes, and trajectory bounding boxes that are not matched at the current depth level participate in the association matching at the next depth level.

9. A computer-readable storage medium, characterized in that, The storage medium stores a computer program, which, when executed by a processor, implements the multi-target tracking method based on binocular camera combined with pseudo-depth estimation as described in any one of claims 1-8.

10. A computer program product, characterized in that: Includes a computer program that, when executed by a processor, implements the multi-target tracking method based on binocular camera combined with pseudo-depth estimation as described in any one of claims 1-8.

Images