Multi-target tracking method, device, readable storage medium and program product

Abstract

Embodiments of the present application provide a multi-target tracking method and device, a readable storage medium and a program product, and relate to the field of video tracking. The method comprises: obtaining an image corresponding to a target frame, the image comprising a plurality of target objects; determining a depth map corresponding to the image, all prediction boxes corresponding to the image, and all detection boxes corresponding to the image; determining an intersection-over-union cost matrix based on the all prediction boxes and the all detection boxes; determining a first depth cost matrix corresponding to the all prediction boxes based on the depth map and the all prediction boxes; determining a second depth cost matrix corresponding to the all detection boxes based on the depth map and the all detection boxes; performing fusion processing based on the intersection-over-union cost matrix, the first depth cost matrix and the second depth cost matrix to determine a fusion cost matrix; and determining a tracking box corresponding to each target object in the plurality of target objects based on the fusion cost matrix. The depth information of the depth map is introduced into the fusion cost matrix, and the accuracy of multi-target tracking is improved.

Description

Technical Field

[0001] This application relates to the field of computer technology, and more specifically, to a multi-target tracking method, apparatus, readable storage medium, and program product. Background Technology

[0002] In existing technologies, multi-object tracking assigns an ID (identity) to multiple target objects in each frame of a video and obtains the tracking trajectory of the behavior of each ID, i.e., the tracking trajectory of the behavior of each target object among the multiple target objects. When two or more target objects overlap, due to misjudgment by the tracking algorithm, the ID of a target object often changes, i.e., an ID switch occurs, resulting in low accuracy of multi-object tracking. Summary of the Invention

[0003] This application addresses the shortcomings of existing methods by proposing a multi-target tracking method, apparatus, device, computer-readable storage medium, and computer program product to solve the problem of how to improve the accuracy of multi-target tracking.

[0004] Firstly, this application provides a multi-target tracking method, including:

[0005] Acquire the image corresponding to the target frame, which includes multiple target objects;

[0006] Determine the depth map, all predicted bounding boxes, and all detected bounding boxes of the image;

[0007] Based on all predicted bounding boxes and all detected bounding boxes, determine the intersection-union cost matrix; based on the depth map and all predicted bounding boxes, determine the first depth cost matrix corresponding to all predicted bounding boxes; based on the depth map and all detected bounding boxes, determine the second depth cost matrix corresponding to all detected bounding boxes.

[0008] Based on the intersection-union cost matrix, the first depth cost matrix, and the second depth cost matrix, a fusion process is performed to determine the fusion cost matrix;

[0009] Based on the fusion cost matrix, the tracking bounding box corresponding to each target object among multiple target objects is determined.

[0010] In one embodiment, determining the depth map corresponding to the image, all predicted bounding boxes corresponding to the image, and all detected bounding boxes corresponding to the image includes:

[0011] The image is input into a depth model to extract depth information, resulting in a depth map. This depth map includes the depth information of each of the multiple target objects; and / or

[0012] The tracking bounding box and image corresponding to the previous frame of the target frame are processed by Kalman filtering to obtain all predicted bounding boxes corresponding to the image; and / or

[0013] The image is input into the detection model for detection processing, and all detection boxes corresponding to the image are obtained.

[0014] In one embodiment, determining the intersection-union cost matrix based on all predicted bounding boxes and all detected bounding boxes includes:

[0015] The cross-union (CUI) cost matrix is ​​obtained by performing cross-union comparison (CUC) on all predicted bounding boxes and all detected bounding boxes.

[0016] In one embodiment, determining the first depth cost matrix corresponding to all predicted bounding boxes based on the depth map and all predicted bounding boxes includes:

[0017] Determine the first ratio between each prediction box and the depth map in all prediction boxes, and the average depth value corresponding to each prediction box;

[0018] Based on each first ratio and each depth average, the first depth cost matrix corresponding to all prediction boxes is determined.

[0019] In one embodiment, determining a first ratio between each prediction box and the depth map across all prediction boxes, and the average depth value corresponding to each prediction box, includes:

[0020] Based on the size of each prediction box in all prediction boxes, the size of the depth map, and a preset first weight, determine the first ratio between each prediction box and the depth map;

[0021] Determine the depth value of each pixel in the depth map corresponding to each prediction box, and the depth value of each pixel is greater than a preset depth threshold;

[0022] Based on the depth value of each pixel and the total number of pixels, the average depth value corresponding to each prediction box is determined by averaging.

[0023] In one embodiment, the first depth cost matrix corresponding to all predicted boxes is determined based on each first ratio and each depth average, including:

[0024] The cost value for each prediction box is obtained by summing the first ratio corresponding to each prediction box and the average depth corresponding to each prediction box.

[0025] Based on the cost value corresponding to each prediction box, the first depth cost matrix corresponding to all prediction boxes is obtained.

[0026] In one embodiment, determining the second depth cost matrix corresponding to all detection boxes based on the depth map and all detection boxes includes:

[0027] Determine the second ratio between each detection box and the depth map in all detection boxes, and the average depth value corresponding to each detection box;

[0028] Based on each second ratio and each depth average, the second depth cost matrix corresponding to all detection boxes is determined.

[0029] In one embodiment, determining a second ratio between each detection box and the depth map across all detection boxes, and the average depth value corresponding to each detection box, includes:

[0030] Based on the size of each detection box in all detection boxes, the size of the depth map, and a preset second weight, a second ratio between each detection box and the depth map is determined.

[0031] Determine the depth value of each pixel in the depth map corresponding to each detection box, and the depth value of each pixel is greater than a preset depth threshold;

[0032] Based on the depth value of each pixel and the total number of pixels, an average value is calculated to determine the average depth value for each detection box.

[0033] In one embodiment, the second depth cost matrix corresponding to all detection boxes is determined based on each second ratio and each depth average, including:

[0034] The cost value for each detection box is obtained by summing the second ratio corresponding to each detection box and the average depth corresponding to each detection box.

[0035] Based on the cost value corresponding to each detection box, the second depth cost matrix corresponding to all detection boxes is obtained.

[0036] In one embodiment, a fusion process is performed based on the intersection-union cost matrix, the first depth cost matrix, and the second depth cost matrix to determine the fusion cost matrix, including:

[0037] The difference between the first depth cost matrix and the second depth cost matrix is ​​used to obtain the third depth cost matrix;

[0038] The third depth cost matrix and the intersection-over-union (IoU) cost matrix are normalized to obtain the normalized third depth cost matrix and the normalized IoU cost matrix.

[0039] The normalized third-depth cost matrix and the normalized intersection-union cost matrix are fused together to determine the fused cost matrix.

[0040] In one embodiment, the normalized third-depth cost matrix and the normalized intersection-union cost matrix are fused to determine the fused cost matrix, including:

[0041] The fusion cost matrix is ​​determined based on the normalized third-depth cost matrix, the third weight corresponding to the normalized third-depth cost matrix, the normalized intersection-union ratio cost matrix, and the fourth weight corresponding to the normalized intersection-union ratio cost matrix.

[0042] In one embodiment, determining the tracking bounding box corresponding to each of the multiple target objects based on the fusion cost matrix includes:

[0043] The fusion cost matrix is ​​processed using the Hungarian algorithm to determine the prediction box and detection box for each target object among multiple target objects.

[0044] Based on the predicted bounding box and the detection bounding box corresponding to each target object, the tracking bounding box corresponding to each target object is determined.

[0045] Secondly, this application provides a multi-target tracking device, comprising:

[0046] The first processing module is used to acquire the image corresponding to the target frame, and the image includes multiple target objects;

[0047] The second processing module is used to determine the depth map corresponding to the image, all prediction boxes corresponding to the image, and all detection boxes corresponding to the image.

[0048] The third processing module is used to determine the intersection-over-union cost matrix based on all predicted boxes and all detected boxes; to determine the first depth cost matrix corresponding to all predicted boxes based on the depth map and all predicted boxes; and to determine the second depth cost matrix corresponding to all detected boxes based on the depth map and all detected boxes.

[0049] The fourth processing module is used to perform fusion processing based on the intersection-union cost matrix, the first depth cost matrix, and the second depth cost matrix to determine the fusion cost matrix;

[0050] The fifth processing module is used to determine the tracking box corresponding to each target object among multiple target objects based on the fusion cost matrix.

[0051] Thirdly, this application provides an electronic device, including: a processor, a memory, and a bus;

[0052] A bus is used to connect the processor and memory;

[0053] Memory, used to store operation instructions;

[0054] A processor is used to execute the multi-target tracking method of the first aspect of this application by invoking operation instructions.

[0055] Fourthly, this application provides a computer-readable storage medium storing a computer program that is used to execute the multi-target tracking method of the first aspect of this application.

[0056] Fifthly, this application provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the multi-target tracking method in the first aspect of this application.

[0057] The technical solution provided in this application has at least the following beneficial effects:

[0058] The process involves acquiring the image corresponding to the target frame, which includes multiple target objects; determining the depth map, all predicted bounding boxes, and all detected bounding boxes corresponding to the image; determining the Cross-Union Comparison (CUC) cost matrix based on the predicted and detected bounding boxes; determining the first depth cost matrix corresponding to all predicted bounding boxes based on the depth map and all predicted bounding boxes; determining the second depth cost matrix corresponding to all detected bounding boxes based on the depth map and all detected bounding boxes; fusing the CUC cost matrix, the first depth cost matrix, and the second depth cost matrix to determine the fused cost matrix; and determining the tracking box corresponding to each of the multiple target objects based on the fused cost matrix. By incorporating depth information from the depth map into the fused cost matrix, the problem of ID switching after target (object) overlap is avoided, thus improving the accuracy of multi-target (multiple target objects) tracking. Attached Figure Description

[0059] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments of this application will be briefly introduced below.

[0060] Figure 1 This is a schematic diagram of the architecture of a multi-target tracking system provided in an embodiment of this application;

[0061] Figure 2 A flowchart illustrating a multi-target tracking method provided in an embodiment of this application;

[0062] Figure 3 A flowchart illustrating a multi-target tracking method provided in an embodiment of this application;

[0063] Figure 4 This is a schematic diagram of the structure of a multi-target tracking device provided in an embodiment of this application;

[0064] Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0065] The embodiments of this application are described below with reference to the accompanying drawings. It should be understood that the embodiments described below with reference to the accompanying drawings are exemplary descriptions for explaining the technical solutions of the embodiments of this application, and do not constitute a limitation on the technical solutions of the embodiments of this application.

[0066] Those skilled in the art will understand that, unless otherwise stated, the singular forms “a,” “an,” “the,” and “the” used herein may also include the plural forms. It should be further understood that the terms “comprising” and “including” as used in embodiments of this application mean that the corresponding feature can be implemented as the presented feature, information, data, step, operation, element, and / or component, but do not exclude implementation as other features, information, data, step, operation, element, component, and / or combinations thereof supported by the art. It should be understood that when we say that an element is “connected” or “coupled” to another element, the one element can be directly connected or coupled to the other element, or it can mean that the one element and the other element establish a connection relationship through an intermediate element. Furthermore, “connected” or “coupled” as used herein can include wireless connection or wireless coupling. The term “and / or” as used herein indicates at least one of the items defined by the term; for example, “A and / or B” indicates implementation as “A,” or implementation as “B,” or implementation as “A and B.”

[0067] It is understood that in the specific implementation of this application, data related to multi-target tracking is involved. When the above embodiments of this application are applied to specific products or technologies, user permission or consent is required, and the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0068] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application will be described in further detail below with reference to the accompanying drawings.

[0069] This application provides a multi-target tracking method for a multi-target tracking system, which relates to fields such as video tracking.

[0070] To better understand and explain the solutions of the embodiments of this application, some technical terms involved in the embodiments of this application will be briefly explained below.

[0071] IoU: In detection tasks, IoU (Intersection of Union) is used as a metric to describe the degree of overlap between two bounding boxes.

[0072] Kalman filtering: Kalman filtering is an algorithm that uses the state equations of a linear system to make an optimal estimate of the system state using the system's input and output observation data.

[0073] Hungarian Algorithm: The Hungarian algorithm is a combinatorial optimization algorithm that solves the task assignment problem in polynomial time.

[0074] Depth map: In 3D computer graphics and computer vision, a depth map is an image or image channel that includes information about the distance from the surface of a scene object to the viewpoint.

[0075] The solutions provided in this application relate to multi-target tracking technology. The technical solutions of this application will be described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of this application will be described below with reference to the accompanying drawings.

[0076] To better understand the solution provided in the embodiments of this application, the solution will be described below in conjunction with a specific application scenario.

[0077] In one embodiment, Figure 1 The diagram shows an architecture schematic of a multi-target tracking system applicable to embodiments of this application. It is understood that the multi-target tracking method provided in these embodiments can be applied to, but is not limited to, applications such as... Figure 1 In the application scenarios shown.

[0078] In this example, as Figure 1 As shown, the architecture of the multi-target tracking system in this example may include, but is not limited to, terminal 10 and database 20. Terminal 10 and database 20 can interact via network 30. Terminal 10 acquires the image corresponding to the target frame, which includes multiple target objects; terminal 10 determines the depth map, all predicted bounding boxes, and all detected bounding boxes corresponding to the image; terminal 10 determines the intersection-union cost matrix based on all predicted bounding boxes and all detected bounding boxes; terminal 10 determines the first depth cost matrix corresponding to all predicted bounding boxes based on the depth map and all predicted bounding boxes; terminal 10 determines the second depth cost matrix corresponding to all detected bounding boxes based on the depth map and all detected bounding boxes; terminal 10 performs fusion processing based on the intersection-union cost matrix, the first depth cost matrix, and the second depth cost matrix to determine the fused cost matrix; terminal 10 determines the tracking box corresponding to each of the multiple target objects based on the fused cost matrix; terminal 10 displays the tracking box corresponding to each target object; terminal 10 sends the tracking box corresponding to each target object to database 20 for storage.

[0079] It is understood that the above is only one example, and this embodiment is not limited here.

[0080] Terminals include, but are not limited to, smartphones (such as Android phones, iOS phones, etc.), mobile phone emulators, tablets, laptops, digital broadcast receivers, MIDs (Mobile Internet Devices), PDAs (Personal Digital Assistants), smart voice interaction devices, smart home appliances, and in-vehicle terminals.

[0081] The aforementioned networks may include, but are not limited to, wired networks and wireless networks. Wired networks include local area networks (LANs), metropolitan area networks (MANs), and wide area networks (WANs). Wireless networks include Bluetooth, Wi-Fi, and other networks that enable wireless communication. Specific details can be determined based on actual application scenario requirements and are not limited here.

[0082] See Figure 2 , Figure 2 This illustration shows a flowchart of a multi-target tracking method provided in an embodiment of this application. This method can be executed by any electronic device, such as a terminal. As an optional implementation, the method can be executed by a terminal. For ease of description, the following descriptions of some optional embodiments will use a terminal as the execution subject of the method. Figure 2 As shown, the multi-target tracking method provided in this application includes the following steps:

[0083] S201, Obtain the image corresponding to the target frame. The image includes multiple target objects.

[0084] Specifically, a target frame is a frame in a video, which can be live or pre-recorded. The image corresponding to the target frame is an image containing multiple target objects, such as people or animals.

[0085] S202, determine the depth map corresponding to the image, all predicted bounding boxes corresponding to the image, and all detection bounding boxes corresponding to the image.

[0086] Specifically, an image corresponds to a depth map, which includes the depth information of each of the multiple target objects, and the depth information of each target object is different; the number of all predicted boxes, the number of all detected boxes, and the number of multiple target objects are the same for the image; for example, if there are 5 target objects in an image, then the number of all predicted boxes and the number of all detected boxes for the image are 5.

[0087] S203, based on all predicted bounding boxes and all detected bounding boxes, determine the intersection-union cost matrix; based on the depth map and all predicted bounding boxes, determine the first depth cost matrix corresponding to all predicted bounding boxes; based on the depth map and all detected bounding boxes, determine the second depth cost matrix corresponding to all detected bounding boxes.

[0088] Specifically, the first depth cost matrix and the second depth cost matrix introduce the depth information of the depth map, that is, the depth information of each target object among multiple target objects.

[0089] Specifically, for example, if there are 5 predicted boxes and 5 detected boxes, determine the cross-union ratio (CUP) between each predicted box and each detected box to obtain the CUP cost matrix, which is a 5×5 matrix.

[0090] The depth map corresponding to the image has the same size as the image; if a prediction box is set in the depth map, then the depth information of the size of the prediction box will be available within the range of the prediction box; if a detection box is set in the depth map, then the depth information of the size of the detection box will be available within the range of the detection box.

[0091] S204. Based on the intersection-union cost matrix, the first depth cost matrix, and the second depth cost matrix, perform fusion processing to determine the fusion cost matrix.

[0092] Specifically, based on the intersection-union cost matrix, the first depth cost matrix, and the second depth cost matrix, a fusion process is performed to introduce the depth information of the depth map into the fusion cost matrix, that is, the depth information of each target object among multiple target objects.

[0093] S205, based on the fusion cost matrix, determines the tracking box corresponding to each target object among multiple target objects.

[0094] Specifically, by introducing depth information from the depth map into the fusion cost matrix, the problem of ID switching after two or more target objects overlap is avoided, which improves the accuracy of determining the tracking box corresponding to each target object among multiple target objects, thus improving the accuracy of tracking multiple target objects.

[0095] In this embodiment, an image corresponding to a target frame is acquired, the image including multiple target objects; a depth map, all predicted bounding boxes, and all detected bounding boxes corresponding to the image are determined; an intersection-union-ratio (IUU) cost matrix is ​​determined based on all predicted bounding boxes and all detected bounding boxes; a first depth cost matrix corresponding to all predicted bounding boxes is determined based on the depth map and all predicted bounding boxes; a second depth cost matrix corresponding to all detected bounding boxes is determined based on the depth map and all detected bounding boxes; a fusion process is performed based on the IUU cost matrix, the first depth cost matrix, and the second depth cost matrix to determine a fused cost matrix; and a tracking box corresponding to each of the multiple target objects is determined based on the fused cost matrix. Thus, by introducing depth information from the depth map into the fused cost matrix, the problem of ID switching after target (object) overlap is avoided, improving the accuracy of multi-target (multiple target objects) tracking.

[0096] In one embodiment, determining the depth map corresponding to the image, all predicted bounding boxes corresponding to the image, and all detected bounding boxes corresponding to the image includes:

[0097] The image is input into a depth model to extract depth information, resulting in a depth map. This depth map includes the depth information of each of the multiple target objects; and / or

[0098] The tracking box and image corresponding to the previous frame of the target frame are processed by Kalman filtering to obtain all predicted boxes corresponding to the image; and / or the image is input into the detection model for detection processing to obtain all detection boxes corresponding to the image.

[0099] Specifically, the depth model is called the depth model; an image is input into the depth model to extract depth information and obtain a depth map corresponding to the image. The depth map includes the depth information of all target objects in the image.

[0100] For example, if there are 5 target objects in a video, these 5 target objects are tracked; the image A corresponding to the previous frame of the target frame contains 5 tracking boxes corresponding to these 5 target objects. These 5 tracking boxes and the image B corresponding to the target frame are processed by Kalman filtering to obtain 5 predicted boxes corresponding to image B; image B is input into the detection model for detection processing to obtain 5 detection boxes corresponding to image B.

[0101] In one embodiment, determining the intersection-union cost matrix based on all predicted bounding boxes and all detected bounding boxes includes:

[0102] The cross-union (CUI) cost matrix is ​​obtained by performing cross-union comparison (CUC) on all predicted bounding boxes and all detected bounding boxes.

[0103] Specifically, for example, all predicted boxes are 3 predicted boxes, namely predicted box A, predicted box B, and predicted box C, and all detected boxes are 3 detected boxes, namely detected box D, detected box E, and detected box F; determine the cross-union ratio (CUP) between predicted box A and detected box D, between predicted box A and detected box E, between predicted box A and detected box F, between predicted box B and detected box D, between predicted box B and detected box E, between predicted box B and detected box F, between predicted box C and detected box D, between predicted box C and detected box E, and between predicted box C and detected box F; based on the above 9 CUP ratios, obtain the CUP cost matrix, which is a 3×3 matrix.

[0104] In one embodiment, determining the first depth cost matrix corresponding to all predicted bounding boxes based on the depth map and all predicted bounding boxes includes:

[0105] Determine the first ratio between each prediction box and the depth map in all prediction boxes, and the average depth value corresponding to each prediction box;

[0106] Based on each first ratio and each depth average, the first depth cost matrix corresponding to all prediction boxes is determined.

[0107] Specifically, the size S of the prediction box is calculated. 预测框 Formula (1) is shown below:

[0108] S 预测框 =W 预测框 ×H 预测框 Formula (1)

[0109] Among them, W 预测框 H represents the width of the prediction box. 预测框 This indicates the height of the prediction box.

[0110] Calculate the size S of the depth map 深度图 Formula (2) is shown below:

[0111] S 深度图 =W 深度图 ×H 深度图 Formula (2)

[0112] Among them, W 深度图 H represents the width of the depth map. 深度图 This indicates the height of the depth map.

[0113] The formula (3) for calculating the first ratio S between the predicted bounding box and the depth map is shown below:

[0114]

[0115] Where b1 is the preset first weight.

[0116] Calculate the average depth corresponding to the predicted bounding box Formula (4) is shown below:

[0117]

[0118] Where d represents the depth value of a pixel, and the depth value of a pixel ranges from [0, 255]. The depth value d of a pixel is greater than the preset depth threshold dt, so that pixels with too small a depth value can be blocked; set D represents the set of all pixels in the prediction box whose depth value d is greater than the preset depth threshold dt; and count represents the total number of all pixels in the prediction box whose depth value d is greater than the preset depth threshold dt.

[0119] Based on each first ratio (S) and the average value of each depth The first depth cost matrix dc1 corresponding to all predicted boxes is determined, and the formula (5) for calculating the first depth cost matrix dc1 is as follows:

[0120]

[0121] in, The values ​​represent the average values ​​at each depth, S1, S2, S3, ... S n This represents the first ratio, where n is a positive integer.

[0122] In one embodiment, determining a first ratio between each prediction box and the depth map across all prediction boxes, and the average depth value corresponding to each prediction box, includes:

[0123] Based on the size of each prediction box in all prediction boxes, the size of the depth map, and a preset first weight, determine the first ratio between each prediction box and the depth map;

[0124] Determine the depth value of each pixel in the depth map corresponding to each prediction box, and the depth value of each pixel is greater than a preset depth threshold;

[0125] Based on the depth value of each pixel and the total number of pixels, the average depth value corresponding to each prediction box is determined by averaging.

[0126] Specifically, based on each prediction box S in all prediction boxes 预测框 Size, depth map size S 深度图 With a preset first weight b1, determine the first ratio S between each prediction box and the depth map.

[0127] Calculate the size S of the prediction box 预测框 Formula (1) is shown below:

[0128] S 预测框 =W 预测框 ×H 预测框 Formula (1)

[0129] Among them, W 预测框 H represents the width of the prediction box. 预测框 This indicates the height of the prediction box.

[0130] Calculate the size S of the depth map 深度图 Formula (2) is shown below:

[0131] S 深度图 =W 深度图 ×H 深度图 Formula (2)

[0132] Among them, W 深度图 H represents the width of the depth map. 深度图This indicates the height of the depth map.

[0133] The formula (3) for calculating the first ratio S between the predicted bounding box and the depth map is shown below:

[0134]

[0135] Where b1 is the preset first weight.

[0136] Determine the depth value d of each pixel in the depth map corresponding to each prediction box, where the depth value d of each pixel is greater than a preset depth threshold dt; based on the depth value d of each pixel and the total number of pixels count, perform averaging to determine the average depth d of each prediction box. 预测框 .

[0137] Calculate the average depth corresponding to the predicted bounding box Formula (4) is shown below:

[0138]

[0139] Where d represents the depth value of a pixel, and the depth value of a pixel ranges from [0, 255]. The depth value d of a pixel is greater than the preset depth threshold dt, so that pixels with too small a depth value can be blocked; set D represents the set of all pixels in the prediction box whose depth value d is greater than the preset depth threshold dt; and count represents the total number of all pixels in the prediction box whose depth value d is greater than the preset depth threshold dt.

[0140] In one embodiment, the first depth cost matrix corresponding to all predicted boxes is determined based on each first ratio and each depth average, including:

[0141] The cost value for each prediction box is obtained by summing the first ratio corresponding to each prediction box and the average depth corresponding to each prediction box.

[0142] Based on the cost value corresponding to each prediction box, the first depth cost matrix corresponding to all prediction boxes is obtained.

[0143] Specifically, the first ratio S corresponding to each prediction box and the average depth corresponding to each prediction box are... Summing is performed to obtain the cost corresponding to each prediction box. Based on the cost value corresponding to each prediction frame The first depth cost matrix dc1 corresponding to all predicted boxes is obtained.

[0144] Based on each first ratio (S) and the average value of each depth The first depth cost matrix dc1 corresponding to all predicted boxes is determined, and the formula (5) for calculating the first depth cost matrix dc1 is as follows:

[0145]

[0146] in, The values ​​represent the average values ​​at each depth, S1, S2, S3, ... S n Indicates the first ratio of each, This represents the cost corresponding to the prediction box n, where n is a positive integer.

[0147] In one embodiment, determining the second depth cost matrix corresponding to all detection boxes based on the depth map and all detection boxes includes:

[0148] Determine the second ratio between each detection box and the depth map in all detection boxes, and the average depth value corresponding to each detection box;

[0149] Based on each second ratio and each depth average, the second depth cost matrix corresponding to all detection boxes is determined.

[0150] Specifically, the size S of the detection frame is calculated. 检测框 Formula (6) is shown below:

[0151] S 检测框 =W 检测框 ×H 检测框 Formula (6)

[0152] Among them, W 检测框 H represents the width of the detection box. 检测框 This indicates the height of the detection frame.

[0153] Calculate the size S of the depth map 深度图 Formula (2) is shown below:

[0154] S 深度图 =W 深度图 ×H 深度图 Formula (2)

[0155] Among them, W 深度图 H represents the width of the depth map. 深度图 This indicates the height of the depth map.

[0156] The formula (7) for calculating the second ratio S between the detection box and the depth map is shown below:

[0157]

[0158] Where b2 is the preset second weight.

[0159] It should be noted that the preset second weight b2 and the preset first weight b1 can be the same.

[0160] Calculate the average depth corresponding to the detection box Formula (8) is shown below:

[0161]

[0162] Where d represents the depth value of a pixel, and the depth value of a pixel is in the range of [0, 255]. The depth value d of the pixel is greater than the preset depth threshold dt, so that pixels with too small a depth value can be blocked; set D represents the set of all pixels in the detection box whose depth value d is greater than the preset depth threshold dt; count represents the total number of all pixels in the detection box whose depth value d is greater than the preset depth threshold dt.

[0163] Based on each second ratio (S) and the average value of each depth The second depth cost matrix dc2 corresponding to all detection boxes is determined, and the formula (9) for calculating the second depth cost matrix dc2 is as follows:

[0164]

[0165] in, The values ​​represent the average values ​​at each depth, S1, S2, S3, ... S n This represents the second ratio, where n is a positive integer.

[0166] In one embodiment, determining a second ratio between each detection box and the depth map across all detection boxes, and the average depth value corresponding to each detection box, includes:

[0167] Based on the size of each detection box in all detection boxes, the size of the depth map, and a preset second weight, a second ratio between each detection box and the depth map is determined.

[0168] Determine the depth value of each pixel in the depth map corresponding to each detection box, and the depth value of each pixel is greater than a preset depth threshold;

[0169] Based on the depth value of each pixel and the total number of pixels, an average value is calculated to determine the average depth value for each detection box.

[0170] Specifically, based on the size S of each detection frame in all detection frames. 检测框 The size S of the depth map 深度图 And a second weight b2 is preset to determine a second ratio S between each detection box and the depth map.

[0171] Calculate the size S of the detection frame 检测框 Formula (6) is shown below:

[0172] S 检测框 =W 检测框 ×H 检测框 Formula (6)

[0173] Among them, W 检测框 H represents the width of the detection box. 检测框 This indicates the height of the detection frame.

[0174] Calculate the size S of the depth map 深度图 Formula (2) is shown below:

[0175] S 深度图 =W 深度图 ×H 深度图 Formula (2)

[0176] Among them, W 深度图 H represents the width of the depth map. 深度图 This indicates the height of the depth map.

[0177] The formula (7) for calculating the second ratio S between the detection box and the depth map is shown below:

[0178]

[0179] Where b2 is the preset second weight.

[0180] Determine the depth value d of each pixel in the depth map corresponding to each detection box, where the depth value d of each pixel is greater than a preset depth threshold dt. Based on the depth value d of each pixel and the total number of pixels count, perform averaging to determine the average depth for each detection box.

[0181] Calculate the average depth corresponding to the detection box Formula (8) is shown below:

[0182]

[0183] Where d represents the depth value of a pixel, and the depth value of a pixel is in the range of [0, 255]. The depth value d of the pixel is greater than the preset depth threshold dt, so that pixels with too small a depth value can be blocked; set D represents the set of all pixels in the detection box whose depth value d is greater than the preset depth threshold dt; count represents the total number of all pixels in the detection box whose depth value d is greater than the preset depth threshold dt.

[0184] In one embodiment, the second depth cost matrix corresponding to all detection boxes is determined based on each second ratio and each depth average, including:

[0185] The cost value for each detection box is obtained by summing the second ratio corresponding to each detection box and the average depth corresponding to each detection box.

[0186] Based on the cost value corresponding to each detection box, the second depth cost matrix corresponding to all detection boxes is obtained.

[0187] Specifically, the second ratio S corresponding to each detection box and the average depth corresponding to each detection box are... Summing is performed to obtain the cost corresponding to each detection box. Based on the cost of each detection frame The second depth cost matrix dc2 corresponding to all detection boxes is obtained.

[0188] Based on each second ratio (S) and the average value of each depth The second depth cost matrix dc2 corresponding to all detection boxes is determined, and the formula (9) for calculating the second depth cost matrix dc2 is as follows:

[0189]

[0190] in, The values ​​represent the average values ​​at each depth, S1, S2, S3, ... S n Indicates the second ratio, This represents the cost corresponding to the prediction box n, where n is a positive integer.

[0191] In one embodiment, a fusion process is performed based on the intersection-union cost matrix, the first depth cost matrix, and the second depth cost matrix to determine the fusion cost matrix, including:

[0192] The difference between the first depth cost matrix and the second depth cost matrix is ​​used to obtain the third depth cost matrix;

[0193] The third depth cost matrix and the intersection-over-union (IoU) cost matrix are normalized to obtain the normalized third depth cost matrix and the normalized IoU cost matrix.

[0194] The normalized third-depth cost matrix and the normalized intersection-union cost matrix are fused together to determine the fused cost matrix.

[0195] Specifically, the difference between the first depth cost matrix dc1 and the second depth cost matrix dc2 is calculated, that is, the first depth cost matrix dc1 and the second depth cost matrix dc2 are subtracted bit by bit to obtain the third depth cost matrix; the third depth cost matrix can be dc1-dc2 or dc2-dc1.

[0196] It should be noted that since the third depth cost matrix and the intersection-union ratio cost matrix are matrices of different data magnitudes, they can be normalized to the same order of magnitude through normalization, thus obtaining the normalized third depth cost matrix Dc and the normalized intersection-union ratio cost matrix M.

[0197] In one embodiment, the normalized third-depth cost matrix and the normalized intersection-union cost matrix are fused to determine the fused cost matrix, including:

[0198] The fusion cost matrix is ​​determined based on the normalized third-depth cost matrix, the third weight corresponding to the normalized third-depth cost matrix, the normalized intersection-union ratio cost matrix, and the fourth weight corresponding to the normalized intersection-union ratio cost matrix.

[0199] Specifically, the fusion cost matrix K is determined based on the normalized third depth cost matrix Dc, the third weight α1 corresponding to the normalized third depth cost matrix, the normalized intersection-union ratio (CUI) cost matrix M, and the fourth weight α2 corresponding to the normalized CUI cost matrix. The formula (10) for calculating the fusion cost matrix K is shown below:

[0200] K = Dc × α1 + M × α2 (Formula 10)

[0201] For example, α1+α2=1.

[0202] In one embodiment, determining the tracking bounding box corresponding to each of the multiple target objects based on the fusion cost matrix includes:

[0203] The fusion cost matrix is ​​processed using the Hungarian algorithm to determine the prediction box and detection box for each target object among multiple target objects.

[0204] Based on the predicted bounding box and the detection bounding box corresponding to each target object, the tracking bounding box corresponding to each target object is determined.

[0205] Specifically, the fusion cost matrix is ​​used to match the predicted bounding boxes and the detected bounding boxes using the Hungarian algorithm. For example, in the previous frame of the target frame, there are 5 target objects, and the tracking boxes of these 5 target objects are tracking box A, tracking box B, tracking box C, tracking box D, and tracking box E. The fusion cost matrix corresponding to the target frame is used to match the corresponding prediction box and detection box for each of these 5 target objects through the Hungarian algorithm, and the matching results are obtained. The matching results are as follows: prediction box A1 and detection box A2 match each other, prediction box B1 and detection box B2 match each other, prediction box C1 and detection box C2 match each other, prediction box D1 and detection box D2 match each other, and prediction box E1 and detection box E2 match each other. Tracking box A, prediction box A1, and detection box A2 are associated with data; tracking box B, prediction box B1, and detection box B2 are associated with data; tracking box C, prediction box C1, and detection box C2 are associated with data; tracking box D, prediction box D1, and detection box D2 are associated with data; and tracking box E, prediction box E1, and detection box E2 are associated with data.

[0206] Based on the predicted bounding box and detection bounding box corresponding to each target object, post-processing is performed to determine the tracking bounding box corresponding to each target object. Post-processing can be mean calculation, for example, calculating the average value between the predicted bounding box A1 and the detection bounding box A2 to obtain the tracking bounding box A' in the target frame; for the target frame, the tracking bounding box A in the previous frame of the target frame is updated to the tracking bounding box A'.

[0207] Applying the embodiments of this application has at least the following beneficial effects:

[0208] The depth cost matrix (third depth cost matrix) is obtained by calculating the difference in depth information between the predicted bounding box (first depth cost matrix) and the detected bounding box (second depth cost matrix). The intersection-union cost matrix and the third depth cost matrix are normalized. Based on the normalized third depth cost matrix, its corresponding third weight, and the normalized intersection-union cost matrix and its corresponding fourth weight, the fusion cost matrix is ​​determined. The fusion cost matrix is ​​then fed into the Hungarian algorithm for optimal matching. Because depth information is incorporated, the Hungarian algorithm can effectively find the matching result with the least depth value loss. This matching result fully considers the depth values ​​between multiple tracking boxes, avoiding abrupt changes in depth values ​​when different tracking boxes are updated. This prevents ID switching problems after target overlap and improves the accuracy of multi-target tracking.

[0209] To better understand the methods provided in the embodiments of this application, the solutions of the embodiments of this application will be further explained below with reference to specific application scenarios.

[0210] In a specific application scenario, such as a multi-target tracking scenario, see [link to relevant documentation]. Figure 3 This illustrates the processing flow of a multi-target tracking method, such as... Figure 3 As shown, the processing flow of the multi-target tracking method provided in this application embodiment includes the following steps:

[0211] S301, the terminal determines the depth map corresponding to the target image, all predicted bounding boxes corresponding to the target image, and all detection bounding boxes corresponding to the target image.

[0212] Specifically, the image corresponding to the target frame is the target image. The target image is input into the depth model to extract depth information, resulting in a depth map corresponding to the target image. The depth map includes the depth information of each target object among multiple target objects. The tracking box corresponding to the previous frame of the target frame and the target image are processed by Kalman filtering to obtain all predicted boxes corresponding to the target image. The target image is input into the detection model for detection processing to obtain all detection boxes corresponding to the target image.

[0213] S302, the terminal determines the intersection-union cost matrix based on all predicted bounding boxes and all detected bounding boxes; the terminal determines the first depth cost matrix corresponding to all predicted bounding boxes based on the depth map and all predicted bounding boxes; the terminal determines the second depth cost matrix corresponding to all detected bounding boxes based on the depth map and all detected bounding boxes.

[0214] S303, the terminal calculates the difference between the first depth cost matrix and the second depth cost matrix to obtain the third depth cost matrix; the third depth cost matrix and the intersection-union ratio cost matrix are normalized respectively to obtain the normalized third depth cost matrix and the normalized intersection-union ratio cost matrix.

[0215] S304, the terminal performs a fusion process on the normalized third-depth cost matrix and the normalized intersection-union cost matrix to determine the fused cost matrix.

[0216] Specifically, the fusion cost matrix is ​​determined based on the normalized third-depth cost matrix, the third weight corresponding to the normalized third-depth cost matrix, the normalized intersection-union ratio cost matrix, and the fourth weight corresponding to the normalized intersection-union ratio cost matrix.

[0217] S305, the terminal processes the fusion cost matrix using the Hungarian algorithm to obtain the matching result.

[0218] Specifically, for example, in the previous frame of the target frame, there are 5 target objects, and the tracking boxes of these 5 target objects are tracking box A, tracking box B, tracking box C, tracking box D and tracking box E. The fusion cost matrix corresponding to the target frame is used to match the corresponding prediction box and detection box for each of these 5 target objects through the Hungarian algorithm, and the matching results are obtained. The matching results are: prediction box A1 and detection box A2 match each other, prediction box B1 and detection box B2 match each other, prediction box C1 and detection box C2 match each other, prediction box D1 and detection box D2 match each other, and prediction box E1 and detection box E2 match each other.

[0219] S306, the terminal associates the tracking box and matching result from the previous frame.

[0220] Specifically, for example, data association is performed on tracking box A, prediction box A1, and detection box A2; data association is performed on tracking box B, prediction box B1, and detection box B2; data association is performed on tracking box C, prediction box C1, and detection box C2; data association is performed on tracking box D, prediction box D1, and detection box D2; and data association is performed on tracking box E, prediction box E1, and detection box E2.

[0221] S307, the terminal performs post-processing, determines the tracking box corresponding to each target object in the target frame, and updates the tracking box in the previous frame of the target frame.

[0222] Specifically, for example, the average value between the predicted box A1 and the detected box A2 is calculated to obtain the tracking box A' in the target frame; for the target frame, the tracking box A in the previous frame of the target frame is updated to the tracking box A'.

[0223] Applying the embodiments of this application has at least the following beneficial effects:

[0224] By incorporating depth information from the depth map into the fusion cost matrix, the IDSwitch problem that occurs after target overlap is avoided, thus improving the accuracy of multi-target tracking.

[0225] This application also provides a multi-target tracking device, the structural schematic diagram of which is shown below. Figure 4 As shown, the multi-target tracking device 40 includes a first processing module 401, a second processing module 402, a third processing module 403, a fourth processing module 404, and a fifth processing module 405.

[0226] The first processing module 401 is used to acquire the image corresponding to the target frame, the image including multiple target objects;

[0227] The second processing module 402 is used to determine the depth map corresponding to the image, all prediction boxes corresponding to the image, and all detection boxes corresponding to the image.

[0228] The third processing module 403 is used to determine the intersection-union cost matrix based on all predicted boxes and all detected boxes; determine the first depth cost matrix corresponding to all predicted boxes based on the depth map and all predicted boxes; and determine the second depth cost matrix corresponding to all detected boxes based on the depth map and all detected boxes.

[0229] The fourth processing module 404 is used to perform fusion processing based on the intersection-union cost matrix, the first depth cost matrix, and the second depth cost matrix to determine the fusion cost matrix;

[0230] The fifth processing module 405 is used to determine the tracking box corresponding to each of the multiple target objects based on the fusion cost matrix.

[0231] In one embodiment, the second processing module 402 is specifically used for:

[0232] The image is input into a depth model to extract depth information, resulting in a depth map. This depth map includes the depth information of each of the multiple target objects; and / or

[0233] The tracking bounding box and image corresponding to the previous frame of the target frame are processed by Kalman filtering to obtain all predicted bounding boxes corresponding to the image; and / or

[0234] The image is input into the detection model for detection processing, and all detection boxes corresponding to the image are obtained.

[0235] In one embodiment, the third processing module 403 is specifically used for:

[0236] The cross-union (CUI) cost matrix is ​​obtained by performing cross-union comparison (CUC) on all predicted bounding boxes and all detected bounding boxes.

[0237] In one embodiment, the third processing module 403 is specifically used for:

[0238] Determine the first ratio between each prediction box and the depth map in all prediction boxes, and the average depth value corresponding to each prediction box;

[0239] Based on each first ratio and each depth average, the first depth cost matrix corresponding to all prediction boxes is determined.

[0240] In one embodiment, the third processing module 403 is specifically used for:

[0241] Based on the size of each prediction box in all prediction boxes, the size of the depth map, and a preset first weight, determine the first ratio between each prediction box and the depth map;

[0242] Determine the depth value of each pixel in the depth map corresponding to each prediction box, and the depth value of each pixel is greater than a preset depth threshold;

[0243] Based on the depth value of each pixel and the total number of pixels, the average depth value corresponding to each prediction box is determined by averaging.

[0244] In one embodiment, the third processing module 403 is specifically used for:

[0245] The cost value for each prediction box is obtained by summing the first ratio corresponding to each prediction box and the average depth corresponding to each prediction box.

[0246] Based on the cost value corresponding to each prediction box, the first depth cost matrix corresponding to all prediction boxes is obtained.

[0247] In one embodiment, the third processing module 403 is specifically used for:

[0248] Determine the second ratio between each detection box and the depth map in all detection boxes, and the average depth value corresponding to each detection box;

[0249] Based on each second ratio and each depth average, the second depth cost matrix corresponding to all detection boxes is determined.

[0250] In one embodiment, the third processing module 403 is specifically used for:

[0251] Based on the size of each detection box in all detection boxes, the size of the depth map, and a preset second weight, a second ratio between each detection box and the depth map is determined.

[0252] Determine the depth value of each pixel in the depth map corresponding to each detection box, and the depth value of each pixel is greater than a preset depth threshold;

[0253] Based on the depth value of each pixel and the total number of pixels, an average value is calculated to determine the average depth value for each detection box.

[0254] In one embodiment, the third processing module 403 is specifically used for:

[0255] The cost value for each detection box is obtained by summing the second ratio corresponding to each detection box and the average depth corresponding to each detection box.

[0256] Based on the cost value corresponding to each detection box, the second depth cost matrix corresponding to all detection boxes is obtained.

[0257] In one embodiment, the fourth processing module 404 is specifically used for:

[0258] The difference between the first depth cost matrix and the second depth cost matrix is ​​used to obtain the third depth cost matrix;

[0259] The third depth cost matrix and the intersection-over-union (IoU) cost matrix are normalized to obtain the normalized third depth cost matrix and the normalized IoU cost matrix.

[0260] The normalized third-depth cost matrix and the normalized intersection-union cost matrix are fused together to determine the fused cost matrix.

[0261] In one embodiment, the fourth processing module 404 is specifically used for:

[0262] The fusion cost matrix is ​​determined based on the normalized third-depth cost matrix, the third weight corresponding to the normalized third-depth cost matrix, the normalized intersection-union ratio cost matrix, and the fourth weight corresponding to the normalized intersection-union ratio cost matrix.

[0263] In one embodiment, the fifth processing module 405 is specifically used for:

[0264] The fusion cost matrix is ​​processed using the Hungarian algorithm to determine the prediction box and detection box for each target object among multiple target objects.

[0265] Based on the predicted bounding box and the detection bounding box corresponding to each target object, the tracking bounding box corresponding to each target object is determined.

[0266] Applying the embodiments of this application has at least the following beneficial effects:

[0267] The process involves acquiring the image corresponding to the target frame, which includes multiple target objects; determining the depth map, all predicted bounding boxes, and all detected bounding boxes corresponding to the image; determining the Cross-Union Comparison (CUC) cost matrix based on the predicted and detected bounding boxes; determining the first depth cost matrix corresponding to all predicted bounding boxes based on the depth map and all predicted bounding boxes; determining the second depth cost matrix corresponding to all detected bounding boxes based on the depth map and all detected bounding boxes; fusing the CUC cost matrix, the first depth cost matrix, and the second depth cost matrix to determine the fused cost matrix; and determining the tracking box corresponding to each of the multiple target objects based on the fused cost matrix. By incorporating depth information from the depth map into the fused cost matrix, the problem of ID switching after target (object) overlap is avoided, thus improving the accuracy of multi-target (multiple target objects) tracking.

[0268] This application also provides an electronic device, the structural schematic diagram of which is shown below. Figure 5 As shown, Figure 5The illustrated electronic device 4000 includes a processor 4001 and a memory 4003. The processor 4001 and the memory 4003 are connected, for example, via a bus 4002. Optionally, the electronic device 4000 may further include a transceiver 4004, which can be used for data interaction between the electronic device and other electronic devices, such as sending and / or receiving data. It should be noted that in practical applications, the transceiver 4004 is not limited to one type, and the structure of the electronic device 4000 does not constitute a limitation on the embodiments of this application.

[0269] Processor 4001 may be a CPU (Central Processing Unit), a general-purpose processor, a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. It can implement or execute the various exemplary logic blocks, modules, and circuits described in conjunction with the disclosure of this application. Processor 4001 may also be a combination that implements computational functions, such as including one or more microprocessor combinations, a combination of a DSP and a microprocessor, etc.

[0270] Bus 4002 may include a pathway for transmitting information between the aforementioned components. Bus 4002 may be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus, etc. Bus 4002 can be divided into address bus, data bus, control bus, etc. For ease of representation, Figure 5 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0271] The memory 4003 may be ROM (Read Only Memory) or other types of static storage devices capable of storing static information and instructions, RAM (Random Access Memory) or other types of dynamic storage devices capable of storing information and instructions, or EEPROM (Electrically Erasable Programmable Read Only Memory), CD-ROM (Compact Disc Read Only Memory) or other optical disc storage, optical disc storage (including compressed optical discs, laser discs, optical discs, digital universal optical discs, Blu-ray discs, etc.), magnetic disk storage media, other magnetic storage devices, or any other medium capable of carrying or storing computer programs and capable of being read by a computer, without limitation herein.

[0272] The memory 4003 stores computer programs that execute embodiments of this application, and its execution is controlled by the processor 4001. The processor 4001 executes the computer programs stored in the memory 4003 to implement the steps shown in the foregoing method embodiments.

[0273] Electronic devices include, but are not limited to, terminals.

[0274] Applying the embodiments of this application has at least the following beneficial effects:

[0275] The process involves acquiring the image corresponding to the target frame, which includes multiple target objects; determining the depth map, all predicted bounding boxes, and all detected bounding boxes corresponding to the image; determining the Cross-Union Comparison (CUC) cost matrix based on the predicted and detected bounding boxes; determining the first depth cost matrix corresponding to all predicted bounding boxes based on the depth map and all predicted bounding boxes; determining the second depth cost matrix corresponding to all detected bounding boxes based on the depth map and all detected bounding boxes; fusing the CUC cost matrix, the first depth cost matrix, and the second depth cost matrix to determine the fused cost matrix; and determining the tracking box corresponding to each of the multiple target objects based on the fused cost matrix. By incorporating depth information from the depth map into the fused cost matrix, the problem of ID switching after target (object) overlap is avoided, thus improving the accuracy of multi-target (multiple target objects) tracking.

[0276] This application provides a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, it can implement the steps and corresponding content of the aforementioned method embodiments.

[0277] This application also provides a computer program product, including a computer program that, when executed by a processor, can implement the steps and corresponding content of the aforementioned method embodiments.

[0278] Based on the same principles as the methods provided in the embodiments of this application, the embodiments of this application also provide a computer program product or computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the methods provided in any of the optional embodiments of this application described above.

[0279] It should be understood that although arrows indicate various operation steps in the flowcharts of this application's embodiments, the order in which these steps are implemented is not limited to the order indicated by the arrows. Unless explicitly stated herein, in some implementation scenarios of this application's embodiments, the implementation steps in each flowchart can be executed in other orders as required. Furthermore, some or all steps in each flowchart, based on the actual implementation scenario, may include multiple sub-steps or multiple stages. Some or all of these sub-steps or stages can be executed at the same time, and each sub-step or stage can also be executed at different times. In scenarios where execution times differ, the execution order of these sub-steps or stages can be flexibly configured according to requirements, and this application's embodiments do not limit this.

[0280] The above description is only an optional implementation method for some implementation scenarios of this application. It should be noted that for those skilled in the art, other similar implementation methods based on the technical concept of this application without departing from the technical concept of this application also fall within the protection scope of the embodiments of this application.

Claims

1. A multi-target tracking method, characterized in that, include: Acquire the image corresponding to the target frame, wherein the image includes multiple target objects; Determine the depth map corresponding to the image, all predicted bounding boxes corresponding to the image, and all detected bounding boxes corresponding to the image; Based on all predicted bounding boxes and all detected bounding boxes, determine the intersection-union cost matrix; based on the depth map and all predicted bounding boxes, determine the first depth cost matrix corresponding to all predicted bounding boxes; Based on the depth map and all the detection boxes, determine the second depth cost matrix corresponding to all the detection boxes; Based on the intersection-union cost matrix, the first depth cost matrix, and the second depth cost matrix, a fusion process is performed to determine the fusion cost matrix; Based on the fusion cost matrix, the tracking bounding box corresponding to each of the plurality of target objects is determined; The step of determining the first depth cost matrix corresponding to all predicted bounding boxes based on the depth map and all predicted bounding boxes includes: Determine a first ratio between each prediction box and the depth map in all prediction boxes, and the average depth value corresponding to each prediction box; The cost value corresponding to each prediction box is obtained by summing the first ratio corresponding to each prediction box and the average depth corresponding to each prediction box; Based on the cost value corresponding to each prediction box, the first depth cost matrix corresponding to all prediction boxes is obtained; The step of determining the second depth cost matrix corresponding to all detection boxes based on the depth map and all detection boxes includes: Determine a second ratio between each prediction box in all prediction boxes and the depth map, and the average depth value corresponding to each prediction box; The cost value corresponding to each prediction box is obtained by summing the second ratio corresponding to each prediction box and the average depth corresponding to each prediction box. Based on the cost value corresponding to each prediction box, the second depth cost matrix corresponding to all prediction boxes is obtained; The step of performing fusion processing based on the intersection-union cost matrix, the first depth cost matrix, and the second depth cost matrix to determine the fusion cost matrix includes: The difference between the first depth cost matrix and the second depth cost matrix is ​​used to obtain the third depth cost matrix; The third depth cost matrix and the intersection-union ratio cost matrix are normalized respectively to obtain the normalized third depth cost matrix and the normalized intersection-union ratio cost matrix. The fusion cost matrix is ​​determined based on the normalized third depth cost matrix, the third weight corresponding to the normalized third depth cost matrix, the normalized intersection-union ratio cost matrix, and the fourth weight corresponding to the normalized intersection-union ratio cost matrix.

2. The method according to claim 1, characterized in that, Determining the depth map corresponding to the image, all predicted bounding boxes corresponding to the image, and all detected bounding boxes corresponding to the image includes: The image is input into a depth model to extract depth information, resulting in a depth map corresponding to the image. The depth map includes the depth information of each of the plurality of target objects; and / or The tracking bounding box corresponding to the previous frame of the target frame and the image are processed by Kalman filtering to obtain all predicted bounding boxes corresponding to the image; and / or The image is input into the detection model for detection processing to obtain all the detection boxes corresponding to the image.

3. The method according to claim 1, characterized in that, Determining the first ratio between each prediction box and the depth map in all prediction boxes, and the average depth value corresponding to each prediction box, includes: Based on the size of each prediction box in all prediction boxes, the size of the depth map, and a preset first weight, a first ratio between each prediction box and the depth map is determined; Determine the depth value of each pixel in the depth map corresponding to each prediction box, wherein the depth value of each pixel is greater than a preset depth threshold; Based on the depth value of each pixel and the total number of pixels, an average value is calculated to determine the average depth value corresponding to each prediction box.

4. An electronic device comprising a memory, a processor, and a computer program stored in the memory, characterized in that, The processor executes the computer program to implement the steps of the method according to any one of claims 1-3.

5. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1-3.

6. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1-3.

Images