Target tracking method, system and electronic device

Abstract

The application provides a target tracking method, a system and an electronic device, and relates to the field of computer vision processing. The target tracking method comprises the following steps: acquiring a detection box and a tracking box in a current frame of to-be-processed data; matching the detection box and the tracking box, and acquiring a first matching result; judging whether the first matching result is a matching failure; if the first matching result is a matching failure, performing low-bit quantization on a re-identification network model to obtain a quantized re-identification network model; inputting the detection box and the tracking box into the quantized re-identification network model to match the detection box and the tracking box again and acquire a second matching result; and if the second matching result is a matching success, adding a target track of the detection box to a tracking track. The target tracking method provided in the application embodiment can not only improve the matching accuracy of target tracking, but also reduce the calculation time consumption of matching.

Description

[0001] This application relates to the field of computer vision processing, and more specifically, to a target tracking method, system, and electronic device. Background Technology

[0002] Multi-object tracking primarily addresses the problem of detecting and positioning identifiable targets in each frame of a video, assigning each target an ID, and maintaining this ID throughout the target's movement. In multi-object tracking, the algorithm detects all targets in each frame of the video, matches newly detected targets with already assigned trajectories, and categorizes unmatched targets as newly appearing targets with new IDs. Targets that leave the video area are no longer tracked and are removed from the existing trajectories.

[0003] Currently, most target trajectory tracking methods use the intersection-union ratio to calculate the cost matrix and the Hungarian matching algorithm to obtain the matching result between the current frame detection box and the tracking trajectory in 3D multi-target tracking algorithms. However, this method has the problem of lacking instance semantic information.

[0004] While some existing target trajectory tracking methods incorporate image semantic information, these methods significantly reduce inference speed, resulting in poor real-time performance. Additionally, some deep learning-based 3D multi-target tracking algorithms have been proposed, but they suffer from high memory consumption and slow inference speed. Summary of the Invention

[0005] The purpose of this application is to provide a target tracking method, system, and electronic device. The target tracking method first matches a 3D detection box and a 3D tracking box. If the 3D detection box is not tracked, the 3D detection box is projected onto multiple cameras to obtain a 2D detection box. Furthermore, a re-identification network with low bit quantization is used to calculate the cost matrix for the second matching, and matching is performed again to obtain the matching result. This not only improves the matching accuracy of target tracking but also reduces the computation time of matching.

[0006] In a first aspect, embodiments of this application provide a target tracking method, which includes: acquiring a detection box and a tracking box in the current frame of data to be processed; matching the detection box and the tracking box and obtaining a first matching result; determining whether the first matching result is a matching failure; if the first matching result is a matching failure, performing low-bit quantization on the re-identification network model to obtain a quantized re-identification network model; inputting the detection box and the tracking box into the quantized re-identification network model to match the detection box and the tracking box again and obtaining a second matching result; if the second matching result is a successful match, adding the target trajectory of the detection box to the tracking trajectory.

[0007] In the above implementation process, the 3D detection box and the 3D tracking box are matched for the first time. If the 3D detection box does not match, the cost matrix for the second matching is calculated using a re-identification network after low-bit quantization, and the matching is performed again to obtain the matching result. Therefore, the target tracking method provided by the embodiments of this application can not only improve the matching accuracy of target tracking, but also reduce the computation time caused by the introduction of the re-identification network.

[0008] Optionally, in this embodiment, the detection box includes a 3D detection box and a 2D detection box; the tracking box includes a 3D tracking box and a 2D tracking box; obtaining the detection box and tracking box in the current frame based on the data to be processed includes: obtaining the current information and historical information of the target trajectory in the current frame based on the data to be processed; obtaining the 3D detection box based on the current information and projecting the 3D detection box onto the multi-camera coordinate system to obtain the 2D detection box; obtaining the 3D tracking box and the 2D tracking box based on the historical information and the 3D detection box.

[0009] In the above implementation process, the detection box and tracking box in the current frame can be obtained through the data to be processed, and the 3D detection box and 2D detection box can be obtained through the current information; the 3D tracking box and 2D tracking box can be obtained based on historical information and the 3D detection box; thus, the 3D detection box, 2D detection box, 3D tracking box and 2D tracking box can be accurately obtained, which is convenient for accurate target tracking.

[0010] Optionally, in this embodiment, the data to be processed includes LiDAR point cloud data; projecting the 3D detection box onto the multi-camera coordinate system to obtain a 2D detection box includes: projecting the 3D detection box onto the multi-camera coordinate system to obtain multiple projected 2D boxes corresponding to the 3D detection box; obtaining the number of point clouds in the 3D detection box and the number of point clouds in the multiple projected 2D boxes; dividing the number of point clouds in the 3D detection box by the number of point clouds in the multiple projected 2D boxes to obtain the occlusion ratio; and taking the projected 2D box corresponding to the smallest occlusion ratio as the 2D detection box.

[0011] In the above implementation process, the 3D detection box is projected onto the multi-camera coordinate system. When one 3D box corresponds to multiple 2D boxes, the 2D projection box with the smallest occlusion ratio is selected as the input of the re-identification network. The occlusion ratio is obtained by dividing the number of point clouds in the 3D detection box in the 3D coordinate system by the number of point clouds in the 2D detection box projected onto the camera coordinate system. Since the target tracking method provided in this application selects the 2D box with the smallest occlusion ratio as the 2D detection box, the accuracy of 3D multi-target tracking and matching can be improved.

[0012] Optionally, in this embodiment of the application, passing the detection box and the tracking box through a quantized re-identification network model to re-match the detection box and the tracking box and obtain a second matching result includes: passing the 2D detection box and the 2D tracking box through the quantized re-identification network model to obtain the semantic features of the 2D detection box instance and the semantic features of the 2D tracking box instance; calculating the cosine distance between the semantic features of the 2D detection box instance and the semantic features of the 2D tracking box instance to obtain a semantic feature cost matrix; and based on the semantic feature cost matrix, re-matching the 2D detection box and the 2D tracking box and obtaining a second matching result.

[0013] In the above implementation process, the detection box and tracking box are processed through a quantized re-identification network model to obtain the semantic features of 2D detection box instances and 2D tracking box instances. Furthermore, the detection box and tracking box are matched again to obtain a second matching result. Therefore, the target tracking method provided in this application introduces instance-level semantic features, overcoming the problem of missing instance semantic information in the existing 3D multi-target tracking matching process.

[0014] Optionally, in this embodiment of the application, matching the detection box and the tracking box and obtaining the first matching result includes: calculating the cross-union ratio distance between the 3D detection box and the 3D tracking box to obtain the 3D cross-union cost matrix; matching the 3D detection box and the 3D tracking box based on the 3D cross-union cost matrix and obtaining the first matching result.

[0015] In the above implementation process, the target tracking method provided in this application embodiment involves two matching operations. The first is a preliminary matching. The first matching uses a 3D IOU cost matrix. After the first matching, there are some detection boxes and tracking boxes that fail to match. For the detection boxes and tracking boxes that fail to match, a second matching is performed. In other words, the target tracking method provided in this application embodiment can achieve preliminary matching through the first matching, thereby reducing the amount of data during the second matching and improving matching efficiency.

[0016] Optionally, in this embodiment of the application, the target tracking method further includes: if the first matching result is a successful match, then adding the target in the 3D detection box to the tracking trajectory.

[0017] In the above implementation process, the cost matrix of the intersection-union cost matrix fusion is matched using the Hungarian algorithm, and a matching matrix of the same size as the input is obtained based on the Hungarian matching algorithm. If the detection box matches a certain trajectory, the corresponding row and column of the output matrix are set to 1. The re-identification features of the matched trajectory are dynamically updated using the corresponding detection re-identification features, and the target in the 3D detection box is added to the tracking trajectory.

[0018] Optionally, in this embodiment of the application, performing low-bit quantization on the re-identification network model to obtain a quantized re-identification network model includes: mapping the weights and / or activation functions of the re-identification network model from high-bit floating-point numbers to low-bit-depth data to obtain a quantized re-identification network model; wherein the weights are quantized using symmetric quantization and the activation functions are quantized using asymmetric quantization.

[0019] In the above implementation process, the introduction of a re-identification network to extract semantic features increases computation time and reduces target tracking efficiency. The weights and / or activation functions of the re-identification network model are mapped from high-bit floating-point numbers to low-bit-depth data to obtain a quantized re-identification network model, thereby effectively solving the time consumption problem caused by the introduction of the re-identification network.

[0020] Secondly, embodiments of this application provide a target tracking system, which includes: a detection box acquisition module, a tracking box acquisition module, a first matching module, a low-bit quantization module, a second matching module, and a matching result recognition module; the detection box acquisition module is used to acquire detection boxes in the current frame of the data to be processed; the tracking box acquisition module is used to acquire tracking boxes in the current frame based on lidar point cloud data; the first matching module is used to match the detection boxes and tracking boxes and acquire a first matching result; the matching result recognition module is used to determine whether the first matching result is a matching failure; the low-bit quantization module is used to perform low-bit quantization on the re-identification network model when the first matching result is a matching failure, so as to obtain a quantized re-identification network model; the second matching module is used to input the detection boxes and tracking boxes into the quantized re-identification network model to match the detection boxes and tracking boxes again and acquire a second matching result; the matching result recognition module is also used to add the target trajectory of the detection box to the tracking trajectory when the second matching result is a successful match.

[0021] Thirdly, embodiments of this application provide an electronic device, which includes a memory and a processor. The memory stores program instructions, and when the processor reads and runs the program instructions, it executes the steps in any of the above implementation methods.

[0022] Fourthly, embodiments of this application also provide a computer-readable storage medium storing computer program instructions, which, when read and executed by a processor, perform the steps in any of the above implementations. Attached Figure Description

[0023] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments of this application will be briefly introduced below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0024] Figure 1 A first flowchart of the target tracking method provided in the embodiments of this application;

[0025] Figure 2 A flowchart illustrating the acquisition of the detection box and tracking box provided in the embodiments of this application;

[0026] Figure 3 This is a schematic diagram of a 3D detection frame provided in an embodiment of this application;

[0027] Figure 4 This is a flowchart illustrating the 2D detection frame acquisition process provided in an embodiment of this application.

[0028] Figure 5 A flowchart illustrating the re-matching process provided in this application embodiment;

[0029] Figure 6 A flowchart of the initial matching process provided in this application embodiment;

[0030] Figure 7 A second flowchart of the target tracking method provided in the embodiments of this application;

[0031] Figure 8 This is a schematic diagram of the modules of the target tracking system provided in the embodiments of this application;

[0032] Figure 9 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0033] The technical solutions of the embodiments of this application will now be described with reference to the accompanying drawings. For example, the flowcharts and block diagrams in the drawings illustrate the architecture, functions, and operations of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in the flowchart or block diagram may represent a module, program segment, or part of code, which contains one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions marked in the blocks may occur in a different order than those marked in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagram and / or flowchart, and combinations of blocks in the block diagram and / or flowchart, can be implemented using a dedicated hardware-based system that performs the specified function or action, or can be implemented using a combination of dedicated hardware and computer instructions. In addition, the functional modules in the various embodiments of the present invention may be integrated together to form an independent part, or each module may exist separately, or two or more modules may be integrated to form an independent part.

[0034] Current 3D multi-target tracking algorithms need to match the trajectory of existing targets based on the detection results of targets in each frame of the LiDAR point cloud; for newly appearing targets, new trajectories need to be generated; and for targets that have exceeded the LiDAR's sensing range, tracking of that trajectory needs to be terminated.

[0035] During the research process, the applicant discovered that current target trajectory tracking algorithms mostly use intersection-union ratio (IoU) to calculate the cost matrix and then use the Hungarian matching algorithm to obtain the matching results between the current frame detection box and the tracking trajectory in 3D multi-target tracking. This method only matches based on the spatial location information of the 3D target detection box and does not incorporate image features into the 3D target. Currently, some studies have begun to incorporate image semantic features into the cost matrix calculation, but they lack instance-level re-id features, have poor modeling of feature distances between samples, and significantly reduce inference speed, resulting in poor real-time performance. In addition, many deep learning-based 3D multi-target tracking algorithms have been proposed, but they suffer from problems such as large memory consumption and slow inference speed.

[0036] Based on this, embodiments of this application provide a target tracking method, system, electronic device, and computer storage medium. The target tracking method first matches a 3D detection box and a 3D tracking box. If the matched 3D detection box is not tracked, the 3D detection box is projected onto multiple cameras to obtain a 2D detection box. Further, a re-identification network with low-bit quantization is used to calculate the cost matrix for the second matching, and matching is performed again to obtain the matching result.

[0037] Please refer to Figure 1 , Figure 1 A flowchart of a target tracking method provided in this application embodiment; the target tracking method includes the following steps:

[0038] Step S100: Obtain the detection box and tracking box in the current frame of the data to be processed.

[0039] In step S100 above, the detection box and tracking box in the current frame of the data to be processed are obtained. It should be noted that the detection box is the detection box for the target trajectory in the frame to be processed, while the tracking box is the trajectory box that is to be predicted or matched from the data to be processed.

[0040] To better understand tracking boxes and detection boxes, we will use a target vehicle in a video as an example. A detection box can be understood as a static concept, which mainly identifies the specific position of a vehicle in a single image. A tracking box, on the other hand, is a dynamic concept that focuses on the relationship between the positions of cars in images within a continuous video stream.

[0041] Step S101: Match the detection box with the tracking box and obtain the first matching result.

[0042] In step S101 above, after obtaining the detection box and the tracking box, the tracking box is matched with the detection box, and the first matching result is obtained.

[0043] Step S102: Determine whether the first matching result is a failed match.

[0044] Step S103: If the first matching result is a matching failure, then perform low-bit quantization on the re-identification network model to obtain the quantized re-identification network model.

[0045] In steps S102-S103 above, if the tracking box and the detection box fail to match, then a re-identification network with low-bit quantization is used to process the test data and thus achieve a second match.

[0046] Step S104: Input the detection box and tracking box into the quantized re-identification network model to match the detection box and tracking box again and obtain the second matching result.

[0047] Step S105: If the second matching result is a successful match, then add the target of the detection box to the tracking trajectory.

[0048] In steps S104-S105 above, the detection box and the tracking box are input into the low-bit quantized re-identification network model to perform a second matching of the detection box and the tracking box, and obtain the second matching result. If the second matching result is successful, the target trajectory in the detection box is added to the tracking trajectory.

[0049] In other words, in the target tracking method provided in this application, if the detection box and tracking box cannot be matched in the first match, a second Hungarian matching is performed using their instance semantic feature cost matrix. For the matched detection box and tracking box, the target in the detection box is added to the tracking trajectory; if neither match is found, it is considered a new target, indicating that a new trajectory needs to be created; for trajectories where no object appears for more than a fixed number of frames, the matching for that trajectory is terminated.

[0050] pass Figure 1 As can be seen, the 3D detection box and the 3D tracking box are matched for the first time. If the 3D detection box is not tracked, the 3D detection box is projected onto multiple cameras to obtain a 2D detection box. Further, the cost matrix for the second matching is calculated using a low-bit quantized re-identification network, and the matching is performed again to obtain the matching result. Therefore, the target tracking method provided in this application embodiment can not only improve the matching accuracy of target tracking, but also reduce the computation time of matching.

[0051] Please refer to Figure 2 , Figure 2 This is a flowchart illustrating the acquisition of the detection box and tracking box provided in the embodiments of this application. Please refer to it. Figure 3 , Figure 3 This is a schematic diagram of a 3D detection frame provided in an embodiment of this application. Figure 3 The target object to be detected is a car, and the bounding box (preferably, the minimum bounding box) around the car is a 3D detection box 500. In an optional embodiment of this example, the detection box includes 3D detection boxes and 2D detection boxes; similarly, the tracking box includes 3D tracking boxes and 2D tracking boxes. The aforementioned acquisition of the detection boxes and tracking boxes in the current frame based on the data to be processed may include the following steps:

[0052] Step S200: Based on the data to be processed, obtain the current information and historical information of the target trajectory in the current frame.

[0053] In step S200 above, the current information and historical information of the target trajectory in the current frame are obtained based on the data to be processed.

[0054] It should be noted that the data to be processed in the target tracking method provided in this application embodiment can be various types of data, such as videos, images, etc.; and the purpose of obtaining the current information and historical information of the current frame through the data to be processed is to obtain the detection box and the tracking box.

[0055] Step S201: Obtain the 3D detection box based on the current information, and project the 3D detection box onto the multi-camera coordinate system to obtain the 2D detection box.

[0056] In step S201 above, after obtaining the current and historical information of the target trajectory in the current frame based on the data to be processed, the 3D detection bounding box is obtained based on the current information. It should be noted that, in the target tracking method provided in this application embodiment, after obtaining the data to be processed, the data to be processed is passed through a lidar detection network, and all target 2D detection bounding boxes can be obtained after passing through the lidar detection network.

[0057] Furthermore, the 3D detection box is projected onto the multi-camera coordinate system to obtain the 2D detection box.

[0058] Step S202: Obtain 3D tracking boxes and 2D tracking boxes based on historical information and 3D detection boxes.

[0059] In step S202 above, after obtaining the current information and historical information of the target trajectory in the current frame based on the data to be processed, the 3D tracking box and the 2D tracking box are obtained based on the historical information; those skilled in the art will understand that the Kalman filter algorithm can be used to obtain the 2D tracking box through the historical information.

[0060] Kalman filtering is an algorithm that uses the state equations of a linear system to optimally estimate the system state using system input and output observation data. Since the observation data includes noise and interference from the system, the optimal estimation can also be viewed as a filtering process. In other words, the tracking box used in this embodiment predicts the target's position in the next frame based on the position of the detection box, which is a linear process.

[0061] pass Figure 2 It can be seen that the detection box and tracking box in the current frame can be obtained through the data to be processed, and the 3D detection box and 2D detection box can be obtained through the current information; the 3D tracking box and 2D tracking box can be obtained based on historical information and the 3D detection box; thus, the 3D detection box, 2D detection box, 3D tracking box and 2D tracking box can be accurately obtained, which is convenient for accurate target tracking.

[0062] Please refer to Figure 4 , Figure 4This is a flowchart illustrating the 2D detection box acquisition process provided in this embodiment. In an optional implementation of this embodiment, the data to be processed includes LiDAR point cloud data. LiDAR point cloud data is a dataset of spatial points obtained by scanning with a 3D LiDAR device. Each point contains 3D coordinate information, which consists of three elements: X, Y, and Z. Some points may also contain color information, reflection intensity information, echo count information, etc. The aforementioned process of projecting the 3D detection box onto the multi-camera coordinate system to obtain the 2D detection box may include the following steps:

[0063] Step S300: Project the 3D detection box onto the multi-camera coordinate system to obtain multiple projected 2D boxes corresponding to the 3D detection box.

[0064] Step S301: Obtain the number of point clouds within the 3D detection box and the number of point clouds within multiple projected 2D boxes.

[0065] In steps S300-S301 above, the 3D detection box is projected onto the multi-camera coordinate system to obtain multiple projected 2D boxes corresponding to the 3D box. Those skilled in the art will understand that when the 3D detection box is projected onto the multi-camera coordinate system, one 3D box may correspond to multiple 2D boxes. Therefore, it is necessary to select the most suitable 2D box as the 2D detection box.

[0066] Step S302: Divide the number of point clouds in the 3D detection box by the number of point clouds in the multiple projected 2D boxes to obtain the occlusion ratio.

[0067] In step S302 above, as mentioned above, when the 3D detection box is projected onto the multi-camera coordinate system, one 3D box may correspond to multiple 2D boxes; therefore, it is necessary to select the most suitable 2D box as the 2D detection box. The target tracking method provided in this application embodiment obtains the number of point clouds in the 3D detection box and the number of point clouds in the projected 2D boxes; further, the number of point clouds in the 3D detection box is divided by the number of point clouds in the multiple projected 2D boxes respectively to obtain the occlusion ratio.

[0068] Step S303: Use the projected 2D box corresponding to the smallest occlusion ratio as the 2D detection box.

[0069] In step S303 above, the number of point clouds in the 3D detection box is divided by the number of point clouds in the multiple projected 2D boxes to obtain multiple occlusion ratios. Then, the 2D box corresponding to the smallest occlusion ratio among the multiple occlusion ratios is taken as the 2D detection box.

[0070] pass Figure 4As can be seen, when a 3D detection box is projected onto the multi-camera coordinate system, and one 3D box corresponds to multiple 2D boxes, the 2D projection box with the smallest occlusion ratio is selected as the input of the re-identification network. The occlusion ratio is obtained by dividing the number of point clouds in the 3D detection box in the 3D coordinate system by the number of point clouds in the 2D detection box projected onto the camera coordinate system. Since the target tracking method provided in this application selects the 2D box with the smallest occlusion ratio as the 2D detection box, the accuracy of 3D multi-target tracking and matching can be improved.

[0071] Please refer to Figure 5 , Figure 5 The flowchart for the rematching provided in this application embodiment; the above-described process of rematching the detection box and tracking box through a quantized re-identification network model to obtain a second matching result includes the following steps:

[0072] Step S400: Pass the 2D detection box and 2D tracking box through the quantized re-identification network model to obtain the semantic features of the 2D detection box instance and the semantic features of the 2D tracking box instance.

[0073] In step S400 above, the 2D detection box and the 2D tracking box are input into the quantized re-identification network model to obtain the semantic features of the 2D detection box instance and the semantic features of the 2D tracking box instance.

[0074] It should be noted that, in the embodiments of this application, for the historical information of each surviving trajectory, the trajectory predicts the current frame according to the target box position in the time sequence, and the target appearance position of the trajectory is used as the tracking box; since the instance semantic features have already been calculated in the data processing of the historical frame, there is no need to reproject to the 2D camera coordinate system and extract the instance semantic features; therefore, the 2D tracking box is not recalculated separately in the above steps.

[0075] Step S401: Calculate the cosine distance between the semantic features of the 2D detection box instance and the semantic features of the 2D tracking box instance to obtain the semantic feature cost matrix.

[0076] In step S401 above, the cosine distance between the semantic features of 2D detection boxes and the semantic features of 2D tracking boxes is calculated to obtain the semantic feature cost matrix. For example, if N detection boxes and M tracking boxes are obtained after passing through the quantized re-identification network model, the cosine distance between the instance semantic features of the N detection boxes and M tracking boxes is calculated as the instance semantic feature cost matrix. It can be understood that this instance semantic feature cost matrix has the number of rows as the number of detection boxes and the number of columns as the number of existing trajectories.

[0077] Step S402: Based on the semantic feature cost matrix, match the 2D detection box and the 2D tracking box again, and obtain the second matching result.

[0078] In step S402 above, based on the semantic feature cost matrix obtained above, the Hungarian matching algorithm is used to obtain the second matching result.

[0079] This section provides a brief introduction to the Hungarian Algorithm. Based on the sufficiency proof of Hall's Theorem (used to determine if a bipartite graph is a perfect match), the Hungarian Algorithm is one of the most common algorithms for partial graph matching. Its core principle is finding augmenting paths; it's an algorithm that uses augmenting paths to find the maximum matching in a bipartite graph. In the field of target tracking, the tracking part mainly consists of data association, which includes two aspects: association algorithms and association metrics. The Hungarian Algorithm is currently a commonly used association algorithm.

[0080] pass Figure 5 As can be seen, by passing the detection box and tracking box through a quantized re-identification network model, 2D detection box instance semantic features and 2D tracking box instance semantic features are obtained; further, the detection box and tracking box are matched again to obtain a second matching result. Therefore, the target tracking method provided in this application embodiment introduces instance-level semantic features, overcoming the problem of missing instance semantic information in the existing 3D multi-target tracking matching process.

[0081] Please refer to Figure 6 , Figure 6 The flowchart for the initial matching provided in this application embodiment; the aforementioned matching of the detection box and the tracking box, and obtaining the first matching result, includes the following steps:

[0082] Step S500: Calculate the cross-union distance between the 3D detection box and the 3D tracking box to obtain the 3D cross-union cost matrix.

[0083] In step S500 above, the matching method for matching the detection box and the tracking box is to calculate the cross-union ratio distance between the 3D detection box and the 3D tracking box, and then obtain the 3D cross-union cost matrix.

[0084] It's important to note that Intersection-over-Union (IoU), a concept used in object detection, is the overlap ratio between the generated candidate bounding boxes and the original ground truth bounding boxes; that is, the ratio of their intersection to their union. Ideally, they should overlap completely, meaning the ratio is 1.

[0085] Step S501: Based on the 3D intersection-union cost matrix, match the 3D detection box and the 3D tracking box, and obtain the first matching result.

[0086] In step S501 above, for each predicted box of a survival trajectory, the intersection-union ratio (IOU) is calculated pairwise with the detection box of the current frame as the 3D IOU cost matrix. Then, the IOU distance between the 3D detection box and the 3D tracking box is used as the cost matrix for the first matching, and the first matching result is obtained based on the Hungarian matching algorithm.

[0087] Those skilled in the art will understand that in the aforementioned scheme, the first matching result is obtained based on the Hungarian matching algorithm; specifically, the cost matrix is ​​used to obtain the matching matrix based on the Hungarian matching algorithm. If an element of the matching matrix is ​​1, it means that the matching at that position is successful, that is, the detection box corresponding to the row matches the trajectory prediction box in the column, and the re-identification features of the matched trajectory are updated with the corresponding target re-identification features.

[0088] pass Figure 6 As can be seen, the target tracking method provided in this application embodiment involves two matching operations. The first is a preliminary matching. The first matching uses a 3D IoU cost matrix. After the first matching, there are some detection boxes and tracking boxes that fail to match. For the detection boxes and tracking boxes that fail to match, a second matching is performed. In other words, the target tracking method provided in this application embodiment can achieve preliminary matching through the first matching, thereby reducing the amount of data during the second matching and improving matching efficiency.

[0089] In an optional embodiment, if the first matching result is a successful match, the target in the 3D detection box is added to the tracking trajectory.

[0090] In one possible embodiment, if a detection box does not match any trajectory, a new trajectory is generated; if the detection box corresponding to a trajectory does not appear for multiple frames, the trajectory is terminated.

[0091] In the above implementation, the cost matrix of the intersection-union cost matrix fusion is matched using the Hungarian algorithm, and a matching matrix of the same size as the input is obtained based on the Hungarian matching algorithm. If the detection box matches a certain trajectory, the corresponding row and column of the output matrix are set to 1. The re-identification features of the matched trajectory are dynamically updated using the corresponding detection re-identification features, and the target in the 3D detection box is added to the tracking trajectory.

[0092] In an optional embodiment, the aforementioned scheme of performing low-bit quantization on the re-identification network model to obtain a quantized re-identification network model includes: mapping the weights and / or activation functions of the re-identification network model from high-bit floating-point numbers to low-bit-depth data to obtain a quantized re-identification network model.

[0093] As will be understood by those skilled in the art, model quantization is a technique that converts floating-point calculations into low-bit-rate fixed-point calculations, which can effectively reduce the computational intensity, parameter size, and memory consumption of the model. In this application, the embodiments perform low-bit quantization on the re-identification network to reduce the time consumed by introducing the re-identification network to extract semantic features.

[0094] The re-identification network is mainly built on CLIP (Contrastive Language–Image Pre-training). CLIP uses 400 million image-text pairs as training samples. Each batch contains 32,768 image-text pairs, where (Ii, Ti) is the i-th image-text pair. The model aims to make these two features as similar as possible and as far away from other features as possible. Therefore, its semantic feature extraction at the target instance level can be used to extract features of each instance in multi-object tracking and to distinguish it from other instances.

[0095] In this embodiment of the application, the 2D detection box coverage area selected in the first step is resized to a 224x224 image; further, the CLIP pre-trained model is input to extract 512-dimensional instance semantic features of the detection box / tracking box coverage area.

[0096] It is worth noting that, to reduce the time consumption caused by introducing CLIP for semantic feature extraction, the weights and / or activation functions of the re-identification network model are mapped from high-bit floating-point numbers to low-bit-depth data. For example, post-training quantization (PTQ) is used to quantize the fp32 precision CLIP model to int8. Specifically, symmetric quantization is used for the weights, and asymmetric quantization is used for the activation values. The minimum MSE of the floating-point weights and quantized weights based on the per-layer model is used to set the weight range. 10% of the test set images are selected as the calibration set to calculate the minimum MSE of the floating-point activation values ​​and quantized activation values, which is then used to set the activation value range.

[0097] Therefore, it can be seen that introducing a re-identification network to extract semantic features increases computation time and reduces target tracking efficiency. Mapping the weights and / or activation functions of the re-identification network model from high-bit floating-point numbers to low-bit-depth data to obtain a quantized re-identification network model effectively solves the time consumption problem caused by introducing the re-identification network.

[0098] Please refer to Figure 7 , Figure 7 This is a second flowchart of a target tracking method provided in an embodiment of this application; the target tracking method processes lidar point cloud data and includes the following steps:

[0099] Step S600: Obtain lidar point cloud data.

[0100] Step S601: Obtain all target 3D detection boxes in the current frame from the lidar point cloud data through the lidar detection network.

[0101] Step S602: Obtain the 2D detection box based on the 3D detection box.

[0102] In step S602 above, the 3D detection box is projected onto the multi-camera coordinate system. When there is one 3D box corresponding to multiple 2D boxes, the 2D projection box with the smallest occlusion ratio is selected as the 2D detection box. The occlusion ratio is obtained by dividing the number of point clouds in the 3D detection box in the 3D coordinate system by the number of point clouds in the 2D detection box projected onto the camera coordinate system.

[0103] Step S603: Calculate the 3D IOU cost matrix based on the 3D detection box and the 3D tracking box.

[0104] Step S604: Input the 3D IOU cost matrix into the Hungarian matching algorithm for the first matching.

[0105] In steps S603-S604 above, for each predicted bounding box of a survival trajectory, the intersection-union ratio (IoU) is calculated pairwise with the detection bounding box of the current frame as the 3D IoU cost matrix. The IoU distance between the 3D detection bounding box and the 3D tracking bounding box is used as the cost matrix for the first matching, and the first matching result is obtained based on the Hungarian matching algorithm.

[0106] Step S605: Add the first matched detection box to the tracking trajectory, and perform a second matching on the unmatched detection boxes.

[0107] Step S606: Obtain the semantic feature cost matrix required for the second matching.

[0108] In step S606 above, for the 2D detection box coverage area selected in the first step, it is resized to 224x224 and then input into the CLIP pre-trained model to extract 512-dimensional instance semantic features of the detection box / tracking box coverage area. The cosine distance of the instance semantic features of N detection boxes and M tracking boxes is calculated as the instance semantic feature cost matrix.

[0109] Step S607: Input the semantic feature cost matrix into the Hungarian matching algorithm for a second matching.

[0110] Step S608: Obtain the matching results.

[0111] In step S608 above, an element of 1 in the matching matrix means that the match at that position is successful; if a detection box does not match any trajectory, a new trajectory is generated; if the detection box corresponding to a trajectory does not appear for multiple frames, the trajectory is terminated. The matching result of each frame is obtained, and the target information of each frame is stored in the corresponding trajectory.

[0112] Please refer to Figure 8 , Figure 8 This is a schematic diagram of the modules of a target tracking system provided in an embodiment of this application; the target tracking system 100 includes: a detection box acquisition module 110, a tracking box acquisition module 120, a first matching module 130, a low-bit quantization module 140, a second matching module 150, and a matching result recognition module 160.

[0113] The detection box acquisition module 110 is used to acquire the detection boxes in the current frame of the data to be processed.

[0114] The tracking box acquisition module 120 is used to acquire the tracking box in the current frame based on the lidar point cloud data.

[0115] The first matching module 130 is used to match the detection box with the tracking box and obtain the first matching result.

[0116] The low-bit quantization module 140 is used to determine whether the first matching result is a matching failure.

[0117] The second matching module 150 is used to perform low-bit quantization on the re-identification network model when the first matching result is a matching failure, so as to obtain the quantized re-identification network model.

[0118] The matching result recognition module 160 is used to input the detection box and tracking box into the quantized re-recognition network model to match the detection box and tracking box again and obtain a second matching result. The matching result recognition module 160 is also used to add the target trajectory of the detection box to the tracking trajectory when the second matching result is a successful match.

[0119] In an optional embodiment, the detection box includes a 3D detection box and a 2D detection box; the tracking box includes a 3D tracking box and a 2D tracking box. The target tracking system 100 further includes a data acquisition module 170. The detection box acquisition module 110 and the tracking box acquisition module 120 acquire the detection box and tracking box in the current frame based on the data to be processed, including: the data acquisition module 170 acquiring current and historical information of the target trajectory in the current frame based on the data to be processed; the detection box acquisition module 110 acquiring the 3D detection box based on the current information and projecting the 3D detection box onto the multi-camera coordinate system to obtain the 2D detection box; and the tracking box acquisition module 120 acquiring the 3D tracking box and the 2D tracking box based on the historical information and the 3D detection box.

[0120] In an optional embodiment, the data to be processed includes LiDAR point cloud data; the detection box acquisition module 110 projects a 3D detection box onto a multi-camera coordinate system to obtain a 2D detection box, including: the detection box acquisition module 110 projects the 3D detection box onto the multi-camera coordinate system to obtain multiple projected 2D boxes corresponding to the 3D detection box; obtains the number of point clouds in the 3D detection box and the number of point clouds in the multiple projected 2D boxes; divides the number of point clouds in the 3D detection box by the number of point clouds in the multiple projected 2D boxes to obtain the occlusion ratio; the detection box acquisition module 110 and the projected 2D box corresponding to the smallest occlusion ratio among the occlusion ratios are used as the 2D detection box.

[0121] In an optional embodiment, the second matching module 150 includes a semantic feature acquisition module 151 and a semantic feature cost matrix acquisition module 152. The second matching module 150 re-matches the detection boxes and tracking boxes using a quantized re-identification network model to obtain a second matching result. This includes: the semantic feature acquisition module 151 passing the 2D detection boxes and 2D tracking boxes through the quantized re-identification network model to obtain semantic features of 2D detection box instances and 2D tracking box instances; the semantic feature cost matrix acquisition module 152 calculating the cosine distance between the semantic features of the 2D detection box instances and the semantic features of the 2D tracking box instances to obtain a semantic feature cost matrix; and the second matching module 150 re-matching the 2D detection boxes and 2D tracking boxes based on the semantic feature cost matrix to obtain a second matching result.

[0122] In an optional embodiment, the first matching module 130 includes an intersection-union cost matrix acquisition module 131. Matching the detection box and the tracking box and obtaining a first matching result includes: the intersection-union cost matrix acquisition module 131 calculating the intersection-union distance between the 3D detection box and the 3D tracking box to obtain a 3D intersection-union cost matrix; based on the 3D intersection-union cost matrix, the first matching module 130 matches the 3D detection box and the 3D tracking box and obtains a first matching result.

[0123] In an optional embodiment, if the matching result recognition module 160 recognizes the first matching result as a successful match, the matching result recognition module 160 adds the target in the 3D detection box to the tracking trajectory.

[0124] In an optional embodiment, the low-bit quantization module 140 performs low-bit quantization on the re-identification network model to obtain the quantized re-identification network model, including: the low-bit quantization module 140 maps the weights and / or activation functions of the re-identification network model from high-bit floating-point numbers to low-bit depth data to obtain the quantized re-identification network model; wherein the weights are quantized using symmetric quantization and the activation functions are quantized using asymmetric quantization.

[0125] Please see Figure 9 , Figure 9 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. An electronic device 300 provided in this application includes: a processor 301 and a memory 302. The memory 302 stores machine-readable instructions executable by the processor 301. When the machine-readable instructions are executed by the processor 301, the method described above is performed.

[0126] Based on the same inventive concept, embodiments of this application also provide a computer-readable storage medium storing computer program instructions, which, when read and executed by a processor, perform the steps in any of the above implementations.

[0127] The computer-readable storage medium can be any medium capable of storing program code, such as Random Access Memory (RAM), Read Only Memory (ROM), Programmable Read-Only Memory (PROM), Erasable Programmable Read-Only Memory (EPROM), Electrically Erasable Programmable Read-Only Memory (EEPROM). The storage medium stores the program, and the processor executes the program after receiving an execution instruction. The method executed by the electronic terminal as defined in any embodiment of this invention can be applied to the processor or implemented by the processor.

[0128] In the embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. The apparatus embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. Furthermore, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Additionally, the displayed or discussed mutual couplings, direct couplings, or communication connections may be through some communication interfaces; indirect couplings or communication connections between devices or units may be electrical, mechanical, or other forms.

[0129] Furthermore, the units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0130] Furthermore, the functional modules in the various embodiments of this application can be integrated together to form an independent part, or each module can exist independently, or two or more modules can be integrated to form an independent part.

[0131] It can be replaced and can be implemented, wholly or partially, through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented, wholly or partially, in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of the present invention are generated.

[0132] The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions may be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means.

[0133] In this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, without necessarily requiring or implying any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising..." does not exclude the presence of additional identical elements in the process, method, article, or apparatus that includes said element.

[0134] The above description is merely an embodiment of this application and is not intended to limit the scope of protection of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of protection of this application.

Claims

1. A target tracking method characterized by, The target tracking method comprises: obtaining a detection box and a tracking box in a current frame of to-be-processed data; wherein the to-be-processed data comprises laser radar point cloud data; the detection box comprises a 3D detection box and a 2D detection box; the tracking box comprises a 3D tracking box and a 2D tracking box; matching the 3D detection box with the 3D tracking box, and obtaining a first matching result; determining whether the first matching result is a matching failure; if the first matching result is a matching failure, performing low-bit quantization on a re-identification network model to obtain a quantized re-identification network model; inputting the 2D detection box and the 2D tracking box into the quantized re-identification network model to match the 2D detection box and the 2D tracking box again, and obtaining a second matching result; if the second matching result is a matching success, adding a target trajectory of the detection box to a tracking trajectory; wherein the obtaining of the detection box and the tracking box in the current frame of to-be-processed data comprises: obtaining current information and historical information of the target trajectory in the current frame according to to-be-processed data; obtaining the 3D detection box according to the current information, and projecting the 3D detection box to a multi-camera coordinate system to obtain a plurality of projection 2D boxes corresponding to the 3D detection box; obtaining a point cloud quantity in the 3D detection box and a plurality of point cloud quantities in the plurality of projection 2D boxes; dividing the point cloud quantity in the 3D detection box by the plurality of point cloud quantities in the plurality of projection 2D boxes respectively to obtain an occlusion ratio; taking a projection 2D box corresponding to a smallest occlusion ratio in the occlusion ratio as the 2D detection box; obtaining the 3D tracking box and the 2D tracking box according to the historical information and the 3D detection box; wherein the inputting of the 2D detection box and the 2D tracking box through the quantized re-identification network model to match the 2D detection box and the 2D tracking box again and obtaining a second matching result comprises: inputting the 2D detection box and the 2D tracking box through the quantized re-identification network model to obtain 2D detection box instance semantic features and 2D tracking box instance semantic features; calculating a cosine distance of the 2D detection box instance semantic features and the 2D tracking box instance semantic features to obtain a semantic feature cost matrix; based on the semantic feature cost matrix, matching the 2D detection box and the 2D tracking box again and obtaining a second matching result.

2. The method of claim 1, wherein, The matching of the 3D detection box with the 3D tracking box and the obtaining of a first matching result comprises: calculating an intersection-over-union distance of the 3D detection box and the 3D tracking box to obtain a 3D intersection-over-union cost matrix; based on the 3D intersection-over-union cost matrix, matching the 3D detection box and the 3D tracking box and obtaining the first matching result.

3. The method of claim 1, wherein, The method further comprises: if the first matching result is a matching success, adding a target in the 3D detection box to a tracking trajectory.

4. The method of claim 1, wherein, The low-bit quantization of the re-identification network model to obtain a quantized re-identification network model comprises: The weights and / or activation functions of the re-identification network model are mapped from high-precision floating-point numbers to low-bit-depth data to obtain a quantized re-identification network model; wherein the weights are quantized in a symmetric manner, and the activation functions are quantized in an asymmetric manner.

5. A target tracking system characterized by, The target tracking system comprises a detection box acquisition module, a tracking box acquisition module, a first matching module, a low-bit quantization module, a second matching module, and a matching result identification module. The detection box acquisition module is configured to acquire a detection box in a current frame of to-be-processed data; wherein the to-be-processed data comprises laser radar point cloud data; the detection box comprises a 3D detection box and a 2D detection box; and the tracking box comprises a 3D tracking box and a 2D tracking box. The tracking box acquisition module is configured to acquire a tracking box in the current frame according to the laser radar point cloud data. The first matching module is configured to match the 3D detection box with the 3D tracking box and acquire a first matching result. The matching result identification module is configured to determine whether the first matching result is a matching failure. The low-bit quantization module is configured to, when the first matching result is a matching failure, quantize a re-identification network model in a low bit to obtain a quantized re-identification network model. The second matching module is configured to input the 2D detection box and the 2D tracking box into the quantized re-identification network model to match the detection box and the tracking box again and acquire a second matching result; wherein the inputting of the 2D detection box and the 2D tracking box into the quantized re-identification network model to match the detection box and the tracking box again and acquire a second matching result comprises: inputting the 2D detection box and the 2D tracking box through the quantized re-identification network model to obtain 2D detection box instance semantic features and 2D tracking box instance semantic features; calculating a cosine distance of the 2D detection box instance semantic features and the 2D tracking box instance semantic features to obtain a semantic feature cost matrix; and based on the semantic feature cost matrix, matching the 2D detection box and the 2D tracking box again and acquiring a second matching result. The matching result identification module is further configured to, when the second matching result is a matching success, add a target trajectory of the detection box to a tracking trajectory. The detection box acquisition module is specifically configured to acquire current information and historical information of the target trajectory in the current frame according to to-be-processed data; acquire the 3D detection box according to the current information and project the 3D detection box to a multi-camera coordinate system to obtain a plurality of projection 2D boxes corresponding to the 3D detection box; acquire a point cloud quantity in the 3D detection box and a plurality of point cloud quantities in the plurality of projection 2D boxes; divide the point cloud quantity in the 3D detection box by the plurality of point cloud quantities in the plurality of projection 2D boxes respectively to obtain an occlusion ratio; take a projection 2D box corresponding to a smallest occlusion ratio in the occlusion ratio as the 2D detection box; and acquire the 3D tracking box and the 2D tracking box according to the historical information and the 3D detection box.

6. An electronic device, comprising: The electronic device comprises a memory and a processor, the memory stores program instructions, and the processor executes the program instructions to perform the steps in the method of any one of claims 1-4.

7. A computer readable storage medium characterized by The computer readable storage medium stores computer program instructions, and the computer program instructions are executed by a processor to perform the steps in the method of any one of claims 1-4.

Images