System and method for 3D multi-object tracking in LiDAR point clouds
By introducing the average processing of mixed time position maps and temporary tracking position maps in the autonomous driving scenarios, combined with the backbone network prediction sports field, the problem of complex correlation steps of existing multi-object tracking methods is solved, and the accurate multi-dimensionality in the autonomous driving scenarios is achieved Object tracking and object ID inheritance improve universality and robustness.
Patent Information
- Application Number
- CN202210372567.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Priority Date
- 2021-08-06
- Filing Date
- 2022-04-11
- Publication Date
- 2024-11-05
- Estimated Expiration
- 2042-04-11
AI Technical Summary
The existing multi-object tracking method is difficult to apply in autonomous driving scenarios, mainly because of the complex correlation steps and the need for manual design of engineering rules and hyperparameters, which affects universality and performance.
By introducing the average processing of mixed time position maps and temporary tracking position maps, the tracking position map of the current point cloud fragment is generated, and the traditional correlation steps are avoided. The backbone network is used to predict the sports field and self-motion update object positions, realize multi-object tracking, and Track objects temporarily blocked without hyperparameter settings.
Accurate multi-object tracking in autonomous driving scenarios, inherit object ID, improve universality and robustness, and avoid performance degradation caused by errors in hyperparameter setting.
Smart Images

Figure CN114937058B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of object tracking technology, and in particular to a method, device, electronic device and computer-readable storage medium for 3D multi-object tracking. Background Art
[0002] In recent years, with the rapid development of autonomous vehicles, tracking based on LiDAR has attracted more and more attention in academia and industry. Among them, 3D multi-object tracking in LiDAR point cloud is a key technology in autonomous driving.
[0003] Most of the current multi-object tracking methods follow the tracking-by-detection pipeline, also known as tracking after detection; this method first detects objects in a single frame and then associates the detection set over time, that is, matching objects detected in different frames through the association step to achieve object tracking. Most existing work focuses on better defining the affinity matrix required for the association step, which makes the association step increasingly complex and difficult to apply to autonomous driving scenarios. Summary of the invention
[0004] In order to solve the technical problem of complex processing of existing object tracking solutions, embodiments of the present invention provide a method, device, electronic device and computer-readable storage medium for multi-object tracking.
[0005] In a first aspect, an embodiment of the present invention provides a method for tracking multiple objects, including:
[0006] Determine a mixed time position graph of the current point cloud segment, wherein the mixed time position graph is used to indicate the first appearance position of a valid object in the current point cloud segment, wherein the valid object is other objects in the current point cloud segment except for the invalid object;
[0007] Converting a tracking position map of a previous point cloud segment into a temporary tracking position map of the current point cloud segment according to the predicted motion field and the self-motion of the local device; the previous point cloud segment is a point cloud segment before the current point cloud segment, and the previous point cloud segment has the same partial point cloud data as the current point cloud segment, and the tracking position map includes position information and object identification of each object;
[0008] The mixed time position map of the current point cloud segment and the temporary tracking position map are averaged to generate the tracking position map of the current point cloud segment, and the tracking position map of the current point cloud segment inherits the object identifier in the tracking position map of the previous point cloud segment.
[0009] In a possible implementation, determining the mixed time position graph of the current point cloud segment includes:
[0010] Based on a preset backbone network, convert the current point cloud segment into a corresponding hybrid time position map; the hybrid time position map includes the confidence of the first appearance position of the valid object in the current point cloud segment;
[0011] The position information in the tracking position map indicates the confidence level of the object at the corresponding position.
[0012] In a possible implementation, before determining the mixed time position graph of the current point cloud segment, the method further includes:
[0013] Generating a plurality of sample point cloud segments according to a plurality of continuous point cloud data, wherein the sample point cloud segments have the same partial point cloud data as other adjacent sample point cloud segments;
[0014] Determine the first appearance position of each valid object in the sample point cloud segment, and generate a sample mixed time position map including the first appearance position of each valid object in the sample point cloud segment;
[0015] The backbone network is obtained by training according to the sample point cloud fragments and the sample mixed time position graph.
[0016] In a possible implementation, the method further includes:
[0017] The motion field is determined based on the backbone network, where the motion field is used to represent motion parameters of an object in the current point cloud segment.
[0018] In a possible implementation, the method further includes:
[0019] When there is a new object in the tracking position map of the current point cloud segment, a corresponding object identifier is added to the new object.
[0020] In a possible implementation, converting the tracking position map of the previous point cloud segment into the temporary tracking position map of the current point cloud segment according to the predicted motion field and the self-motion of the local device includes:
[0021] updating the position information of each object in the tracking position map of the previous point cloud segment according to the motion field of the previous point cloud segment to generate a first tracking position map; updating the position information of each object in the first tracking position map according to the self-motion of the current point cloud segment to generate the temporary tracking position map; or
[0022] According to one of the motion field and the self-motion of the current point cloud fragment, the position information of each object in the tracking position map of the previous point cloud fragment is updated to generate a second tracking position map; according to the other of the motion field and the self-motion of the current point cloud fragment, the position information of each object in the second tracking position map is updated to generate the temporary tracking position map.
[0023] In a possible implementation, the method further includes:
[0024] Implementing multi-object tracking based on the tracking position graph of the current point cloud segment; or
[0025] Multi-object tracking is implemented based on the first tracking position map or the second tracking position map.
[0026] In a possible implementation, the averaging the mixed time position map of the current point cloud segment and the temporary tracking position map to generate the tracking position map of the current point cloud segment includes:
[0027] The mixed time position map of the current point cloud segment and the temporary tracking position map are averaged, and a non-maximum suppression operation is performed on the result of the average processing to generate the tracking position map of the current point cloud segment.
[0028] In a possible implementation, the first appearance position of the valid object is the first appearance position of the center of the valid object;
[0029] The position information is the position information of the center of the corresponding object.
[0030] In a possible implementation manner, the point cloud data of the last frame of the previous point cloud segment is the same as the point cloud data of the first frame of the current point cloud segment.
[0031] In a second aspect, an embodiment of the present invention further provides a multi-object tracking device, comprising:
[0032] A mixed time position graph determination module, used to determine a mixed time position graph of a current point cloud segment, wherein the mixed time position graph is used to indicate the first appearance position of a valid object in the current point cloud segment, wherein the valid object is other objects in the current point cloud segment except for an invalid object;
[0033] a motion conversion module, configured to convert a tracking position map of a previous point cloud segment into a temporary tracking position map of the current point cloud segment according to the predicted motion field and the self-motion of the local device; the previous point cloud segment is a point cloud segment before the current point cloud segment, and the previous point cloud segment has the same partial point cloud data as the current point cloud segment, and the tracking position map includes position information and object identification of each object;
[0034] A tracking position map determination module is used to average the mixed time position map of the current point cloud segment and the temporary tracking position map to generate a tracking position map of the current point cloud segment, and the tracking position map of the current point cloud segment inherits the object identifier in the tracking position map of the previous point cloud segment.
[0035] In a third aspect, an embodiment of the present invention provides an electronic device, comprising a bus, a transceiver, a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the transceiver, the memory, and the processor are connected via the bus, and the computer program, when executed by the processor, implements the steps in any one of the above-described methods for multi-object tracking.
[0036] In a fourth aspect, an embodiment of the present invention further provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps in any one of the above-mentioned methods for tracking multiple objects.
[0037] The method, device, electronic device and computer-readable storage medium for multi-object tracking provided by the embodiments of the present invention can not only accurately generate the tracking position map of the current point cloud segment by averaging the mixed time position map and the temporary tracking position map of the current point cloud segment, but also inherit the object ID. Based on the object ID, the same object in different point cloud segments can be associated, and multi-object tracking can be achieved without the association step in the traditional solution. The mixed time position map only indicates the first appearance position of the valid object, and can normally track temporarily occluded objects without the need to set additional hyperparameters, and has stronger versatility. BRIEF DESCRIPTION OF THE DRAWINGS
[0038] In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the background technology, the drawings required for use in the embodiments of the present invention or the background technology will be described below.
[0039] Figure 1 A flowchart of a method for tracking multiple objects provided by an embodiment of the present invention is shown;
[0040] Figure 2 A schematic diagram showing an application scenario of the method for multi-object tracking provided by an embodiment of the present invention is shown;
[0041] Figure 3a A schematic diagram showing generation of a mixed time position graph in a method for tracking multiple objects provided by an embodiment of the present invention is shown;
[0042] Figure 3b An overall flow chart of a method for tracking multiple objects provided by an embodiment of the present invention is shown;
[0043] Figure 4 Another overall flow chart of a method for tracking multiple objects provided by an embodiment of the present invention
[0044] Figure 5 A schematic diagram showing the structure of a multi-object tracking device provided by an embodiment of the present invention;
[0045] Figure 6 A schematic structural diagram of an electronic device for executing a method for multi-object tracking provided by an embodiment of the present invention is shown. DETAILED DESCRIPTION
[0046] Current object tracking schemes inevitably require a heuristic matching step for association detection, i.e., an association step, and there are usually hand-crafted engineering rules in the association step; for example, tracking based on the Kalman filter is sensitive to the choice of the covariance matrix, and the distance metrics and maximum distance threshold allowed for each specific object category will also greatly affect the final performance. In addition, the objects involved in object tracking mainly include: tracked objects, new born objects, and dead objects; current object tracking schemes require hyper-parameters to manage new born objects and dead objects, such as the maximum number of frames to retain dead objects and the minimum number of frames before initializing new born objects. Such rules and hyper-parameters are usually related to data sets and models, making them less versatile when applied to new scenarios.
[0047] The embodiment of the present invention provides a totally heuristic-free method for tracking multiple objects, that is, multiple object tracking can be achieved without requiring an association step. Figure 1 FIG. 1 is a flowchart of a method for tracking multiple objects provided by an embodiment of the present invention, wherein the method can track multiple objects around a local device. Figure 1 As shown, the method includes steps 101-103.
[0048] Step 101: Determine a mixed time position map of the current point cloud segment, where the mixed time position map is used to indicate the first appearance position of a valid object in the current point cloud segment, where the valid object is other objects in the current point cloud segment except for an invalid object.
[0049] In the embodiment of the present invention, the local device refers to a device that needs to track other surrounding objects, such as Figure 2 As shown, the local device 10 can be, for example, a vehicle in an autonomous driving scenario, or a processing device in the vehicle, etc.; in general, the local device 10 can also be the subject that executes the multi-object tracking method. The local device 10 can be provided with a laser radar 20 that can collect surrounding point cloud data. Based on the fact that the laser radar 20 can collect multiple frames of point cloud data, it is possible to select parts of multiple continuous point cloud data in turn to form corresponding point cloud fragments, that is, the point cloud fragment contains multiple continuous point cloud data within a certain time period; for example, the point cloud fragment contains point cloud data of the t-1th frame and the tth frame, or the point cloud fragment contains point cloud data of the t-1th frame, the tth frame and the t+1th frame. The current point cloud fragment refers to the point cloud fragment that currently needs to be processed. Among them, Figure 2 The laser radar 20 is a schematic structure and is not used to limit the shape of the laser radar 20; the laser radar 20 can collect point cloud data of other objects around the local device 10, such as collecting point cloud data of other vehicles 31, pedestrians 32, buildings 33 and other objects.
[0050] Furthermore, the concept of a hybrid time-position map is introduced in an embodiment of the present invention, and a hybrid time-position map of a point cloud segment can be generated based on at least two frames or all of the point cloud data in the point cloud segment. Specifically, the hybrid time-position map is used to represent the first appearance position of each valid object in a certain point cloud segment, wherein the valid object refers to other objects in the point cloud segment except for the invalid object, that is, the valid objects include tracked objects and new objects. Among them, the tracked object refers to an object that always exists in the corresponding point cloud segment, the new object refers to an object that does not initially exist in the corresponding point cloud segment but exists at the end, and the invalid object refers to an object that initially exists in the corresponding point cloud segment but does not exist at the end. Accordingly, the hybrid time-position map of the current point cloud segment can represent the first appearance position of the valid object in the current point cloud segment. For example, the current point cloud fragment contains point cloud data of the t-1th frame, the tth frame and the t+1th frame; among them, if object A always exists, then object A is the tracked object, and the first appearance position of object A is its position in the t-1th frame; if object B only exists in the t-1th frame and the tth frame, that is, the object B disappears in the t+1th frame, then object B is an invalid object, so the mixed time position diagram does not need to pay attention to object B; if object C only exists in the t+1th frame, it means that object C is most likely newly generated in the t+1th frame, that is, object C does not exist before the t+1th frame, and the object C is a new object, and the first appearance position of object C is its position in the t+1th frame.
[0051] Optionally, the center of the object can be used to represent the position of the object, that is, the first appearance position of a valid object is the first appearance position of the center of the valid object. At this time, the hybrid time position map can represent the first appearance position of the center of the valid object in the current point cloud segment. The hybrid time position map can also be called a hybrid time centerness map (Hybrid-Time Centerness Map).
[0052] Step 102: Convert the tracking position map of the previous point cloud fragment into a temporary tracking position map of the current point cloud fragment according to the predicted motion field and the self-motion of the local device; the previous point cloud fragment is a point cloud fragment before the current point cloud fragment, and the previous point cloud fragment has the same partial point cloud data as the current point cloud fragment, and the tracking position map contains the position information and object identification of each object.
[0053] In an embodiment of the present invention, there are other point cloud fragments before the current point cloud fragment, that is, the previous point cloud fragment; and the previous point cloud fragment has the same part of point cloud data as the current point cloud fragment, that is, the previous point cloud fragment and the current point cloud fragment have the same frame of point cloud data, and the two partially overlap in time. For example, the current point cloud fragment contains point cloud data of the t-1th frame, the tth frame and the t+1th frame, and the previous point cloud fragment may contain point cloud data of the t-2th frame, the t-1th frame and the tth frame, and the two have the same point cloud data of the t-1th frame and the tth frame. In general, the number of frames contained in each point cloud fragment is the same, for example, they all contain 3 frames of point cloud data. In addition, in order to make full use of all point cloud data and ensure good timeliness, only the last frame of point cloud data in the current point cloud fragment is not included in the previous point cloud fragment. For example, every two frames can form a point cloud segment, that is, the point cloud data of the t-1th frame and the tth frame are one point cloud segment, and the point cloud data of the tth frame and the t+1th frame are another point cloud segment... In this way, the collected point cloud data can be processed in a timely manner. Optionally, the previous point cloud segment and the current point cloud segment have and only have one frame of point cloud data in common, that is, the point cloud data of the last frame of the previous point cloud segment is the same as the point cloud data of the first frame of the current point cloud segment. Generally, a point cloud segment contains two frames of point cloud data, and the first frame of the current point cloud segment is the same as the last frame of the previous point cloud segment.
[0054] The embodiments of the present invention can determine the tracking location map corresponding to each point cloud segment, so that multi-object tracking can be achieved based on the tracking location map. Specifically, the tracking location map contains the location information of each object around the local device, and the location information indicates the location of the object, or the location information indicates the confidence of the object at a certain location, wherein if it can be determined that an object is at a unique location, the confidence of the object at that location is 1. In addition, each object in the tracking location map is provided with a corresponding identification (identity, ID), i.e., an object identification or object ID, to facilitate the distinction between different objects and to facilitate subsequent tracking based on the object ID. Optionally, the tracking location map can be displayed in a graphical user interface (such as a graphical user interface). Figure 6 The user interface 1160 in is used for display) so as to be output to the user of the user interface.
[0055] Among them, when determining the tracking position map of the current point cloud segment, the tracking position map of the previous point cloud segment needs to be used. Specifically, the ego-motion of the local device can be determined based on the motion parameters of the local device itself. For example, when the local device is a vehicle, the speed, direction, etc. of the vehicle can be determined based on the sensors on the vehicle; when the local device is stationary, its ego-motion is zero. In addition, the embodiment of the present invention also predicts the motion parameters of each object to form a motion field, that is, the motion field can represent the motion parameters of other objects around the local device, such as the moving speed and moving direction of the object. As described above, the tracking position map of the previous point cloud segment can represent the position information of each object in the previous point cloud segment. Therefore, the position information of each object can be updated in combination with the predicted motion field and the ego-motion of the local device, so as to obtain the possible position information of each object in the current point cloud segment, and then a corresponding tracking position map can be formed. The embodiment of the present invention refers to the tracking position map obtained at this time as a temporary tracking position map.
[0056] Optionally, in the case where the center of an object is used to represent the location of the object, the above-mentioned tracking position map can also be called a tracked centralness map, and the position information in the tracking position map represents the position information of the center of the corresponding object. In addition, the initial tracking position map can be obtained from the initial mixed time centrality map. For example, the initial mixed time centrality map can be directly used as the initial tracking position map, or the initial mixed time centrality map can be subjected to non-maximum suppression (NMS) processing to generate the initial tracking position map.
[0057] Step 103: average the mixed time position map and the temporary tracking position map of the current point cloud segment to generate the tracking position map of the current point cloud segment, and the tracking position map of the current point cloud segment inherits the object identification in the tracking position map of the previous point cloud segment.
[0058] In the embodiment of the present invention, the tracking position map of the previous point cloud fragment contains an object identifier, so that the temporary tracking position map converted therefrom also contains the corresponding object identifier; the averaging processing is performed in the above step 103, and the object identifier in the tracking position map of the current point cloud fragment can be determined based on the object identifier in the temporary tracking position map, so that the tracking position map can inherit the object identifier, and the same object has the same object identifier in different tracking position maps, so that the object can be tracked.
[0059] The temporary tracking position map is a tracking position map obtained based on the motion parameters of the local device and the object, and the temporary tracking position map can represent the position information of each object in the current point cloud segment; the mixed time position map of the current point cloud segment can represent the first appearance position of each valid object in the current point cloud segment, so the mixed time position map can also represent the position information of each valid object; and the mixed time position map represents the first appearance position of the valid object. For the tracked object, its first appearance position is the position of the first frame of the current point cloud segment, and the previous point cloud segment and the current point cloud segment partially overlap in time, that is, the previous point cloud segment also contains the first frame of the current point cloud segment. The temporary tracking position map obtained after updating the tracking position map of the previous point cloud segment can also represent the position of the object in the first frame of the current point cloud segment to a certain extent. Therefore, the mixed time position map and the temporary tracking position map of the current point cloud segment can represent the same meaning. In an ideal case, the tracked object has exactly the same position information in the two. Therefore, by averaging the two, the position information of each object can be determined more accurately, thereby generating the tracking position map of the current point cloud segment.
[0060] Moreover, although the mixed time position map does not contain the object ID, the temporary tracking position map is obtained by updating the tracking position map of the previous point cloud fragment. Therefore, the temporary tracking position map contains the object ID of each object. The tracking position map of the current point cloud fragment generated after averaging processing can inherit the object ID in the temporary tracking position map, that is, it can inherit the object ID in the tracking position map of the previous point cloud fragment, so that the same object has the same object ID in different tracking position maps, so that object tracking can be conveniently achieved based on the object ID.
[0061] In addition, the failed object is removed from the mixed time position map. Through the above-mentioned averaging process, the failed object can be weakened in the tracking position map of the current point cloud fragment. In the subsequent rounds of processing, the failed object can be weakened multiple times until it is determined that the failed object has completely disappeared. If the failed object disappears temporarily due to occlusion, and then the laser radar 20 collects the point cloud of the failed object again, the failed object can be enhanced through averaging processing at this time, so that the temporarily disappeared object can be tracked normally. In order to track temporarily disappeared objects, traditional object tracking schemes need to manually set hyperparameters such as the maximum number of frames to retain failed objects. The scheme provided by the embodiment of the present invention does not need to set similar hyperparameters, does not need to debug hyperparameters, has strong versatility, and can also avoid performance degradation due to incorrect hyperparameter settings.
[0062] Optionally, after averaging the mixed time position map and temporary tracking position map of the current point cloud fragment, a non-maximum suppression operation is performed on the averaged result to generate a tracking position map of the current point cloud fragment to highlight the position of each object in the tracking position map.
[0063] Those skilled in the art can understand that the above-mentioned averaging process can adopt simple arithmetic averaging process, or can adopt other methods such as weighted averaging process, which is not limited in this embodiment.
[0064] Optionally, when a new object exists in the tracking position map of the current point cloud fragment, a corresponding object identifier is added to the new object. By adding an object identifier to the new object, it is convenient to track the new object based on the object identifier later. It should be noted that if a new object A exists in the mixed time position map of the current point cloud fragment, and the temporary tracking position map does not contain the object A, then the object A is a new object in the tracking position map of the current point cloud fragment; if a new object A exists in the mixed time position map of the current point cloud fragment, and the temporary tracking position map also contains the object A, it means that the object A is a temporarily disappeared object, and the object A is not a new object in the tracking position map of the current point cloud fragment, that is, at this time, the object ID of object A in the temporary tracking position map can be inherited, and there is no need to generate a new object ID.
[0065] A method for multi-object tracking provided by an embodiment of the present invention introduces a mixed time position map and a tracking position map. By averaging the mixed time position map and the temporary tracking position map of the current point cloud segment, not only can the tracking position map of the current point cloud segment be accurately generated, but also the inherited object ID can be realized. Based on the object ID, the same object in different point cloud segments can be associated, and multi-object tracking can be realized without the association step in the traditional solution. The mixed time position map only indicates the first appearance position of the valid object, and can normally track temporarily occluded objects without the need to set additional hyperparameters, and has stronger versatility.
[0066] On the basis of the above-mentioned embodiment, the embodiment of the present invention can determine the mixed time position map of each point cloud segment based on a preset backbone network. Specifically, the process of the above-mentioned step 101 "determining the mixed time position map of the current point cloud segment" includes: based on the preset backbone network, converting the current point cloud segment into a corresponding mixed time position map. For example, the current point cloud segment is input into the backbone network, and the mixed time position map of the current point cloud segment can be output; the mixed time position map contains the confidence of the first appearance position of the valid object in the current point cloud segment, that is, the mixed time position map represents the first appearance position of the valid object in the form of confidence; accordingly, the position information in the tracking position map represents the confidence of the object at the corresponding position. In the embodiment of the present invention, the mixed time position map and the mixed position map can both be a Gaussian heatmap, in which the numerical value represents the corresponding confidence level. Generally, in the mixed time position map, the tracked objects and the new objects have a larger confidence, and the confidence of the failed objects is smaller; this embodiment can determine which objects are failed objects by the size of the confidence, and the confidence of the failed objects can be set to zero.
[0067] Optionally, when training the backbone network, a corresponding training set needs to be set in advance, and a mixed time position map of sample point cloud segments in the training set, ie, a sample mixed time position map, needs to be determined. The process of training the backbone network may include steps A1-A3.
[0068] Step A1: generating a plurality of sample point cloud segments according to a plurality of continuous point cloud data, wherein a sample point cloud segment has part of the same point cloud data as other adjacent sample point cloud segments.
[0069] Step A2: Determine the first appearance position of each valid object in the sample point cloud segment, and generate a sample mixed time position map containing the first appearance position of each valid object in the sample point cloud segment.
[0070] Step A3: A backbone network is obtained by training the sample point cloud fragments and the sample mixed time position map.
[0071] In the embodiment of the present invention, the sample point cloud segment is also a point cloud segment, and each point cloud data of the sample point cloud segment is marked with the position of each object, such as the center position, etc. Therefore, based on the point cloud data in the sample point cloud segment, a corresponding sample mixed time position map can be generated, and the sample mixed time position map contains the accurate first appearance position of each valid object. In other words, the confidence of the first appearance position of each valid object is 1.
[0072] For example, each point cloud segment contains two frames of point cloud data. Assume that the sample point cloud segment includes the point cloud data of the t-1th frame and the tth frame, and in the point cloud data of the t-1th frame and the tth frame, the ground-truth objects are If the object in the t-1th frame with the object in frame t are the same object A, then object A is the tracked object, and in the sample mixed time position map, the first appearance position of object A is its position in the t-1 frame. If object B does not exist in the t-1 frame, but exists in the t frame, it means that object B is a new object, and in the sample mixed time position map, the first appearance position of object B is its position in the t frame. If object C only appears in the t-1 frame but disappears in the t frame, object C is an invalid object, and object C does not exist in the sample mixed time position map. The above allocation strategy is executed for all true value objects in the sample point cloud fragment, and the sample mixed time position map of the sample point cloud fragment can be obtained.
[0073] Optionally, the backbone network can also be used to predict the motion field, for example, the backbone network includes multiple branches, one of which is used to determine the mixed time position map, and another branch is used to predict the motion field. For example, the current point cloud segment is input into the backbone network, and the motion field that can represent the motion parameters of the object in the current point cloud segment can be obtained based on the output results of the corresponding branches.
[0074] In addition, the backbone network may also include a regression branch for determining the height, bounding box size, and heading angle of each object. During the training process, the focal loss may be used to represent the loss function for determining the hybrid time center graph branch, the standard loss may be used to represent the loss functions of other branches, and the total loss of the backbone network may be obtained in a weighted manner; of course, other forms of loss functions may also be used, which are not limited in this embodiment.
[0075] In addition, optionally, when converting to obtain a temporary tracking position map, the predicted motion field and the self-motion of the local device may be processed separately, and the desired temporary tracking position map may be obtained after two processes. Specifically, the process of the above step 102 "converting the tracking position map of the previous point cloud segment into the temporary tracking position map of the current point cloud segment according to the predicted motion field and the self-motion of the local device" includes:
[0076] Step B1: updating the position information of each object in the tracking position map of the previous point cloud segment according to the motion field of the previous point cloud segment to generate a first tracking position map.
[0077] Step B2: updating the position information of each object in the first tracking position map according to the self-motion of the current point cloud segment to generate a temporary tracking position map.
[0078] Alternatively, the above process may also include:
[0079] Step C1: updating the position information of each object in the tracking position map of the previous point cloud segment according to one of the motion field and the self-motion of the current point cloud segment to generate a second tracking position map.
[0080] Step C2: updating the position information of each object in the second tracking position map according to the other of the motion field and the self-motion of the current point cloud segment to generate a temporary tracking position map.
[0081] In the embodiment of the present invention, a round of processing is performed on each point cloud segment, that is, determining the mixed time position map, motion field, self-motion of the local device, etc. of the point cloud segment, and then determining the temporary tracking position map of the point cloud segment, etc. Wherein, as shown in the above steps B1-B2, the temporary tracking position map can be generated based on the motion field of the previous point cloud segment and the self-motion of the current point cloud segment; or, as shown in the above steps C1-C2, the tracking position map of the previous point cloud segment can also be converted into a temporary tracking position map based on the motion field and self-motion of the current point cloud segment.
[0082] For example, the tracking position map of the previous point cloud segment is T t-1 , the sports field is M t-1 , the motion field of the current point cloud fragment is M t , the local device's ego motion is In the embodiment of the present invention, the sports field M can be used as the basis for t-1 The tracking location map t-1 Updated to the first tracking position map Τ' t-1 , and then based on the self-motion of the current point cloud segment The first tracking position map T' t-1 The temporary tracking position map converted to the current point cloud fragment is Τ" t Alternatively, it can also be based on the motion field M of the current point cloud segment t or self-movement The tracking location map t-1 Convert to the second tracking position map Τ' t , and then the temporary tracking position map of the current point cloud fragment is obtained as Τ" t .
[0083] In addition, the tracking position map in the embodiment of the present invention can represent the position information of each object, so other objects around the local device can be tracked based on the tracking position map to achieve multi-object tracking. Alternatively, the sports field or the self-motion of the local device can be introduced for tracking, for example, multi-object tracking can be achieved based on the first tracking position map or the second tracking position map. In addition, optionally, the first tracking position map or the second tracking position map can also be displayed in the user interface (such as Figure 6 The user interface 1160 in is used for display) so as to be output to the user of the user interface.
[0084] A process of the multi-object tracking method is described in detail below through an embodiment.
[0085] In the embodiment of the present invention, the point cloud data is voxelized into a regular grid using a pillar, and the features of each pillar are extracted, so that the point cloud data of each frame can be represented in the form of a bird's eye view (BEV). For details, see Figure 3a In addition, each point cloud segment contains point cloud data of two adjacent frames. Figure 3a Four frames of point cloud data are shown, namely, the point cloud data P of the t-2th frame t-2 , point cloud data P of frame t-1 t-1 , point cloud data P of the tth frame t , point cloud data P of frame t+1 t+1 , the four-frame point cloud data can be composed of three point cloud fragments, namely {P t-2 ,P t-1}、{P t-1 ,P t}、{P t ,P t+1}, based on the pre-trained backbone network, the mixed time position map D of each point cloud segment can be obtained t-1 , D t , D t+1 .
[0086] For ease of description, Figure 3a The point cloud data in only involves a small number of objects, including objects A, B, C, and D. The local device is represented by the rectangular box in the middle of the point cloud data. The local device moves forward at a constant speed, that is, the self-motion of the local device remains unchanged; objects A and B are always present and tracked objects, and object A is a stationary object, object B also moves forward, and the speed of object B is greater than the speed of the local device. Objects C and D are both stationary objects, and object C is an object that disappears in the middle, and object D is a new object. For the convenience of comparison, Figure 3a In some point cloud data, the dotted boxes represent the position of an object in the previous frame. The objects represented by these dotted boxes do not exist in the real point cloud data. For example, the point cloud data Pt In addition to the solid line representing the position of object A, different dotted boxes are used to represent the position of object A at the t-1th frame and the t-2th frame respectively.
[0087] For point cloud fragments {P t-2 ,P t-1}, where objects A, B, and C are all tracked objects, and the corresponding mixed time position graph D t-1 The first appearance position of each object A, B, C can be represented in {P t-2, that is, the position of object A, B, C at the t-2th frame. It should be noted that the mixed time position map in the actual scene is a heat map. For the convenience of description, the embodiment of the present invention accurately describes the first appearance position of each object in the mixed time position map in the form of a point. Similarly, for the point cloud segment {P t-1 ,P t}, there are failed objects C and new objects D, and the corresponding mixed time position graph D t contains objects A, B, and D, and the first appearance position of the tracked objects A and B is their position in the t-1 frame, and the first appearance position of the new object D is its position in the t frame. t ,P t+1}, where objects A, B, and D are all tracked objects, so objects A, B, and D are in the mixed time position graph D t+1 The first appearance position of is its position in the tth frame.
[0088] In addition, this embodiment adopts the above steps B1-B2 to obtain a temporary tracking position map, that is, to obtain a tracking position map T of each point cloud segment. t-1 After that, based on the current sports field M t-1 Update to obtain the first tracking position map Τ' t-1 , in the next point cloud segment, based on the self-motion of the next point cloud segment, The first tracking position map T' t-1 Converted to temporary tracking position map Τ" t , and the cycle continues.
[0089] Figure 3b Shown based on Figure 3a Specifically, for a point cloud segment {P t-2 ,P t-1}, based on the mixed time position graph D t-1 The tracking position map T of the point cloud fragment can be determined t-1 , and then obtain the corresponding temporary tracking position map T" t . Since it is necessary to combine the previous point cloud fragments {P t-3 ,Pt-2}Get the tracking position map T t-1 , to simplify the description, Figure 3b The temporary tracking location map T is not provided. t The detailed process is not shown, but the temporary tracking position diagram T" is directly shown. t In the case where all objects in the point cloud segment are tracked objects, it can be simply considered that the temporary tracking position map T" is obtained by updating the motion field and self-motion t The point cloud data P of the last frame in the point cloud segment t-1 They are basically the same, so here we directly use objects A, B, and C in the point cloud data P t-1 The position in the temporary tracking position map T' represents each object t The position in.
[0090] In determining the point cloud fragment {P t-1 ,P t Mixed time position diagram D t After that, the mixed time position diagram D t and temporary tracking position map T" t The point cloud fragment {P t-1 ,P t}'s tracking position map T t For example, the averaging process may be an arithmetic averaging process, that is, T t =(D t +T” t ) / 2. Due to the mixed time position diagram D t The tracked objects A and B still have a high confidence after averaging (theoretically, the tracked objects A and B are in the mixed time position map D). t and temporary tracking position map T" t The same position in the image), for example, its confidence is 1; while the confidence of the failed object C and the new object is not very high, for example, the confidence of both is 0.5; Figure 3b The confidence level of the object is represented by different grayscale values.
[0091] Furthermore, the motion field of each point cloud segment can be determined based on the backbone network. For example, the point cloud segment {P t-1 ,P t Sports field M t , the sports field M t Represents the motion parameters of each valid object in the t-1 frame and the t frame. At this time, objects A and D are stationary, and object B moves forward at a certain speed. t The tracking position map T tUpdate to the first tracking position map T' t , such as Figure 3b shown.
[0092] For point cloud fragments {P t ,P t+1}, based on the backbone network, the hybrid time position graph D can be determined t+1 and the corresponding self-motion Based on this self-movement Update the first tracking position map T' t , the corresponding temporary tracking position map T" can be determined t+1 In the embodiment of the present invention, updating based on self-motion refers to converting the first tracking position map to the current coordinate system of the current point cloud fragment, thereby generating a temporary tracking position map in the current coordinate system. Since the local device in this embodiment moves forward, the surrounding objects A, B, C, and D move in opposite directions, and the coordinate system conversion can be achieved. Among them, object C is an invalid object, and at this time, object C can be updated in the temporary tracking position map T" t+1 The position in the first tracking position map T' can also be maintained t This embodiment does not limit this.
[0093] After that, the mixed time position diagram D t+1 and temporary tracking position map T" t+1 Perform summation to obtain the corresponding tracking position map T t+1 In this tracking position diagram T t+1 In the process, the confidence of the failed object C is further reduced, for example, to 0.25. After multiple iterations, the confidence of the failed object C tends to zero, which can indicate that the object C is a failed object. If the object C is temporarily occluded, in the subsequent iterations, the confidence of the object C in the tracking position map will increase because the object C reappears in the mixed time position map, so that the object C can continue to be tracked, which can reduce the impact of temporary occlusion.
[0094] For a new object D, track the location map T t+1 It can express the confidence of the object D at a certain position, such as Figure 3b As shown, the confidence of the object D in the upper position is 0.5, and the confidence of the object D in the lower position is 0.25, that is, in the tracking position map T t+1 In the point cloud data P, object D is more likely to be in the upper position, which is consistent with the position of object D in the point cloud data P. t The positions in the tracking position map are basically the same. Among them, the non-maximum suppression process can be performed on the averaged result, which can more accurately represent the confidence of the position of each object in the tracking position map.
[0095] In summary, if Figure 3b As shown, the multi-object tracking method provided by the embodiment of the present invention includes steps ① to ⑦.
[0096] Step ①: The current point cloud fragment is {P t-1 ,P t}, the previous point cloud fragment is {P t-2 ,P t-1}, at which point the local device's self-motion can be determined According to the backbone network, the mixed time position map D of the current point cloud segment can be determined t and sports field M t .
[0097] Step ②: Generate a temporary tracking position map T" t .
[0098] Among them, the self-motion of the current point cloud fragment can be and the motion field M of the previous point cloud fragment t-1 , the tracking position map T of the previous point cloud fragment t-1 Convert to this temporary tracking position map T" t , Figure 3b The tracking position map T is not shown in FIG. t-1 .
[0099] Step 3: Mixing time position diagram D t and temporary tracking position map T" t Perform averaging to obtain the tracking position map T of the current point cloud segment t .
[0100] Step ④: Use the motion field M of the current point cloud fragment t Update the tracking location map T t , get the first tracking position map T' t .
[0101] The above steps ① to ④ are one round of iterative processing. At this time, based on the tracking position map T t Or the first tracking position map T' t To achieve multi-object tracking. Then, the point cloud fragment {P t ,P t+1} is taken as the current point cloud segment. The previous point cloud segment is {P t-1 ,P t}, and then execute a new round of iterative processing. For details, please refer to steps ⑤ to ⑦ below.
[0102] Step ⑤: The current point cloud fragment is {P t ,P t+1}, determine the local device's self-motion at this time And according to the backbone network, the mixed time position map D of the current point cloud fragment can be determined t+1 and sports field M t+1 .
[0103] Step 6: Based on self-movement Update the first tracking position map T' obtained in the previous round t , and then a temporary tracking position map T" can be generated t+1 .
[0104] Step 7: Mixing time position diagram D t+1 and temporary tracking position map T" t +1 performs averaging processing to obtain the tracking position map T of the current point cloud segment t+1 .
[0105] It should be noted that the mixed time position map does not contain the object identification of each object, and only the tracking position map (including the temporary tracking position map, the first tracking position map, the second tracking position map, etc.) contains the object identification of each object. t When creating a new object D, you need to add a corresponding object identifier to the new object D; while the remaining objects A, B, and C can use the previous object identifiers.
[0106] The above schematically explains the process of the multi-object tracking method. The processing effect of this method on actual point cloud data can be seen in Figure 4 shown.
[0107] like Figure 4 As shown, the backbone network used in the embodiment of the present invention includes three branches, namely, a position branch, a motion branch and a regression branch. The position branch is used to generate a mixed time position map, the motion branch is used to predict information related to motion parameters such as the motion field, and the regression branch is used to determine the height, bounding box size, orientation, etc. of each object. t-1 and P t The point cloud fragments formed can be used to determine the mixed time position graph D by the position branch t ; Update the first tracking position map T' of the previous point cloud segment based on the self-motion t-1 , we can get the temporary tracking position map T" t , and then the tracking position map T of the current point cloud fragment can be obtained by averaging the two. t ; At the same time, you can read out (Read-Off) the temporary tracking position map T" t The object ID in the tracking position map T can be updated based on the motion field determined by the motion branch. t, thereby obtaining the first tracking position map T' of the current point cloud segment t ; and, based on the regression branch, the motion offset of all objects within the time period of the current point cloud segment is regressed, so that tracking can be achieved based on the first tracking position map to obtain the result of multi-object tracking.
[0108] Currently, the widely used multi-object tracking methods include 3D Kalman filter and center point (CenterPoint). The embodiment of the present invention uses open data sets, such as nuScenes and Waymo, to compare the method provided by the embodiment of the present invention with the above two existing methods. During the comparison process, two backbone networks are used to extract body pillars (Pillar) and voxels (Voxel) for verification respectively; and the above two traditional methods both involve association steps, and their tracking performance is highly dependent on the detection results. Therefore, the results of body pillar / voxel-detection (Pillar / Voxel-Det) and the results of body pillar / voxel-tracking (Pillar / Voxel-Track) combined with detection and tracking are compared here. Among them, the average multiple object tracking accuracy (Average Multiple Object Tracking Accuracy, AMOTA) is used as the main evaluation indicator, and the comparison results can be seen in Table 1 below.
[0109] Table 1
[0110]
[0111] As shown in Table 1, in the case of the body column-based backbone network and the voxel-based backbone network, the method provided by the embodiment of the present invention is superior to the center point and Kalman filter methods and has better performance. In addition, the center point and Kalman filter methods both require setting hyperparameters, while the method provided by the embodiment of the present invention completely gets rid of the situation of manually setting thresholds and has higher robustness.
[0112] The above describes in detail the method for tracking multiple objects provided by an embodiment of the present invention. The method can also be implemented by a corresponding device. The following describes in detail the device for tracking multiple objects provided by an embodiment of the present invention.
[0113] Figure 5 FIG. 1 is a schematic diagram showing the structure of a device for tracking multiple objects provided by an embodiment of the present invention. Figure 5 As shown, the multi-object tracking device includes: a mixed time position map determining module 51, a motion conversion module 52 and a tracking position map determining module 53.
[0114] The mixed time position map determination module 51 is used to determine the mixed time position map of the current point cloud segment, and the mixed time position map is used to indicate the first appearance position of the valid object in the current point cloud segment, and the valid object is other objects in the current point cloud segment except the invalid object.
[0115] The motion conversion module 52 is used to convert the tracking position map of the previous point cloud fragment into a temporary tracking position map of the current point cloud fragment according to the predicted motion field and the self-motion of the local device; the previous point cloud fragment is a point cloud fragment before the current point cloud fragment, and the previous point cloud fragment has the same partial point cloud data as the current point cloud fragment, and the tracking position map contains the position information and object identification of each object.
[0116] The tracking position map determination module 53 is used to average the mixed time position map of the current point cloud fragment and the temporary tracking position map to generate the tracking position map of the current point cloud fragment, and the tracking position map of the current point cloud fragment inherits the object identification in the tracking position map of the previous point cloud fragment.
[0117] Based on the above embodiment, the mixed time position graph determining module 51 determines the mixed time position graph of the current point cloud segment including:
[0118] Based on a preset backbone network, the current point cloud segment is converted into a corresponding hybrid time position map; the hybrid time position map contains the confidence of the first appearance position of a valid object in the current point cloud segment; the position information in the tracking position map represents the confidence of the object at the corresponding position.
[0119] Based on the above embodiment, the device further includes a training module.
[0120] Before the mixed time position graph determining module 51 determines the mixed time position graph of the current point cloud segment, the training module is used to:
[0121] A plurality of sample point cloud segments are generated according to a plurality of continuous point cloud data, wherein the sample point cloud segments are identical to part of the point cloud data of other adjacent sample point cloud segments; the first appearance position of each valid object in the sample point cloud segments is determined, and a sample mixed time position map including the first appearance position of each valid object in the sample point cloud segments is generated; and the backbone network is obtained by training according to the sample point cloud segments and the sample mixed time position map.
[0122] On the basis of the above embodiment, the device further includes a motion field determination module, which is used to: determine the motion field based on the backbone network, and the motion field is used to represent the motion parameters of the object in the current point cloud segment.
[0123] On the basis of the above embodiment, the tracking position map determining module 53 is further used for: when there is a new object in the tracking position map of the current point cloud segment, adding a corresponding object identifier to the new object.
[0124] Based on the above embodiment, the motion conversion module 52 converts the tracking position map of the previous point cloud segment into the temporary tracking position map of the current point cloud segment according to the predicted motion field and the self-motion of the local device, including:
[0125] updating the position information of each object in the tracking position map of the previous point cloud segment according to the motion field of the previous point cloud segment to generate a first tracking position map; updating the position information of each object in the first tracking position map according to the self-motion of the current point cloud segment to generate the temporary tracking position map; or
[0126] According to one of the motion field and the self-motion of the current point cloud fragment, the position information of each object in the tracking position map of the previous point cloud fragment is updated to generate a second tracking position map; according to the other of the motion field and the self-motion of the current point cloud fragment, the position information of each object in the second tracking position map is updated to generate the temporary tracking position map.
[0127] Based on the above embodiment, the device also includes a tracking module, which is used to: implement multi-object tracking based on the tracking position map of the current point cloud segment; or implement multi-object tracking based on the first tracking position map or the second tracking position map.
[0128] On the basis of the above embodiment, the tracking position map determining module 53 performs average processing on the mixed time position map of the current point cloud segment and the temporary tracking position map to generate the tracking position map of the current point cloud segment, including:
[0129] The mixed time position map of the current point cloud segment and the temporary tracking position map are averaged, and a non-maximum suppression operation is performed on the result of the average processing to generate the tracking position map of the current point cloud segment.
[0130] Based on the above embodiment, the first appearance position of the valid object is the first appearance position of the center of the valid object; and the position information is the position information of the center of the corresponding object.
[0131] Based on the above embodiment, the point cloud data of the last frame of the previous point cloud segment is the same as the point cloud data of the first frame of the current point cloud segment.
[0132] In addition, an embodiment of the present invention further provides an electronic device, including a bus, a transceiver, a memory, a processor, and a computer program stored in the memory and executable on the processor. The transceiver, the memory, and the processor are respectively connected via a bus. When the computer program is executed by the processor, each process of the above-mentioned multi-object tracking method embodiment is implemented, and the same technical effect can be achieved. To avoid repetition, it will not be described here.
[0133] For details, see Figure 6 As shown, an embodiment of the present invention further provides an electronic device, which may be the local device 10 described above, or a part of the local device 10, and which can obtain point cloud data collected by the laser radar 20. The electronic device includes a bus 1110, a processor 1120, a transceiver 1130, a bus interface 1140, a memory 1150, and a user interface 1160.
[0134] In the embodiment of the present invention, the electronic device further includes: a computer program stored in the memory 1150 and executable on the processor 1120, and when the computer program is executed by the processor 1120, each process of the above-mentioned multi-object tracking method embodiment is implemented.
[0135] The transceiver 1130 is configured to receive and send data under the control of the processor 1120 .
[0136] In an embodiment of the present invention, the bus architecture (represented by bus 1110), bus 1110 may include any number of interconnected buses and bridges, and bus 1110 connects various circuits including one or more processors represented by processor 1120 and a memory represented by memory 1150.
[0137] Bus 1110 represents one or more of any of several types of bus structures, including a memory bus and memory controller, a peripheral bus, an Accelerate Graphical Port (AGP), a processor, or a local bus using any of a variety of bus architectures. By way of example and not limitation, such architectures include: Industry Standard Architecture (ISA) bus, Micro Channel Architecture (MCA) bus, Enhanced ISA (EISA) bus, Video Electronics Standards Association (VESA), Peripheral Component Interconnect (PCI) bus.
[0138] The processor 1120 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method embodiment may be completed by an integrated logic circuit of hardware in the processor or an instruction in the form of software. The above processors include: a general-purpose processor, a central processing unit (CPU), a network processor (NP), a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA), a complex programmable logic device (CPLD), a programmable logic array (PLA), a microcontroller unit (MCU) or other programmable logic devices, discrete gates, transistor logic devices, discrete hardware components. The methods, steps and logic block diagrams disclosed in the embodiments of the present invention may be implemented or executed. For example, the processor may be a single-core processor or a multi-core processor, and the processor may be integrated into a single chip or located in multiple different chips.
[0139] Processor 1120 can be a microprocessor or any conventional processor. The method steps disclosed in conjunction with the embodiments of the present invention can be directly executed by a hardware decoding processor, or by a combination of hardware and software modules in the decoding processor. The software module can be located in a readable storage medium known in the art, such as a random access memory (RAM), a flash memory (FlashMemory), a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), a register, etc. The readable storage medium is located in a memory, and the processor reads the information in the memory and completes the steps of the above method in conjunction with its hardware.
[0140] The bus 1110 may also connect various other circuits such as peripheral devices, voltage regulators or power management circuits, and the bus interface 1140 provides an interface between the bus 1110 and the transceiver 1130, which are well known in the art. Therefore, the embodiments of the present invention will not be further described.
[0141] The transceiver 1130 may be one element or multiple elements, such as multiple receivers and transmitters, and provides a unit for communicating with various other devices on a transmission medium. For example, the transceiver 1130 receives external data from other devices, and the transceiver 1130 is used to send data processed by the processor 1120 to other devices. Depending on the nature of the computer system, a user interface 1160 may also be provided, such as a touch screen, a physical keyboard, a display, a mouse, a speaker, a microphone, a trackball, a joystick, and a stylus.
[0142] It should be understood that in an embodiment of the present invention, the memory 1150 may further include a memory remotely arranged relative to the processor 1120, and these remotely arranged memories may be connected to the server through a network. One or more parts of the above-mentioned network may be an ad hoc network, an intranet, an extranet, a virtual private network (VPN), a local area network (LAN), a wireless local area network (WLAN), a wide area network (WAN), a wireless wide area network (WWAN), a metropolitan area network (MAN), the Internet, a public switched telephone network (PSTN), a plain old telephone service network (POTS), a cellular telephone network, a wireless network, a wireless fidelity (Wi-Fi) network, and a combination of two or more of the above-mentioned networks. For example, the cellular telephone network and the wireless network can be a Global System for Mobile Communications (GSM) system, a Code Division Multiple Access (CDMA) system, a Worldwide Interoperability for Microwave Access (WiMAX) system, a General Packet Radio Service (GPRS) system, a Wideband Code Division Multiple Access (WCDMA) system, a Long Term Evolution (LTE) system, a LTE Frequency Division Duplex (FDD) system, a LTE Time Division Duplex (TDD) system, an Advanced Long Term Evolution (LTE-A) system, a Universal Mobile Telecommunications (UMTS) system, an Enhanced Mobile Broadband (Enhance Mobile Broadband, eMBB) system, a massive Machine Type of Communication (mMTC) system, an Ultra-Reliable Low Latency Communications (UltraReliable Low Latency Communications, uRLLC) system, etc.
[0143] It should be understood that the memory 1150 in the embodiment of the present invention may be a volatile memory or a non-volatile memory, or may include both a volatile memory and a non-volatile memory. Among them, the non-volatile memory includes: a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory.
[0144] Volatile memory includes: Random Access Memory (RAM), which is used as an external cache. By way of example but not limitation, many forms of RAM are available, such as: Static RAM (SRAM), Dynamic RAM (DRAM), Synchronous DRAM (SDRAM), Double Data Rate Synchronous Dynamic Random Access Memory (DDRSDRAM), Enhanced Synchronous Dynamic Random Access Memory (ESDRAM), Synchronous Link Dynamic Random Access Memory (SLDRAM) and Direct Rambus RAM (DRRAM). The memory 1150 of the electronic device described in the embodiment of the present invention includes but is not limited to the above and any other suitable types of memory.
[0145] In the embodiment of the present invention, the memory 1150 stores the following elements of the operating system 1151 and the application program 1152: executable modules, data structures, or subsets thereof, or extended sets thereof.
[0146] Specifically, the operating system 1151 includes various system programs, such as a framework layer, a core library layer, a driver layer, etc., which are used to implement various basic services and process hardware-based tasks. The application 1152 includes various applications, such as a media player (Media Player) and a browser (Browser), which are used to implement various application services. The program for implementing the method of the embodiment of the present invention may be included in the application 1152. The application 1152 includes applets, objects, components, logic, data structures, and other computer system executable instructions that perform specific tasks or implement specific abstract data types.
[0147] In addition, an embodiment of the present invention further provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the various processes of the above-mentioned multi-object tracking method embodiment are implemented and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.
[0148] Computer readable storage media include: permanent and non-permanent, removable and non-removable media, which are tangible devices that can retain and store instructions for use by instruction execution devices. Computer readable storage media include: electronic storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, and any suitable combination of the above. Computer readable storage media include: phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), non-volatile random access memory (NVRAM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassette storage, magnetic tape disk storage or other magnetic storage devices, memory sticks, mechanical encoding devices (such as punched cards or raised structures in grooves with instructions recorded thereon) or any other non-transmission medium that can be used to store information that can be accessed by a computing device. As defined in the embodiments of the present invention, computer-readable storage media do not include temporary signals themselves, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (such as light pulses passing through fiber optic cables), or electrical signals transmitted through wires.
[0149] In the several embodiments provided in the present application, it should be understood that the disclosed devices, electronic devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the modules or units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, or it can be an electrical, mechanical or other form of connection.
[0150] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, and may be located in one location or distributed on multiple network units. Some or all of the units may be selected according to actual needs to solve the problem to be solved by the embodiments of the present invention.
[0151] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.
[0152] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiment of the present invention is essentially or part of the contribution to the prior art, or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a computer device (including: a personal computer, a server, a data center or other network device) to perform all or part of the steps of the method described in each embodiment of the present invention. The above-mentioned storage medium includes various media that can store program codes as listed above.
[0153] In the description of the embodiments of the present invention, those skilled in the art should know that the embodiments of the present invention can be implemented as methods, devices, electronic devices and computer-readable storage media. Therefore, the embodiments of the present invention can be specifically implemented in the following forms: complete hardware, complete software (including firmware, resident software, microcode, etc.), a combination of hardware and software. In addition, in some embodiments, the embodiments of the present invention can also be implemented in the form of a computer program product in one or more computer-readable storage media, and the computer-readable storage medium contains computer program code.
[0154] The above-mentioned computer-readable storage medium may adopt any combination of one or more computer-readable storage media. Computer-readable storage media include: electrical, magnetic, optical, electromagnetic, infrared or semiconductor systems, devices or devices, or any combination of the above. More specific examples of computer-readable storage media include: portable computer disks, hard disks, random access memories (RAM), read-only memories (ROM), erasable programmable read-only memories (EPROM), flash memories, optical fibers, compact disk read-only memories (CD-ROM), optical storage devices, magnetic storage devices or any combination of the above. In an embodiment of the present invention, a computer-readable storage medium may be any tangible medium containing or storing a program, which may be used by or in combination with an instruction execution system, device, or device.
[0155] The computer program code contained in the above-mentioned computer-readable storage medium can be transmitted using any appropriate medium, including: wireless, wire, optical cable, radio frequency (RF) or any suitable combination thereof.
[0156] The computer program code for performing the operation of the embodiments of the present invention can be written in assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-related instructions, microcode, firmware instructions, state setting data, integrated circuit configuration data, or in one or more programming languages or a combination thereof, wherein the programming language includes an object-oriented programming language, such as Java, Smalltalk, C++, and also includes a conventional procedural programming language, such as C language or a similar programming language. The computer program code can be executed completely on the user's computer, partially on the user's computer, as an independent software package, partially on the user's computer, partially on a remote computer, and completely on a remote computer or server. In the case of a remote computer, the remote computer can be connected to the user's computer or to an external computer through any type of network, including a local area network (LAN) or a wide area network (WAN).
[0157] The embodiments of the present invention describe the provided methods, devices, and electronic devices through flowcharts and / or block diagrams.
[0158] It should be understood that each box in the flowchart and / or block diagram and the combination of boxes in the flowchart and / or block diagram can be implemented by computer-readable program instructions. These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer or other programmable data processing device to produce a machine, and these computer-readable program instructions are executed by a computer or other programmable data processing device to produce a device that implements the functions / operations specified by the boxes in the flowchart and / or block diagram.
[0159] These computer-readable program instructions may also be stored in a computer-readable storage medium that enables a computer or other programmable data processing device to work in a specific manner. In this way, the instructions stored in the computer-readable storage medium produce an instruction device product including functions / operations specified in the blocks in the flowchart and / or block diagram.
[0160] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device, so that a series of operational steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby enabling the instructions executed on the computer or other programmable data processing apparatus to provide a process for implementing the functions / operations specified in the blocks in the flowchart and / or block diagram.
[0161] The above is only a specific implementation of the embodiment of the present invention, but the protection scope of the embodiment of the present invention is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the embodiment of the present invention, which should be included in the protection scope of the embodiment of the present invention. Therefore, the protection scope of the embodiment of the present invention should be based on the protection scope of the claims.
Claims
1. A method for tracking multiple objects, characterized in that: include: Determine a mixed time position graph of the current point cloud segment, wherein the mixed time position graph is used to indicate the first appearance position of a valid object in the current point cloud segment, wherein the valid object is other objects in the current point cloud segment except for the invalid object; Converting a tracking position map of a previous point cloud segment into a temporary tracking position map of the current point cloud segment according to the predicted motion field and the self-motion of the local device; the previous point cloud segment is a point cloud segment before the current point cloud segment, and the previous point cloud segment has the same partial point cloud data as the current point cloud segment, and the tracking position map includes position information and object identification of each object; Averaging the mixed time position map of the current point cloud segment and the temporary tracking position map to generate a tracking position map of the current point cloud segment, wherein the tracking position map of the current point cloud segment inherits the object identifier in the tracking position map of the previous point cloud segment; Determining the mixed time position graph of the current point cloud segment includes: Based on a preset backbone network, convert the current point cloud segment into a corresponding hybrid time position map; the hybrid time position map includes the confidence of the first appearance position of the valid object in the current point cloud segment; The position information in the tracking position map represents the confidence of the object at the corresponding position; The step of converting the tracking position map of the previous point cloud segment into the temporary tracking position map of the current point cloud segment according to the predicted motion field and the self-motion of the local device comprises: updating the position information of each object in the tracking position map of the previous point cloud segment according to the motion field of the previous point cloud segment to generate a first tracking position map; updating the position information of each object in the first tracking position map according to the self-motion of the current point cloud segment to generate the temporary tracking position map; or According to one of the motion field and the self-motion of the current point cloud fragment, the position information of each object in the tracking position map of the previous point cloud fragment is updated to generate a second tracking position map; according to the other of the motion field and the self-motion of the current point cloud fragment, the position information of each object in the second tracking position map is updated to generate the temporary tracking position map.
2. The method according to claim 1, characterized in that: Before determining the mixed time position graph of the current point cloud segment, the method further includes: Generating a plurality of sample point cloud segments according to a plurality of continuous point cloud data, wherein the sample point cloud segments have the same partial point cloud data as other adjacent sample point cloud segments; Determine the first appearance position of each valid object in the sample point cloud segment, and generate a sample mixed time position map including the first appearance position of each valid object in the sample point cloud segment; The backbone network is obtained by training according to the sample point cloud fragments and the sample mixed time position graph.
3. The method according to claim 1, characterized in that Also includes: The motion field is determined based on the backbone network, where the motion field is used to represent motion parameters of an object in the current point cloud segment.
4. The method according to claim 1, characterized in that Also includes: When there is a new object in the tracking position map of the current point cloud segment, a corresponding object identifier is added to the new object.
5. The method according to claim 1, characterized in that Also includes: Implementing multi-object tracking based on the tracking position graph of the current point cloud segment; or Multi-object tracking is implemented based on the first tracking position map or the second tracking position map.
6. The method according to claim 1, characterized in that The averaging of the mixed time position map of the current point cloud segment and the temporary tracking position map to generate the tracking position map of the current point cloud segment includes: The mixed time position map of the current point cloud segment and the temporary tracking position map are averaged, and a non-maximum suppression operation is performed on the result of the average processing to generate the tracking position map of the current point cloud segment.
7. The method according to claim 1, characterized in that The first appearance position of the valid object is the first appearance position of the center of the valid object; The position information is the position information of the center of the corresponding object.
8. The method according to claim 1, characterized in that The point cloud data of the last frame of the previous point cloud segment is the same as the point cloud data of the first frame of the current point cloud segment.
9. A device for tracking multiple objects, characterized in that: include: A mixed time position graph determination module, used to determine a mixed time position graph of a current point cloud segment, wherein the mixed time position graph is used to indicate the first appearance position of a valid object in the current point cloud segment, wherein the valid object is other objects in the current point cloud segment except for an invalid object; a motion conversion module, configured to convert a tracking position map of a previous point cloud segment into a temporary tracking position map of the current point cloud segment according to the predicted motion field and the self-motion of the local device; the previous point cloud segment is a point cloud segment before the current point cloud segment, and the previous point cloud segment has the same partial point cloud data as the current point cloud segment, and the tracking position map includes position information and object identification of each object; a tracking position map determining module, configured to average the mixed time position map of the current point cloud segment and the temporary tracking position map to generate a tracking position map of the current point cloud segment, wherein the tracking position map of the current point cloud segment inherits the object identifier in the tracking position map of the previous point cloud segment; Determining the mixed time position graph of the current point cloud segment includes: Based on a preset backbone network, convert the current point cloud segment into a corresponding hybrid time position map; the hybrid time position map includes the confidence of the first appearance position of the valid object in the current point cloud segment; The position information in the tracking position map represents the confidence of the object at the corresponding position; The step of converting the tracking position map of the previous point cloud segment into the temporary tracking position map of the current point cloud segment according to the predicted motion field and the self-motion of the local device comprises: updating the position information of each object in the tracking position map of the previous point cloud segment according to the motion field of the previous point cloud segment to generate a first tracking position map; updating the position information of each object in the first tracking position map according to the self-motion of the current point cloud segment to generate the temporary tracking position map; or According to one of the motion field and the self-motion of the current point cloud fragment, the position information of each object in the tracking position map of the previous point cloud fragment is updated to generate a second tracking position map; according to the other of the motion field and the self-motion of the current point cloud fragment, the position information of each object in the second tracking position map is updated to generate the temporary tracking position map.
10. An electronic device, comprising a bus, a transceiver, a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the transceiver, the memory, and the processor are connected via the bus, wherein: When the computer program is executed by the processor, the steps in the method for multi-object tracking according to any one of claims 1 to 8 are implemented.
11. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps in the method for multi-object tracking according to any one of claims 1 to 8 are implemented.
