A method, apparatus, electronic device, and storage medium for generating target trajectories.

Abstract

This invention discloses a method, apparatus, electronic device, and storage medium for generating target trajectories. The method includes: inputting at least two frames of target point cloud data into a target detection model; outputting pose detection results and corresponding actual detection results for each of the at least two frames of target point cloud data through the target detection model; based on the pose detection results and the motion model, selecting at least two target detection results for a target object from each candidate detection result corresponding to the actual detection results of the at least two frames of target point cloud data; the target object is determined by selecting at least one candidate object, and the at least two target detection results belong to the actual detection results corresponding to target point cloud data in different frames; and generating the trajectory of the target object based on the at least two target detection results corresponding to the target object. This solution can achieve relatively comprehensive and reliable target detection and generate accurate target trajectories.

Description

Technical Field

[0001] This invention relates to the field of autonomous driving technology, and in particular to a method, apparatus, electronic device, and storage medium for generating target trajectories. Background Technology

[0002] With the development of autonomous driving technology, the perception and understanding of the environment by vehicles has become particularly important. Target trajectory generation is a key link in constructing traffic scenarios, responsible for obtaining the target's driving path based on the vehicle's current position, the target's position, and surrounding environmental information. However, 2D image perception lacks depth information; in 3D point cloud perception, target information is discrete and disordered in time, and target detection results are unstable, with occluded targets being difficult to detect, directly affecting target trajectory prediction. It can be said that 3D target detection algorithms are a crucial factor affecting the safety of autonomous driving systems. Currently, 3D target detection schemes for point clouds are mainly divided into point-based and mesh-based methods. Point-based methods suffer from inference time bottlenecks because they directly process point clouds, requiring point cloud sampling, resulting in high computational costs. Furthermore, they are prone to losing fine-grained 3D information during feature learning. Mesh-based methods inevitably lose 3D information with coordinate transformations, and their effectiveness largely depends on the size of the mesh cells. This method struggles to balance the efficiency gains from smaller mesh sizes with the increased memory requirements. Summary of the Invention

[0003] This invention provides a target trajectory generation method, apparatus, electronic device, and storage medium, which can achieve relatively comprehensive and reliable target detection and generate accurate target trajectories.

[0004] In a first aspect, the present invention provides a method for generating a target trajectory, comprising:

[0005] At least two frames of target point cloud data are input into a target detection model. The target detection model outputs pose detection results and actual detection results corresponding to the at least two frames of target point cloud data. The at least two frames of target point cloud data are point cloud data acquired continuously in terms of acquisition time. The actual detection result corresponding to each frame of target point cloud data includes candidate detection results of at least one candidate object contained in each frame of target point cloud data. The candidate detection results include the category and position of the candidate object. The pose detection results are used to indicate the pose relationship between two adjacent frames of target point cloud data in terms of acquisition time. The target detection model is used to parse the at least two frames of point cloud data that are consecutive in terms of acquisition time and outputs category detection results and position detection results of objects in each frame of point cloud data, as well as the pose relationship between two adjacent frames of point cloud data.

[0006] Based on the pose detection results and motion model, at least two target detection results of the target object are selected from each candidate detection result corresponding to the actual detection results of at least two frames of target point cloud data; the target object is determined by selecting the at least one candidate object, and the at least two target detection results belong to the actual detection results corresponding to target point cloud data in different frames;

[0007] Based on the detection results of at least two targets corresponding to the target object, the trajectory of the target object is generated.

[0008] Secondly, the present invention also provides a target trajectory generation device, comprising:

[0009] A detection module is used to input at least two frames of target point cloud data into a target detection model, and output pose detection results and actual detection results corresponding to the at least two frames of target point cloud data through the target detection model. The at least two frames of target point cloud data are point cloud data acquired continuously in the acquisition time. The actual detection result corresponding to each frame of target point cloud data includes candidate detection results of at least one candidate object contained in each frame of target point cloud data. The candidate detection results include the category and position of the candidate object. The pose detection results are used to indicate the pose relationship between two adjacent frames of target point cloud data in the acquisition time. The target detection model is used to parse the at least two frames of point cloud data that are consecutive in the acquisition time, and output the category detection results and position detection results of objects in each frame of point cloud data, as well as the pose relationship between two adjacent frames of point cloud data.

[0010] The target detection result determination module is used to select at least two target detection results of a target object from each candidate detection result corresponding to the actual detection result of at least two frames of target point cloud data, based on the pose detection result and the motion model; the target object is determined by selecting the at least one candidate object, and the at least two target detection results belong to the actual detection results corresponding to target point cloud data in different frames;

[0011] The trajectory generation module is used to generate the trajectory of the target object based on the detection results of at least two targets corresponding to the target object.

[0012] Thirdly, embodiments of the present invention also provide an electronic device, including: one or more processors; and a storage device for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the target trajectory generation method provided in any embodiment of the present invention.

[0013] Fourthly, embodiments of the present invention also provide a storage medium containing computer-executable instructions, which, when executed by a computer processor, are used to perform the target trajectory generation method provided in any embodiment of the present invention.

[0014] The technical solution provided by this invention involves inputting at least two frames of target point cloud data into a target detection model, and outputting pose detection results and actual detection results corresponding to the at least two frames of target point cloud data, wherein the at least two frames of target point cloud data are point cloud data acquired continuously in terms of acquisition time, and the actual detection result corresponding to each frame of target point cloud data includes candidate detection results of at least one candidate object contained in each frame of target point cloud data. The candidate detection results include the category and position of the candidate object, and the pose detection results are used to indicate the pose relationship between two frames of target point cloud data that are adjacent in terms of acquisition time. Based on the pose detection results and the motion model, at least two target detection results of the target object are selected from each candidate detection result corresponding to the actual detection results of the at least two frames of target point cloud data. The target object is determined by selecting at least one candidate object, and the at least two target detection results belong to the actual detection results corresponding to target point cloud data in different frames. Based on the at least two target detection results corresponding to the target object, the trajectory of the target object is generated. The solution of this invention can achieve more comprehensive and reliable target detection and generate accurate target trajectories. In particular, it can solve problems such as short-term occlusion and disappearance of targets, making the final generated target trajectory more comprehensive and accurate.

[0015] The above description of the invention is merely an overview of the technical solution of the present invention. In order to better understand the technical means of the present invention and to implement it in accordance with the contents of the specification, and to make the above and other objects, features and advantages of the present invention more apparent and understandable, specific embodiments of the present invention are described below. Attached Figure Description

[0016] The above and other features, advantages, and aspects of the various embodiments of the present invention will become more apparent from the accompanying drawings and the following detailed description. Throughout the drawings, the same or similar reference numerals denote the same or similar elements. It should be understood that the drawings are schematic, and the originals and elements are not necessarily drawn to scale.

[0017] Figure 1 This is a flowchart illustrating a target trajectory generation method provided in an embodiment of the present invention;

[0018] Figure 2 This is a flowchart illustrating another target trajectory generation method provided in an embodiment of the present invention;

[0019] Figure 3aA schematic diagram of an equally spaced cylindrical space provided in an embodiment of the present invention;

[0020] Figure 3b This is a schematic diagram illustrating the effect of stacking bird's-eye view features corresponding to point cloud data, as provided in an embodiment of the present invention.

[0021] Figure 3c This is a schematic diagram illustrating the training process of an object detection model provided in an embodiment of the present invention;

[0022] Figure 4 A flowchart illustrating another target trajectory generation method provided in an embodiment of the present invention;

[0023] Figure 5 This is a schematic diagram of the structure of a target trajectory generation device provided in an embodiment of the present invention;

[0024] Figure 6 This is a schematic diagram of the structure of an electronic device for implementing a target trajectory generation method provided in an embodiment of the present invention. Detailed Implementation

[0025] Embodiments of the present invention will now be described in more detail with reference to the accompanying drawings. While some embodiments of the invention are shown in the drawings, it should be understood that the invention can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of the invention. It should be understood that the drawings and embodiments of the present invention are for illustrative purposes only and are not intended to limit the scope of protection of the invention. It should be understood that the various steps described in the method embodiments of the present invention may be performed in different orders and / or in parallel. Furthermore, the method embodiments may include additional steps and / or omit the steps shown. The scope of the invention is not limited in this respect.

[0026] The term "comprising" and its variations as used herein are open-ended inclusions, meaning "including but not limited to". The term "based on" means "at least partially based on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". Definitions of other terms will be given in the description below.

[0027] It should be noted that the concepts of "first," "second," etc., mentioned in this invention are only used to distinguish different devices, modules, or units, and are not used to limit the order of functions performed by these devices, modules, or units or their interdependencies. It should also be noted that the modifications "a" or "a plurality of" mentioned in this invention are illustrative rather than restrictive, and those skilled in the art should understand that, unless explicitly stated in the context, they should be understood as "one or more." The names of messages or information exchanged between multiple devices in the embodiments of this invention are for illustrative purposes only and are not intended to limit the scope of these messages or information.

[0028] Figure 1 This is a flowchart illustrating a target trajectory generation method provided in an embodiment of the present invention. This embodiment is applicable to the construction and generation of a target object's trajectory. The method can be executed by a target trajectory generation device, which can be implemented in software and / or hardware and is generally integrated into any electronic device with network communication capabilities, such as a mobile terminal, PC, or server. Figure 1 As shown, the target trajectory generation method of this invention embodiment may include the following process:

[0029] S101. Input at least two frames of target point cloud data into the target detection model, and output the pose detection result and the actual detection result corresponding to each of the at least two frames of target point cloud data. The at least two frames of target point cloud data are point cloud data acquired continuously in the acquisition time. The actual detection result corresponding to each frame of target point cloud data includes the candidate detection result of at least one candidate object contained in each frame of target point cloud data. The candidate detection result includes the category and position of the candidate object. The pose detection result is used to indicate the pose relationship between two adjacent frames of target point cloud data in the acquisition time. The target detection model is used to parse the at least two consecutive frames of point cloud data in the acquisition time and output the category detection result and position detection result of the object in each frame of point cloud data, as well as the pose relationship between two adjacent frames of point cloud data.

[0030] Point cloud data refers to a collection of points in three-dimensional space. A frame of point cloud data contains the three-dimensional coordinates and reflection intensity information of multiple different signal reflection points. Different reflection points constitute different objects, such as trees, cars, and pedestrians. Each frame of point cloud data also has its own corresponding acquisition time. For example, the time when a vehicle's onboard LiDAR acquires a frame of point cloud data is the acquisition time corresponding to that frame.

[0031] The target detection model requires at least two frames of target point cloud data that are acquired consecutively in terms of acquisition time. Continuous acquisition time can mean that the acquisition time for each frame of point cloud data is progressively increasing. For example, if point cloud data is acquired every 1 second, and the target detection model receives three frames of target point cloud data, then the first frame must be acquired 1 second earlier than the second frame, and the second frame must be acquired 1 second earlier than the third frame.

[0032] The object detection model can be used to parse at least two consecutive frames of point cloud data acquired in time, outputting the category detection results and position detection results of objects in each frame, as well as the pose relationship between adjacent frames. The object category detection result reflects the category to which the detected object belongs, and the object position detection result reflects the position information of the detected object in the point cloud data of its respective frame. The pose relationship reflects the positional change relationship between corresponding points in adjacent frames. Specifically, because the point cloud data is acquired continuously in time, there must be a certain positional change relationship between corresponding points of the same object in two frames, which can be described using the pose relationship.

[0033] Specifically, at least two frames of target point cloud data can be input into the target detection model. The target detection model outputs pose detection results and the corresponding actual detection results for each of the at least two frames of target point cloud data. The pose detection results indicate the pose relationship between two adjacent frames of point cloud data input to the target detection model. The actual detection results for each frame of target point cloud data include candidate detection results for at least one candidate object contained in that frame, and these candidate detection results include the category and location of the candidate object.

[0034] For example, taking the input of two frames of target point cloud data into a target detection model for processing, the target detection model can output the pose detection result between the two frames of point cloud data, the actual detection result corresponding to the first frame of target point cloud data, and the actual detection result corresponding to the second frame of target point cloud data. The actual detection result corresponding to the first frame of target point cloud data includes the candidate detection results for each candidate object contained in the first frame of target point cloud data. Assuming there are three candidate objects, the actual detection result corresponding to the first frame of target point cloud data would then include the candidate detection results for candidate object a1, candidate object a2, and candidate object a3; and the candidate detection result for each candidate object includes the category and position of that candidate object. Similarly, the actual detection result corresponding to the second frame of target point cloud data also contains specific relevant information of the same category, which will not be elaborated upon here.

[0035] It should be noted that the pose detection result output by the target detection model indicates the pose relationship between two adjacent frames of target point cloud data in terms of acquisition time, at least two frames. If there are only two frames of input target point cloud data, there will be only one output pose detection result, which is the pose detection result between the two input target point cloud data frames. If there are three frames of input target point cloud data, there will be two output pose detection results: one is the pose detection result between the first and second input target point cloud data frames, and the other is the pose detection result between the second and third input target point cloud data frames. And so on. The same principle will not be elaborated here.

[0036] S102. Based on the pose detection results and motion model, select at least two target detection results of the target object from each candidate detection result corresponding to the actual detection results of at least two frames of target point cloud data; the target object is determined by selecting at least one candidate object, and the at least two target detection results belong to the actual detection results corresponding to target point cloud data of different frames.

[0037] Since the actual detection results corresponding to each frame of target point cloud data output by the object detection model include the candidate detection results of each candidate object in the point cloud data of that frame, and the candidate objects in different frames may essentially belong to the same object, it is possible to determine the relevant candidate detection results belonging to the same object from the candidate detection results corresponding to the point cloud data of each frame, and use them as the target detection results corresponding to that same object.

[0038] Specifically, based on pose detection results and motion models, at least two target detection results can be selected from the candidate detection results corresponding to the actual detection results of at least two frames of target point cloud data to identify the target object. Here, the target object is determined by filtering at least one candidate object, and the identified at least two target detection results belong to the actual detection results corresponding to target point cloud data from different frames.

[0039] For example, taking the actual detection results corresponding to two frames of target point cloud data as an example, assume that the actual detection results corresponding to the first frame of target point cloud data include three candidate objects, and the actual detection results corresponding to the second frame of target point cloud data include four candidate objects, that is: the actual detection results corresponding to the first frame of target point cloud data include the candidate detection results corresponding to candidate object a1, candidate object a2, and candidate object a3; the actual detection results corresponding to the second frame of target point cloud data include the candidate detection results corresponding to candidate object b1, candidate object b2, candidate object b3, and candidate object b4. Then, among the candidate objects corresponding to these different frames, there are different candidate objects that are essentially the same object. For example, based on the pose detection results and motion model, it is determined that candidate object a1 and candidate object b2 are essentially the same object (specifically, a tree), and candidate object a2 and candidate object b3 are essentially other objects that are the same object (specifically, a pedestrian). Correspondingly, the candidate detection results for candidate object a1 and candidate object b2 constitute a set of target detection results for the "tree" object; the candidate detection results for candidate object a2 and candidate object b3 constitute another set of target detection results for the "pedestrian" object. And so on, target detection results belonging to the same object can be determined from the candidate detection results corresponding to the point cloud data in each frame.

[0040] S103. Generate the trajectory of the target object based on at least two target detection results corresponding to the target object.

[0041] Since the at least two target detection results corresponding to the identified target object are candidate detection results from point cloud data of different frames, each target detection result reflects the category and location information of the target object in the point cloud data of its respective frame. Since the point cloud data of different frames also have different acquisition times, the trajectory of the target object can be generated based on the location information of the target object in the at least two target detection results and the acquisition time of the point cloud data associated with the at least two target detection results.

[0042] The technical solution provided by this invention involves inputting at least two frames of target point cloud data into a target detection model, which then outputs pose detection results and the corresponding actual detection results for each of the at least two frames of target point cloud data. Based on the pose detection results and the motion model, at least two target detection results for the target object are selected from the candidate detection results corresponding to the actual detection results of the at least two frames of target point cloud data. The target object is determined by selecting at least one candidate object, and the at least two target detection results belong to the actual detection results corresponding to target point cloud data from different frames. Based on the at least two target detection results corresponding to the target object, the trajectory of the target object is generated. Using the solution of this invention, relatively comprehensive and reliable target detection can be achieved, and accurate target trajectories can be generated. In particular, it can solve problems such as short-term occlusion and disappearance of targets, making the final generated target trajectory more comprehensive and accurate.

[0043] Figure 2 This is a flowchart illustrating another target trajectory generation method provided by an embodiment of the present invention. The technical solution of this embodiment further optimizes the training process of the target detection model in the aforementioned embodiments based on the technical solutions of the above embodiments. This embodiment can be combined with various optional solutions in one or more of the above embodiments. Figure 2 As shown, the target trajectory generation method of this invention may include the following process:

[0044] S201. Obtain at least two consecutive frames of historical point cloud data within a historical time period.

[0045] Specifically, vehicle-mounted LiDAR can be used to collect at least two consecutive frames of historical point cloud data under different scenarios within a historical time period. For example, the acquired historical point cloud data can be represented as (x, y, z, r, timestamp), where x, y, and z reflect the coordinate values ​​of the signal reflection points contained in the point cloud data in the three-dimensional spatial coordinate system, r is the reflection intensity of the signal reflection point, and timestamp is the timestamp of signal acquisition, which is essentially the acquisition time corresponding to that frame of historical point cloud data.

[0046] S202. Determine the pre-annotation results corresponding to at least two frames of historical point cloud data. The pre-annotation results include the historical category detection results and historical position detection results of the objects in each frame of historical point cloud data, as well as the historical pose relationship between two adjacent frames of historical point cloud data.

[0047] Specifically, for each frame of historical point cloud data acquired, category and location information annotations can be performed on each object contained in the historical point cloud data, thus obtaining the historical category detection results and historical location detection results corresponding to each object in each frame of historical point cloud data. Simultaneously, the historical pose relationships between at least two adjacent frames of historical point cloud data are also annotated. All these annotation information are then used as the pre-annotation results corresponding to at least two frames of historical point cloud data.

[0048] As an optional but non-limiting implementation, the process of determining the historical pose relationship between two adjacent frames of historical point cloud data may include the following steps A1-A4:

[0049] Step A1: Determine the transformation matrix of corresponding points in two adjacent frames of historical point cloud data using the nearest neighbor method.

[0050] Since the acquisition time of two adjacent frames of historical point cloud data is continuous, the transformation matrices between adjacent historical point cloud data will be similar. Specifically, a random sampling method can be used to preprocess at least two frames of historical point cloud data to reduce the number of signal reflection points related to unwanted objects in the historical point cloud data. For example, this can be to remove signal reflection points related to the road surface in the historical point cloud data, thereby reducing the number of signal reflection points in the historical point cloud data and reducing the computational load of the scheme.

[0051] Then, the nearest neighbor method is used to determine the corresponding points P = {p1, p2, ..., p} in two adjacent frames of historical point cloud data. n} and Q = {q1,q2,...,q n}, where p1, p2, ..., p n For points in the earliest frame of historical point cloud data, q1, q2, ..., q n For points in the historical point cloud data collected at a later time, points p1 and q1 are considered as one pair of corresponding points, points p2 and q2 are considered as another pair of corresponding points, and so on, until point p... n and point q n Let be a set of corresponding points. For these different sets of corresponding points, the transformation matrix (R,t) can be determined based on the following formula:

[0052] q i =Rp i +t;

[0053] Where R is the rotation matrix; t is the translation vector between two corresponding points; and i is the subscript of each pair of corresponding points, i∈[1,n].

[0054] It should be noted that the corresponding points determined by the nearest neighbor method can be considered as a coarse correspondence, which is not precise enough. Therefore, further processing can be carried out in subsequent schemes to determine more accurate corresponding points.

[0055] Step A2: Update the historical point cloud data of two adjacent frames based on the transformation matrix to determine the new corresponding points.

[0056] Specifically, based on the calculated transformation matrix (R,t), the points in the later-collected historical point cloud data frame between two adjacent frames can be updated, that is, the corresponding point Q mentioned above is updated to obtain the new point Q. update ={q 11 ,q 22 ,...,q nn Specifically, you can use Q. update The formula =RP+t is used for calculation.

[0057] Step A3: Optimize the transformation matrix iteratively based on the new corresponding points until the convergence condition is met to obtain the optimal transformation matrix; the convergence condition is that the error of corresponding points in two adjacent frames of historical point cloud data reaches a set threshold.

[0058] Specifically, the updated corresponding point Q update ={q 11 ,q 22 ,...,q nn} and P = {p1, p2, ..., p n The transformation matrix is ​​then solved in a new round, thus achieving iterative optimization of the transformation matrix. This process is repeated until a convergence condition is met, yielding the optimal transformation matrix. The convergence condition is that the error between corresponding points in two adjacent frames of historical point cloud data reaches a set threshold. That is, the error between corresponding points P and Q... update Once the error between the two reaches a set threshold, the optimal transformation matrix can be obtained.

[0059] Step A4: Use the optimal transformation matrix as the historical pose relationship between two adjacent frames of historical point cloud data.

[0060] Once the optimal transformation matrix is ​​determined, it can be used as the historical pose relationship between two adjacent frames of historical point cloud data.

[0061] S203. Input at least two frames of historical point cloud data and pre-labeled results into the target detection model.

[0062] Specifically, at least two frames of historical point cloud data, the detection results and location information of each object annotation contained in each frame of historical point cloud data, and the historical pose relationship between two adjacent frames of historical point cloud data can be input into the target detection model to train the target detection model.

[0063] S204. The target detection model fuses the rasterized historical point cloud data of each frame based on the temporal information corresponding to the historical point cloud data of each frame to obtain the 3D point cloud features of the historical point cloud data of each frame.

[0064] The temporal information corresponding to each frame of historical point cloud data is the acquisition time of that frame. Rasterization is the process of processing each frame of historical point cloud data to convert it into a 3D cylindrical mesh representation. Specifically, after knowing the temporal information of each frame of historical point cloud data, the rasterized frames can be fused using an object detection model based on the temporal information to obtain the 3D point cloud features of each frame.

[0065] As an optional but non-limiting implementation, the process of rasterizing historical point cloud data for each frame may include the following steps B1-B3:

[0066] Step B1: Based on the historical pose relationship and the time deviation information between each frame of historical point cloud data, transform each frame of historical point cloud data to the same coordinate system to align the historical point cloud data of each frame.

[0067] The time deviation information refers to the deviation in acquisition time between each frame of historical point cloud data and the earliest acquired frame. Specifically, this can be obtained by subtracting the acquisition time of the earliest acquired frame from the acquisition time of each frame of historical point cloud data.

[0068] Specifically, based on the historical pose relationships between adjacent historical point cloud data and the time deviation information between each frame of historical point cloud data, each frame of historical point cloud data can be transformed into the same coordinate system. In the same coordinate system, each frame of historical point cloud data can be represented as (x... i ,y i ,z i ,r i ,t_offset i ), where x i ,y i ,z i These respectively reflect the coordinate values ​​of the signal reflection points contained in the historical point cloud data of that frame in the same transformed coordinate system, r it_offset represents the reflection intensity at the signal reflection point. i This provides the time deviation information corresponding to the historical point cloud data for this frame. Transforming the historical point cloud information of each frame to the same coordinate system not only aligns the historical point cloud data of each frame, but also intuitively reflects the temporal differences and progression of historical point cloud data in adjacent frames. It also allows for a clear view of the specific situation of each frame's point cloud data, which is more conducive to the subsequent fusion of rasterized point cloud data.

[0069] Step B2: Divide the historical point cloud data of each frame into equal-spaced cylindrical spaces so that the points in the point cloud data are distributed in each cylindrical space.

[0070] In this context, an equally spaced cylindrical space refers to a space where the distance between each cylindrical element in the cross-section is equal. Specifically, for a frame of historical point cloud data, equally spaced cylindrical spaces are used to divide the data so that each point in the frame can be distributed within its corresponding cylindrical space. For example, Figure 3a This is a schematic diagram of an equally spaced cylindrical space provided in an embodiment of the present invention, as shown below. Figure 3a As shown, the equally spaced cylindrical space consists of 4x4 cylinders, each with the same length and width. This cylindrical space allows for the division of the historical point cloud data frame, ensuring that each point in the frame is distributed within one of these 4x4 cylinders. This process is repeated for each frame of historical point cloud data, achieving spatial division for each frame. It is important to note that even when dividing historical point cloud data across different frames, the length and width of each cylinder must be identical; that is, the division must adhere to the same standardized cylindrical space. This process enables the rasterization of each frame of historical point cloud data.

[0071] It should be noted that the number of points distributed within each of these 4x4 cylinders may vary, depending on the points contained in the historical point cloud data of that frame being divided. Meanwhile, the number of cylinders used to divide the historical point cloud data is not limited; it needs to be determined based on the number and range of points in the historical point cloud data. If a cylinder space contains parts that have no points inside after division, such a cylinder is called an empty cylinder, which can be padded with 0s in subsequent calculations. Conversely, cylinders containing points are considered non-empty cylinders.

[0072] Step B3: Based on the center point of each cylinder, perform dimensionality enhancement on the points in each frame of point cloud data that belong to the same cylinder space.

[0073] For each frame of historical point cloud data, dimensionality enhancement can be performed on each point in each frame to incorporate local residual information into the point cloud information, thereby enhancing the features of the point cloud data. Specifically, for a frame of historical point cloud data, the offset between each point in that frame and the center point of the column to which it belongs can be used as a new feature dimension and added to the data corresponding to each point to achieve dimensionality enhancement for each point.

[0074] For example, continue as follows Figure 3a As shown, to the Figure 3a Taking the dimensionality enhancement of the points distributed in the bottom left corner cylinder as an example, first determine the coordinates of the center point of the bottom left corner cylinder as (x... c ,y c ,z c Then, determine the offset of each point within the cylinder relative to the center point, and perform dimensionality enhancement on each point based on the obtained offsets. After dimensionality enhancement, each point is represented as (x... i ,y i ,z i ,r i ,t_offset i ,x c ,y c ,z c ,x p ,y p ), where x p The reflected signal point (x) i ,y i ,z i ) relative to the center point (x) of its respective column c ,y c ,z c The difference in coordinates along the X-axis of the coordinate system; y p The reflected signal point (x) i ,y i ,z i ) relative to the center point (x) of its respective column c ,y c ,z c The difference in coordinates along the Y-axis of the coordinate system. That is, x... p ,y pEssentially, this reflects the offset of the signal reflection point relative to the center point of its respective cylinder. After adding this offset, each point gains 10 dimensions, thus increasing the dimensionality of the data. Similarly, the same principle is applied to points distributed within other cylinders; that is, all points within the entire 4x4 cylinder are processed, achieving dimensionality increase for each point in that frame of historical point cloud data. The same principle applies to all frames of historical point cloud data, achieving dimensionality increase for each frame.

[0075] After dimensionality enhancement, the feature tensor corresponding to each frame of historical point cloud data is (D, P, N), where D represents the number of feature dimensions of each point after rasterization, N represents the number of points distributed in each non-empty cylinder, and P represents the total number of non-empty cylinders. It should be noted that an upper limit can be set for the parameter N. If the number of points distributed in a non-empty cylinder exceeds the upper limit of N, only N points are taken to downsample the point cloud data, thereby reducing the complexity of the data and the amount of computation.

[0076] As an optional but non-limiting implementation, the rasterized historical point cloud data of each frame is fused based on the temporal information corresponding to the historical point cloud data of each frame by the object detection model to obtain the 3D point cloud features of the historical point cloud data of each frame. This may include the following steps C1-C2:

[0077] Step C1: The rasterized historical point cloud data of each frame is processed by the object detection model based on a sparse convolutional neural network to obtain the bird's-eye view features of each frame of historical point cloud data from the bird's-eye view perspective.

[0078] Specifically, for the obtained feature tensor (D,P,N), the rasterized historical point cloud data of each frame is processed by the object detection model based on a sparse convolutional neural network to obtain the bird's-eye view features of each frame of historical point cloud data under the bird's-eye view.

[0079] Step C2: Based on the temporal information corresponding to each frame of historical point cloud data, the target detection model stacks the bird's-eye view features corresponding to each frame of historical point cloud data in the time dimension to obtain the 3D point cloud features of each frame of historical point cloud data.

[0080] Specifically, since temporal information can reflect the temporal relationship of historical point cloud data in each frame, the bird's-eye view features corresponding to each frame of historical point cloud data can be stacked in the time dimension. Based on the temporal information, the degree of translation of each bird's-eye view feature during the stacking process can be determined. Then, by using the backbone neural network of the object detection model for feature extraction, the 3D point cloud features of each frame of historical point cloud data can be obtained. For example, the backbone neural network can be a PointPillar backbone neural network.

[0081] For example, taking the stacking of bird's-eye view features from two frames of historical point cloud data as an example, such as... Figure 3b As shown, Figure 3b This is a schematic diagram illustrating the effect of stacking bird's-eye view features corresponding to point cloud data, as provided in an embodiment of the present invention. Figure 3b The orange column on the upper left is T. k-1 The bird's-eye view features corresponding to the frame of historical point cloud data, with a corresponding acquisition time of T. k-1 time; Figure 3b The blue pillar on the lower left is T. k The bird's-eye view features corresponding to the frame of historical point cloud data, with a corresponding acquisition time of T. k Timing. Based on the temporal information between two frames of point cloud data, it can be determined that the two need to be offset by a cylinder space during the stacking process. Therefore, after stacking the two in the time dimension, the 3D BEV feature can be obtained, which is... Figure 3b The stacking effect on the right side.

[0082] As an optional but non-limiting implementation, before using the backbone neural network module of the object detection model for feature extraction, the following is also included:

[0083] After stacking the various bird's-eye view features along the time dimension, the resulting 3DBEV features, which incorporate time information, can be supplemented with RGB feature information corresponding to each frame of historical point cloud data. The RGB feature information can be extracted by performing 2D convolution on the RGB images corresponding to each frame of historical point cloud data captured in actual shooting. After adding the corresponding RGB feature information, feature extraction is then performed using the backbone neural network module of the object detection model.

[0084] S205. A 3D target detection head based on a target detection network performs 3D target detection and pose relationship prediction on the 3D point cloud features of each frame of historical point cloud data.

[0085] The 3D object detection head includes a classification network and a regression network, enabling 3D object detection based on the 3D point cloud features of each frame of historical point cloud data. Specifically, it can detect the category and location of objects within each frame of historical point cloud data. The classification network obtains the category and confidence score of each object in each frame of historical point cloud data. The regression network obtains the position and size of the 3D bounding box of each object in each frame of historical point cloud data. The 3D bounding box can be represented by a seven-dimensional vector (x, y, z, w, l, h, θ), where x, y, and z represent the geometric center coordinates of the 3D bounding box; w, l, and h represent the width, length, and height of the 3D bounding box; and θ represents the heading angle of the 3D bounding box. It can also predict the pose relationship between adjacent frames of historical point cloud data.

[0086] S206. Based on the model prediction results and pre-labeling results output by the object detection model, determine the loss function of the object detection model.

[0087] Specifically, the model prediction results output by the object detection model include the category detection results and location detection results of each object contained in each frame of historical point cloud data, as well as the predicted pose relationship between two adjacent frames of historical point cloud data. Based on the model prediction results and pre-labeling results output by the object detection model, the loss function of the object detection model can be determined. Three loss functions are used during the training process of the object detection model: Focal Loss is used as the loss function for the classification network; Smooth L1 loss is used as the loss function for the regression network; and a loss function that uses the cosine similarity of matrices to construct the pose relationship is used. The pose relationship loss function L1... pose It can be the following formula:

[0088]

[0089] Among them, R gt The rotation matrix is ​​the optimal transformation matrix in the pre-annotated results; t gt R is the translation vector in the optimal transformation matrix of the pre-labeled results. pre The rotation matrix is ​​the optimal transformation matrix in the model prediction results; t pre is the translation vector in the optimal transformation matrix of the model prediction result; m is the number of frames of historical point cloud data input during the current target model training process. It should be noted that the number of frames of historical point cloud data input during the target model training process is not necessarily better the more it is; it needs to be determined based on the actual situation of model training. For example, it could be two consecutive frames or three consecutive frames.

[0090] S207. Training ends when the loss function meets the convergence condition, and the object detection model is obtained.

[0091] Specifically, the convergence condition can be that the values ​​of all loss functions decrease and tend to stabilize, or that the number of training iterations exceeds a set threshold. Training ends when the loss functions meet the convergence condition, thus obtaining the object detection model. Figure 3c As shown, Figure 3c This is a schematic diagram illustrating the training process of an object detection model provided in an embodiment of the present invention. Figure 3c The process involves inputting three consecutive frames of historical point cloud data. Finally, using a 3D object detection head, the system can output the confidence detection results (which determine the object's category), 3D bounding box detection results (which determine the object's position), and pose relationship prediction results between adjacent frames of point cloud data.

[0092] S208. Input at least two frames of target point cloud data into the target detection model, and output the pose detection result and the actual detection result corresponding to each of the at least two frames of target point cloud data through the target detection model. The at least two frames of target point cloud data are point cloud data acquired continuously in terms of acquisition time. The actual detection result corresponding to each frame of target point cloud data includes the candidate detection result of at least one candidate object contained in each frame of target point cloud data. The candidate detection result includes the category and position of the candidate object. The pose detection result is used to indicate the pose relationship between two frames of target point cloud data that are adjacent in terms of acquisition time in the at least two frames of target point cloud data.

[0093] S209. Based on the pose detection results and motion model, select at least two target detection results of the target object from each candidate detection result corresponding to the actual detection results of at least two frames of target point cloud data; the target object is determined by selecting at least one candidate object, and the at least two target detection results belong to the actual detection results corresponding to target point cloud data of different frames.

[0094] S210. Generate the trajectory of the target object based on at least two target detection results corresponding to the target object.

[0095] The technical solution provided by this invention involves inputting at least two frames of target point cloud data into a target detection model, which then outputs pose detection results and the corresponding actual detection results for each of the at least two frames of target point cloud data. Based on the pose detection results and the motion model, at least two target detection results for the target object are selected from the candidate detection results corresponding to the actual detection results of the at least two frames of target point cloud data. The target object is determined by selecting at least one candidate object, and the at least two target detection results belong to the actual detection results corresponding to target point cloud data from different frames. Based on the at least two target detection results corresponding to the target object, the trajectory of the target object is generated. Using the solution of this invention, relatively comprehensive and reliable target detection can be achieved, and accurate target trajectories can be generated. In particular, it can solve problems such as short-term occlusion and disappearance of targets, making the final generated target trajectory more comprehensive and accurate.

[0096] Figure 4 This is a flowchart illustrating another target trajectory generation method provided by an embodiment of the present invention. The technical solution of this embodiment further optimizes the process in the aforementioned embodiments of selecting at least two target detection results from candidate detection results corresponding to the actual detection results of at least two frames of target point cloud data, based on pose detection results and a motion model. This embodiment can be combined with various optional solutions in one or more of the above embodiments. For example... Figure 4 As shown, the target trajectory generation method of this invention embodiment may include the following process:

[0097] S401. Input at least two frames of target point cloud data into the target detection model, and output the pose detection result and the actual detection result corresponding to each of the at least two frames of target point cloud data through the target detection model. The at least two frames of target point cloud data are point cloud data acquired continuously in the acquisition time. The actual detection result corresponding to each frame of target point cloud data includes the candidate detection result of at least one candidate object contained in each frame of target point cloud data. The candidate detection result includes the category and position of the candidate object. The pose detection result is used to indicate the pose relationship between two adjacent frames of target point cloud data in the acquisition time. The target detection model is used to parse the at least two consecutive frames of point cloud data in the acquisition time and output the category detection result and position detection result of the object in each frame of point cloud data, as well as the pose relationship between two adjacent frames of point cloud data.

[0098] S402. Based on the actual detection results corresponding to at least two frames of target point cloud data, determine the first actual detection result and the second actual detection result, wherein the first actual detection result is the actual detection result corresponding to the current frame point cloud data, and the second actual detection result is the actual detection result corresponding to the next frame point cloud data; the current frame point cloud data and the next frame point cloud data are two adjacent frames of point cloud data, and the acquisition time corresponding to the current frame point cloud data is earlier than the acquisition time corresponding to the next frame point cloud data.

[0099] Specifically, the input to the target detection model consists of at least two consecutive frames of target point cloud data acquired in the acquisition time, and the output of the target detection model is the pose detection result and the actual detection result corresponding to each of the at least two frames of target point cloud data. From these actual detection results, a first actual detection result and a second actual detection result can be determined. Essentially, the first and second actual detection results are the actual detection results corresponding to two adjacent frames of target point cloud data; that is, the first actual detection result is the actual detection result corresponding to the current frame of point cloud data, and the second actual detection result is the actual detection result corresponding to the next frame of point cloud data. Furthermore, the acquisition time of the current frame of point cloud data is earlier than the acquisition time of the next frame of point cloud data.

[0100] For example, suppose that from the various actual detection results output by the object detection model, the first actual detection result is determined to be T. k The actual detection result corresponding to the frame target point cloud data, the second actual detection result is T. k+1 The actual detection result corresponding to the frame target point cloud data. Then T k Frame target point cloud data and T k+1 The target point cloud data in a frame consists of point cloud data from two adjacent frames, and T k The acquisition time corresponding to the target point cloud data in frame is earlier than T. k+1 The acquisition time corresponding to the target point cloud data in the frame.

[0101] S403. Determine the next moment corresponding to the next frame of point cloud data.

[0102] Once the next frame of point cloud data is determined, the next time step corresponding to that frame can be determined. For example, if T... k+1 If the target point cloud data in frame T is the point cloud data in the next frame, then the corresponding next time step is T. k+1 time.

[0103] S404. Based on the pose detection results, convert the current frame point cloud data and the next frame point cloud data to the same coordinate system.

[0104] Specifically, based on the pose detection results output by the target detection model, the coordinates of points in the current frame point cloud data and the next frame point cloud data can be transformed so that the two frames of point cloud data are transformed into the same coordinate system.

[0105] S405. Track each object in the current frame point cloud data, and predict the prediction result of each object at the next moment based on the state parameters corresponding to each object in the current frame point cloud data and the motion model; the prediction result includes the category prediction result and the position prediction result of each object at the next moment.

[0106] The motion model can predict the state of an object at the next moment by using its current state information. For example, the motion model can be a 3D Kalman filter model. The state parameters corresponding to each object in the current frame point cloud data can be determined based on the actual detection results of the current frame point cloud data, specifically based on the candidate detection results corresponding to each candidate object in the actual detection results. These state parameters include the category of the object, as well as the position and size of the 3D bounding box.

[0107] Specifically, each object in the current frame point cloud data can be tracked. Based on the state parameters of each object in the current frame point cloud data, a 3D Kalman filter model is used to predict the prediction result of each object at the next time step. The prediction result includes the category prediction result and the position prediction result of each object at the next time step. The category prediction result is the category corresponding to each object, and the position prediction result is the predicted position and size of the 3D bounding box corresponding to each object.

[0108] For example, to T k+1 The prediction of each target object at time T will be used as an example for illustration, assuming that T has already been determined. k The target point cloud data in the frame is the point cloud data of the current frame, and the adjacent T k+1 The target point cloud data in frame T is the point cloud data for the next frame, and T k T corresponding to the frame target point cloud data k The time is the current time, T k+1 T corresponding to the frame target point cloud data k+1 The timeframe is the next timeframe. Specifically, firstly, regarding the output of the object detection model about T... k The actual detection results corresponding to the frame target point cloud data can determine T. kThe frame contains the category, position, and size of the 3D bounding boxes for each object. For example, suppose we can obtain the position and size of the 3D bounding boxes for object e1, object f1, and object g1. Based on the position and size of the 3D bounding boxes for each object, we can determine the position of each of objects e1, f1, and g1 within the current time frame. k The state parameters corresponding to the given time. Therefore, during the prediction process using the 3D Kalman filter model, the prediction will be based on the state parameters of object e1, object f1, and object g1 at the current time T. k The state parameters corresponding to time T are used to predict the state of each of the three objects at time T. k+1 The state parameters corresponding to time T, which are the predicted values: object e1 at time T k+1 The position and size of the 3D bounding box at time T, and the position and size of object f1 at time T k+1 The position and size of the 3D bounding box at time T, and the position and size of object g1 at time T. k+1 The position and size of the 3D bounding box at any given time.

[0109] S406. Perform similarity matching between the predicted result and each candidate detection result corresponding to the second actual detection result to obtain the similarity matching result.

[0110] Specifically, the state parameters of each object in the obtained prediction results can be transformed into the same coordinate system as the current frame point cloud data and the next frame point cloud data. That is, this coordinate system now contains not only the actual detection results corresponding to the two frames of point cloud data output by the object detection model—specifically, the actual detection result corresponding to the current frame point cloud data (i.e., the first actual detection result) and the actual detection result corresponding to the next frame point cloud data (i.e., the second actual detection result); but also the prediction results corresponding to the objects in the current frame point cloud data obtained by using a 3D Kalman filter model at the next time step. It should be noted that the time corresponding to the prediction result and the next time step corresponding to the next frame point cloud data are essentially the same time step; for example, both correspond to time T. k+1 At that moment, the predicted result can be matched with each candidate detection result corresponding to the second actual detection result in this coordinate system based on similarity. Specifically, 3D Intersection over Union (IUI) can be used to calculate the intersection-union ratio first, and then the Hungarian algorithm can be used to match the target and the trajectory to obtain the similarity matching result.

[0111] S407. Based on the similarity matching results, associate the candidate detection results used to describe the same target object in the first actual detection result and the second actual detection result to filter out at least two target detection results used to describe the same target object.

[0112] Specifically, based on the similarity matching results, it can be determined which objects in the second actual detection result can match which objects in the prediction result. If they can match, it means that they are essentially describing the same object at the same time. Therefore, it can be determined that the object in the second actual detection result can correspond to a certain object in the first actual detection result. Then, the candidate detection results in the first and second actual detection results that describe the same target object can be associated to filter out at least two target detection results that describe the same target object.

[0113] For example, if a 3D Kalman filter model is used for T k After predicting the three objects (object e1, object f1, and object g1) contained in the frame target point cloud data, T can be obtained. k+1 The prediction results for each of these three objects at time T are: the prediction result for object e1, the prediction result for object f1, and the prediction result for object g1. This assumes that the object detection model has already output T. k+1 The frame target point cloud data contains candidate detection results for each candidate object. For example, we obtain candidate detection results for candidate object e2, candidate object f2, candidate object g2, and candidate object h2. Then, in the process of performing similarity matching between the predicted result and the candidate detection results corresponding to the second actual detection result, the essence is to determine which objects can be successfully matched between the "relevant objects of the predicted result (objects e1, f1, and g1)" and the "relevant objects of the second actual detection result (candidate objects e2, f2, g2, and h2)". Two objects that can be successfully matched indicate that they essentially describe the same object. For example, if after matching, it is determined that the predicted result of object e1 can match the candidate detection result corresponding to candidate object f2, then it means that object e1 and candidate object f2 essentially describe the same object in T. k+1 The same object at time T (e.g., a pedestrian). Then, since the prediction result for object e1 is derived from T... k The object e1 contained in the frame target point cloud data is predicted to be in T. k+1 Given the situation at a given time, then, by using this intermediate bridge, we can determine "T". k The objects "e1" and "T" contained in the frame target point cloud data k+1The candidate object f2" contained in the frame target point cloud data is essentially the same object (e.g., a pedestrian), which is the target object. Then, the candidate detection results describing the same target object in the first and second actual detection results can be correlated to filter out at least two target detection results describing the same target object. For example, filtering out T... k Frame target point cloud data and T k+1 In the frame target point cloud data, which parts describe the detection results of pedestrians, which parts describe the detection results of trees, and which parts describe the detection results of vehicles?

[0114] S408. Generate the trajectory of the target object based on at least two target detection results corresponding to the target object.

[0115] For example, T is determined through the preceding steps. k Frame target point cloud data and T k+1 Which of the target point cloud data in the frame describes the detection results of pedestrians, that is, at least two target detection results corresponding to the target object are obtained. Then, based on these at least two target detection results, the trajectory of the pedestrian can be generated.

[0116] As an optional but non-limiting implementation, after obtaining the similarity matching result, the target trajectory generation method further includes: if the similarity matching result exceeds the matching threshold, the matching is successful, and the motion model is updated based on the generated target object's trajectory; if the similarity matching result does not exceed the matching threshold, the matching is unsuccessful, and the actual detection result and prediction result are retained until the preset number of lifetimes are reached before being cleaned up.

[0117] Specifically, if the similarity matching result exceeds the matching threshold, the match is successful, and the motion model is updated based on the generated target object's trajectory. If the similarity matching result does not exceed the matching threshold, the match fails, and the actual detection result and prediction result are retained. For example, during the matching process, some objects in the prediction result may not match any candidate object in the actual detection result. Such objects may have been missed during the actual point cloud data acquisition process and have disappeared, thus preventing them from being matched in the prediction result. Similarly, some candidate objects in the second actual detection result may not match any of the related objects in the prediction result. Such candidate objects may be newly acquired objects in the actual acquisition scene. During the actual scene acquisition, the earlier frame of point cloud data naturally did not capture information about these newly appearing objects, resulting in the prediction result not including these new objects, and subsequent matching methods will not be able to match them. Objects that fail to match in this way can be retained in the actual detection result and prediction result until a preset lifespan is reached before being cleaned up. For example, after matching, if T is found... k In the frame target point cloud data, there is an object s that is completely unrelated to T. k+1 For any object in the frame target point cloud data, the detection results related to that object s can be retained, and then further analyzed with the Tth frame target point cloud data. k+2 Match and associate frame target point cloud data, or with T k+3 The frame target point cloud data is matched and associated. Once it can be matched with the object in the target point cloud data collected later, the trajectory of the object s can be generated. Although the relevant information of the object s is occluded at some point, the trajectory of the object s can still be generated in the end. This method can solve the problems of the target being occluded or disappearing for a short time, making the target detection results more stable and reliable, and generating the target trajectory at the same time.

[0118] The technical solution provided by this invention involves inputting at least two frames of target point cloud data into a target detection model, which then outputs pose detection results and the corresponding actual detection results for each of the at least two frames of target point cloud data. Based on the pose detection results and the motion model, at least two target detection results for the target object are selected from the candidate detection results corresponding to the actual detection results of the at least two frames of target point cloud data. The target object is determined by selecting at least one candidate object, and the at least two target detection results belong to the actual detection results corresponding to target point cloud data from different frames. Based on the at least two target detection results corresponding to the target object, the trajectory of the target object is generated. Using the solution of this invention, relatively comprehensive and reliable target detection can be achieved, and accurate target trajectories can be generated. In particular, it can solve problems such as short-term occlusion and disappearance of targets, making the final generated target trajectory more comprehensive and accurate.

[0119] Figure 5 This is a schematic diagram of a target trajectory generation device provided in an embodiment of the present invention. This embodiment is applicable to the construction and generation of trajectories for target objects. The target trajectory generation device can be implemented in software and / or hardware, and is generally integrated into any electronic device with network communication capabilities, such as a mobile terminal, PC, or server. Figure 5 As shown, the target trajectory generation device of this embodiment may include a detection module 510, a target detection result determination module 520, and a trajectory generation module 530. Wherein:

[0120] The detection module 510 is used to input at least two frames of target point cloud data into a target detection model, and output pose detection results and actual detection results corresponding to the at least two frames of target point cloud data through the target detection model. The at least two frames of target point cloud data are point cloud data acquired continuously in the acquisition time. The actual detection result corresponding to each frame of target point cloud data includes candidate detection results of at least one candidate object contained in each frame of target point cloud data. The candidate detection results include the category and position of the candidate object. The pose detection results are used to indicate the pose relationship between two adjacent frames of target point cloud data in the acquisition time. The target detection model is used to perform data parsing on at least two consecutive frames of point cloud data in the acquisition time, and outputs category detection results and position detection results of objects in each frame of point cloud data, as well as the pose relationship between two adjacent frames of point cloud data.

[0121] The target detection result determination module 520 is used to select at least two target detection results of a target object from each candidate detection result corresponding to the actual detection result of at least two frames of target point cloud data based on the pose detection result and the motion model; the target object is determined by selecting the at least one candidate object, and the at least two target detection results belong to the actual detection results corresponding to target point cloud data in different frames;

[0122] The trajectory generation module 530 is used to generate the trajectory of the target object based on the detection results of at least two targets corresponding to the target object.

[0123] As an optional but non-limiting implementation, the target trajectory generation device further includes: a target detection model training module, used to acquire at least two consecutive frames of historical point cloud data within a historical time period; determine the pre-labeling results corresponding to the at least two frames of historical point cloud data, the pre-labeling results including historical category detection results and historical position detection results corresponding to objects in each frame of historical point cloud data, as well as the historical pose relationship between two adjacent frames of historical point cloud data; input the at least two frames of historical point cloud data and the pre-labeling results into the target detection model; fuse the rasterized frames of historical point cloud data based on the temporal information corresponding to each frame of historical point cloud data through the target detection model to obtain 3D point cloud features of each frame of historical point cloud data; perform 3D target detection and pose relationship prediction on the 3D point cloud features of each frame of historical point cloud data based on the 3D target detection head of the target detection network; determine the loss function of the target detection model based on the model prediction results output by the target detection model and the pre-labeling results; and end training when the loss function satisfies the convergence condition to obtain the target detection model.

[0124] As an optional but non-limiting implementation, the target trajectory generation device further includes: a historical pose relationship determination module, used to determine the transformation matrix of corresponding points in two adjacent frames of historical point cloud data using the nearest neighbor method; update the two adjacent frames of historical point cloud data based on the transformation matrix to determine new corresponding points; perform iterative optimization of the transformation matrix based on the new corresponding points until a convergence condition is met to obtain the optimal transformation matrix; the convergence condition is that the error of corresponding points in two adjacent frames of historical point cloud data reaches a set threshold; and use the optimal transformation matrix as the historical pose relationship of the two adjacent frames of historical point cloud data.

[0125] As an optional but non-limiting implementation, the target trajectory generation device further includes: a point cloud data rasterization module, used to transform the historical point cloud data of each frame to the same coordinate system based on the historical pose relationship and the time deviation information between each frame of historical point cloud data, so as to align the historical point cloud data of each frame; to divide the historical point cloud data of each frame based on equally spaced cylindrical spaces, so that the points in the point cloud data are distributed in each cylindrical space; and to increase the dimensionality of the points in each frame of point cloud data belonging to the same cylindrical space based on the center point of each cylinder.

[0126] As an optional but non-limiting implementation, the object detection model training module is used to fuse the rasterized historical point cloud data of each frame based on the temporal information corresponding to each frame of historical point cloud data by the object detection model, to obtain the 3D point cloud features of each frame of historical point cloud data. Specifically, this includes: processing the rasterized historical point cloud data of each frame based on a sparse convolutional neural network by the object detection model to obtain the bird's-eye view features of each frame of historical point cloud data from a bird's-eye view perspective; and stacking the bird's-eye view features of each frame of historical point cloud data in the time dimension based on the temporal information corresponding to each frame of historical point cloud data by the object detection model to obtain the 3D point cloud features of each frame of historical point cloud data.

[0127] As an optional but non-limiting implementation, the target detection result determination module 520 includes an actual detection result determination unit, a next moment determination unit, a transformation unit, a tracking prediction unit, a matching unit, and an association unit. Specifically: the actual detection result determination unit is used to determine a first actual detection result and a second actual detection result based on the actual detection results corresponding to the at least two frames of target point cloud data, wherein the first actual detection result is the actual detection result corresponding to the current frame of point cloud data, and the second actual detection result is the actual detection result corresponding to the next frame of point cloud data; the current frame of point cloud data and the next frame of point cloud data are two adjacent frames of point cloud data, and the acquisition time corresponding to the current frame of point cloud data is earlier than the acquisition time corresponding to the next frame of point cloud data; the next moment determination unit is used to determine the next moment corresponding to the next frame of point cloud data; the transformation unit is used to transform the current frame of point cloud data and the next frame of point cloud data to the same coordinate system based on the pose detection result. The system includes: a tracking and prediction unit, used to track each object in the current frame point cloud data, and predict the prediction result of each object at the next time step based on the motion model according to the state parameters corresponding to each object in the current frame point cloud data; the prediction result includes the category prediction result and the position prediction result corresponding to each object at the next time step; a matching unit, used to perform similarity matching between the prediction result and each candidate detection result corresponding to the second actual detection result to obtain a similarity matching result; and an association unit, used to associate the candidate detection results in the first actual detection result and the second actual detection result that describe the same target object based on the similarity matching result, so as to filter out at least two target detection results that describe the same target object.

[0128] As an optional but non-limiting implementation, the target trajectory generation device further includes: a lifecycle management unit, used to: if the similarity matching result exceeds the matching threshold, then the matching is successful, and the motion model is updated based on the generated target object's trajectory; if the similarity matching result does not exceed the matching threshold, then the matching is unsuccessful, and the actual detection result and the prediction result are retained until the lifecycle reaches a preset number of times before being cleaned up.

[0129] The target trajectory generation device provided in this embodiment of the invention can be used to execute the target trajectory generation method, and has the corresponding functional modules and beneficial effects for executing the target trajectory generation method.

[0130] It is worth noting that the various units and modules included in the above-mentioned device are only divided according to functional logic, but are not limited to the above division, as long as the corresponding functions can be realized; in addition, the specific names of each functional unit are only for easy differentiation and are not used to limit the protection scope of the embodiments of the present invention.

[0131] Figure 6 This is a schematic diagram of an electronic device for implementing a target trajectory generation method according to an embodiment of the present invention. The following refers to... Figure 6 The diagram illustrates a structural schematic of an electronic device 610 suitable for implementing embodiments of the present invention. The terminal devices in these embodiments may include, but are not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), in-vehicle terminals (e.g., in-vehicle navigation terminals), and fixed terminals such as digital TVs and desktop computers. Figure 6 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments of the present invention.

[0132] like Figure 6 As shown, the electronic device 610 includes at least one processor 611 and a memory, such as a read-only memory (ROM) 612 or a random access memory (RAM) 613, communicatively connected to the at least one processor 611. The memory stores computer programs executable by the at least one processor. The processor 611 can perform various appropriate actions and processes based on the computer program stored in the ROM 612 or loaded from storage unit 618 into the RAM 613. The RAM 613 may also store various programs and data required for the operation of the electronic device 610. The processor 611, ROM 612, and RAM 613 are interconnected via a bus 614. An input / output (I / O) interface 615 is also connected to the bus 614.

[0133] Multiple components in electronic device 610 are connected to I / O interface 615, including: input unit 616, such as keyboard, mouse, etc.; output unit 617, such as various types of displays, speakers, etc.; storage unit 618, such as disk, optical disk, etc.; and communication unit 619, such as network card, modem, wireless transceiver, etc. Communication unit 619 allows electronic device 610 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0134] Processor 611 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of processor 611 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. Processor 611 executes the target trajectory generation method provided in any embodiment of the present invention.

[0135] In particular, according to embodiments of the present invention, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of the present invention include a computer program product comprising a computer program carried on a non-transitory computer-readable medium, the computer program containing program code for performing the target trajectory generation method shown in the flowcharts. When executed by a processing device, the computer program performs the functions defined in the target trajectory generation method of the embodiments of the present invention.

[0136] The names of the messages or information exchanged between the multiple devices in the embodiments of the present invention are for illustrative purposes only and are not intended to limit the scope of these messages or information.

[0137] The electronic device provided in this embodiment of the invention and the target trajectory generation method provided in the above embodiments belong to the same inventive concept. Technical details not described in detail in this embodiment can be found in the above embodiments, and this embodiment has the same beneficial effects as the above embodiments.

[0138] This invention provides a computer storage medium storing a computer program that, when executed by a processor, implements the target trajectory generation method provided in the above embodiments.

[0139] It should be noted that the computer-readable medium described above in this invention can be a computer-readable signal medium, a computer-readable storage medium, or any combination thereof. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this invention, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In this invention, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium can be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wires, optical fibers, RF (radio frequency), etc., or any suitable combination thereof.

[0140] Computer program code for performing the operations of this invention can be written in one or more programming languages ​​or a combination thereof, including but not limited to object-oriented programming languages ​​such as Java, Smalltank, and C++, as well as conventional procedural programming languages ​​such as "C" or similar languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0141] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0142] The units described in the embodiments of this invention can be implemented in software or hardware. The names of the units are not necessarily limiting in certain circumstances. The functions described above can be performed at least in part by one or more hardware logic components. For example, exemplary types of hardware logic components that can be used, without limitation, include: Field Programmable Gate Arrays (FPGAs), Application-Specific Integrated Circuits (ASICs), Application Standard Products (ASSPs), System-on-Chip (SoCs), Complex Programmable Logic Devices (CPLDs), etc.

[0143] In the context of this invention, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. Machine-readable media can include, but are not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0144] The above description is merely a preferred embodiment of the present invention and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of disclosure in this invention is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-described concept. For example, technical solutions formed by substituting the above features with (but not limited to) technical features with similar functions disclosed in this invention.

[0145] Furthermore, while the operations are described in a specific order, this should not be construed as requiring these operations to be performed in the specific order shown or in sequential order. In certain circumstances, multitasking and parallel processing may be advantageous. Similarly, while several specific implementation details are included in the above discussion, these should not be construed as limiting the scope of the invention. Certain features described in the context of individual embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented individually or in any suitable sub-combination in multiple embodiments.

[0146] Although the subject matter has been described using language specific to structural features and / or methodological logic, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. Rather, the specific features and actions described above are merely illustrative examples of implementing the claims.

Claims

1. A method for generating a target trajectory, characterized in that, The method includes: At least two frames of target point cloud data are input into a target detection model. The target detection model outputs pose detection results and actual detection results corresponding to the at least two frames of target point cloud data. The at least two frames of target point cloud data are point cloud data acquired continuously in terms of acquisition time. The actual detection result corresponding to each frame of target point cloud data includes candidate detection results of at least one candidate object contained in each frame of target point cloud data. The candidate detection results include the category and position of the candidate object. The pose detection results are used to indicate the pose relationship between two adjacent frames of target point cloud data in terms of acquisition time. The target detection model is used to parse the at least two frames of point cloud data that are consecutive in terms of acquisition time and outputs category detection results and position detection results of objects in each frame of point cloud data, as well as the pose relationship between two adjacent frames of point cloud data. Based on the pose detection results and motion model, at least two target detection results of the target object are selected from each candidate detection result corresponding to the actual detection results of at least two frames of target point cloud data; the target object is determined by selecting the at least one candidate object, and the at least two target detection results belong to the actual detection results corresponding to target point cloud data in different frames; Based on the detection results of at least two targets corresponding to the target object, the trajectory of the target object is generated.

2. The method according to claim 1, characterized in that, The training process of the target detection model includes: Acquire at least two consecutive frames of historical point cloud data within a historical time period; Determine the pre-labeling results corresponding to the at least two frames of historical point cloud data. The pre-labeling results include the historical category detection results and historical position detection results of the objects in each frame of historical point cloud data, as well as the historical pose relationship between two adjacent frames of historical point cloud data. The at least two frames of historical point cloud data and the pre-labeling results are input into the target detection model; The target detection model fuses the rasterized historical point cloud data of each frame based on the temporal information corresponding to each frame of historical point cloud data to obtain the 3D point cloud features of each frame of historical point cloud data. Based on the target detection model, the 3D target detection head performs 3D target detection and pose relationship prediction on the 3D point cloud features of each frame of historical point cloud data. Based on the model prediction results output by the target detection model and the pre-labeling results, the loss function of the target detection model is determined; Training ends when the loss function meets the convergence condition, thus obtaining the target detection model.

3. The method according to claim 2, characterized in that, The process of determining the historical pose relationship between two adjacent frames of historical point cloud data includes: The transformation matrix of corresponding points in two adjacent frames of historical point cloud data is determined by the nearest neighbor method. The historical point cloud data of two adjacent frames are updated based on the transformation matrix to determine the new corresponding points; Based on the new corresponding points, the transformation matrix is ​​iteratively optimized until the convergence condition is met to obtain the optimal transformation matrix; the convergence condition is that the error of the corresponding points in two adjacent frames of historical point cloud data reaches a set threshold. The optimal transformation matrix is ​​used as the historical pose relationship between two adjacent frames of historical point cloud data.

4. The method according to claim 3, characterized in that, The process of rasterizing each frame of historical point cloud data includes: Based on the historical pose relationship and the time deviation information between each frame of historical point cloud data, each frame of historical point cloud data is transformed into the same coordinate system to align the historical point cloud data. The historical point cloud data of each frame is divided into cylindrical spaces with equal spacing so that the points in the point cloud data are distributed in each cylindrical space. Dimensionality is increased for each frame of point cloud data corresponding to points belonging to the same column space based on the center point of each column.

5. The method according to claim 2, characterized in that, The step involves fusing the rasterized historical point cloud data from each frame based on the temporal information corresponding to each frame of historical point cloud data using the target detection model, to obtain the 3D point cloud features of each frame of historical point cloud data, including: By processing the rasterized historical point cloud data of each frame using a target detection model based on a sparse convolutional neural network, the bird's-eye view features of each frame of historical point cloud data under the bird's-eye view are obtained. Based on the temporal information corresponding to each frame of historical point cloud data, the target detection model stacks the bird's-eye view features corresponding to each frame of historical point cloud data in the time dimension to obtain the 3D point cloud features of each frame of historical point cloud data.

6. The method according to claim 1, characterized in that, Based on the pose detection results and motion model, at least two target detection results of the target object are selected from each candidate detection result corresponding to the actual detection results of at least two frames of target point cloud data, including: A first actual detection result and a second actual detection result are determined based on the actual detection results corresponding to each of the at least two frames of target point cloud data. The first actual detection result is the actual detection result corresponding to the current frame of point cloud data, and the second actual detection result is the actual detection result corresponding to the next frame of point cloud data. The current frame of point cloud data and the next frame of point cloud data are two adjacent frames of point cloud data, and the acquisition time of the current frame of point cloud data is earlier than the acquisition time of the next frame of point cloud data. Determine the next moment corresponding to the next frame of point cloud data; Based on the pose detection results, the current frame point cloud data and the next frame point cloud data are transformed to the same coordinate system; Each object in the current frame point cloud data is tracked, and based on the state parameters corresponding to each object in the current frame point cloud data, the prediction result of each object at the next time step is predicted based on the motion model; the prediction result includes the category prediction result and the position prediction result of each object at the next time step. The prediction result is matched with each candidate detection result corresponding to the second actual detection result to obtain the similarity matching result; Based on the similarity matching results, the candidate detection results used to describe the same target object in the first actual detection result and the second actual detection result are associated to filter out at least two target detection results used to describe the same target object.

7. The method according to claim 6, characterized in that, After obtaining the similarity matching result, the method further includes: If the similarity matching result exceeds the matching threshold, the matching is successful, and the motion model is updated based on the trajectory of the generated target object. If the similarity matching result does not exceed the matching threshold, the matching is unsuccessful, and the actual detection result and the prediction result are retained until the preset number of lifetimes are reached before being cleaned up.

8. A target trajectory generation device, characterized in that, The device includes: A detection module is used to input at least two frames of target point cloud data into a target detection model, and output pose detection results and actual detection results corresponding to the at least two frames of target point cloud data through the target detection model. The at least two frames of target point cloud data are point cloud data acquired continuously in the acquisition time. The actual detection result corresponding to each frame of target point cloud data includes candidate detection results of at least one candidate object contained in each frame of target point cloud data. The candidate detection results include the category and position of the candidate object. The pose detection results are used to indicate the pose relationship between two adjacent frames of target point cloud data in the acquisition time. The target detection model is used to parse the at least two frames of point cloud data that are consecutive in the acquisition time, and output the category detection results and position detection results of objects in each frame of point cloud data, as well as the pose relationship between two adjacent frames of point cloud data. The target detection result determination module is used to select at least two target detection results of a target object from each candidate detection result corresponding to the actual detection result of at least two frames of target point cloud data, based on the pose detection result and the motion model; the target object is determined by selecting the at least one candidate object, and the at least two target detection results belong to the actual detection results corresponding to target point cloud data in different frames; The trajectory generation module is used to generate the trajectory of the target object based on the detection results of at least two targets corresponding to the target object.

9. An electronic device, characterized in that, The electronic device includes: One or more processors; Storage device for storing one or more programs. When the one or more programs are executed by the one or more processors, the one or more processors implement the target trajectory generation method as described in any one of claims 1-7.

10. A storage medium containing computer-executable instructions, characterized in that, The computer-executable instructions, when executed by a computer processor, are used to perform the target trajectory generation method as described in any one of claims 1-7.

Images