Target trajectory generation method and device, electronic equipment and storage medium

Abstract

The invention discloses a target trajectory generation method and device, electronic equipment and a storage medium, and the method comprises the steps: inputting at least two frames of target point cloud data into a target detection model, and outputting a pose detection result and actual detection results corresponding to the at least two frames of target point cloud data through the target detection model; screening out at least two target detection results of the target object from candidate detection results corresponding to actual detection results of the at least two frames of target point cloud data based on the pose detection result and the motion model; the target object is determined by screening at least one candidate object, and the at least two target detection results belong to actual detection results corresponding to the target point cloud data of different frames; and generating a track of the target object based on the at least two target detection results corresponding to the target object. According to the scheme, relatively comprehensive and reliable target detection can be realized, and an accurate target track is generated.

Description

Technical Field

[0001] The present invention relates to the field of autonomous driving technology, and in particular to a target trajectory generation method, device, electronic device and storage medium. Background Art

[0002] With the development of autonomous driving technology, the perception and understanding of the environment by vehicles has become particularly important. Target trajectory generation is a key link in constructing traffic scenes. It is responsible for obtaining the target's driving path based on the vehicle's current position, target position, and surrounding environment information. However, in 2D image perception, depth information is lacking; in 3D point cloud perception, target information is discrete and disordered in time series. At the same time, the target detection results will be unstable and occluded targets are difficult to detect, which directly affects the prediction of target trajectory. It can be said that 3D target detection algorithm is an important factor affecting the safety of autonomous driving systems. At present, 3D target detection schemes for point clouds are mainly divided into point-based and grid-based methods. Point-based methods have a bottleneck in inference time. Since point clouds are directly processed, point cloud sampling is required, which has high computational cost. At the same time, it is easy to cause 3D fine-grained information loss in feature learning. The grid-based method will inevitably lead to the loss of 3D information with the conversion of coordinates. Its effect depends largely on the size of the grid unit. This method is difficult to balance the efficacy brought by the smaller grid size and the efficiency affected by the increase in memory. Summary of the invention

[0003] The present invention provides a target trajectory generation method, device, electronic device and storage medium, which can achieve relatively comprehensive and reliable target detection and generate accurate target trajectory.

[0004] In a first aspect, the present invention provides a method for generating a target trajectory, comprising:

[0005] Input at least two frames of target point cloud data into a target detection model, and output a pose detection result and an actual detection result corresponding to each of the at least two frames of target point cloud data through the target detection model, wherein the at least two frames of target point cloud data are point cloud data collected continuously in acquisition time, and the actual detection result corresponding to each frame of the target point cloud data includes a candidate detection result of at least one candidate object contained in each frame of the target point cloud data, and the candidate detection result includes a category and a position of the candidate object, and the pose detection result is used to indicate a pose relationship between two frames of target point cloud data that are adjacent in acquisition time in at least two frames of target point cloud data, and the target detection model is used to perform data analysis on at least two frames of point cloud data that are continuous in acquisition time, and output a category detection result and a position detection result of an object in each frame of point cloud data and a pose relationship between two adjacent frames of point cloud data;

[0006] Based on the pose detection result and the motion model, at least two target detection results of the target object are screened out from each candidate detection result corresponding to the actual detection result of at least two frames of target point cloud data; the target object is screened and determined for the at least one candidate object, and the at least two target detection results belong to the actual detection results corresponding to the target point cloud data of different frames;

[0007] A trajectory of the target object is generated based on the at least two target detection results corresponding to the target object.

[0008] In a second aspect, the present invention further provides a target trajectory generating device, comprising:

[0009] A detection module, used for inputting at least two frames of target point cloud data into a target detection model, and outputting a pose detection result and an actual detection result corresponding to each of the at least two frames of target point cloud data through the target detection model, wherein the at least two frames of target point cloud data are point cloud data collected continuously in acquisition time, and the actual detection result corresponding to each frame of the target point cloud data includes a candidate detection result of at least one candidate object contained in each frame of the target point cloud data, and the candidate detection result includes a category and a position of the candidate object, and the pose detection result is used to indicate a pose relationship between two frames of target point cloud data that are adjacent in acquisition time in at least two frames of target point cloud data, and the target detection model is used to perform data analysis on at least two frames of point cloud data that are continuous in acquisition time, and output a category detection result and a position detection result of an object in each frame of point cloud data and a pose relationship between two adjacent frames of point cloud data;

[0010] A target detection result determination module is used to screen out at least two target detection results of a target object from each candidate detection result corresponding to the actual detection result of at least two frames of target point cloud data based on the pose detection result and the motion model; the target object is the actual detection result corresponding to the target point cloud data of different frames after screening and determination of the at least one candidate object;

[0011] The trajectory generation module is used to generate a trajectory of the target object based on the at least two target detection results corresponding to the target object.

[0012] In a third aspect, an embodiment of the present invention further provides an electronic device, comprising: one or more processors; a storage device for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement a target trajectory generation method as provided in any embodiment of the present invention.

[0013] In a fourth aspect, an embodiment of the present invention further provides a storage medium comprising computer executable instructions, wherein the computer executable instructions, when executed by a computer processor, are used to execute a target trajectory generation method as provided in any embodiment of the present invention.

[0014] The technical solution provided by the present invention inputs at least two frames of target point cloud data into a target detection model, and outputs a pose detection result and an actual detection result corresponding to each of the at least two frames of target point cloud data through the target detection model, wherein the at least two frames of target point cloud data are point cloud data collected continuously in acquisition time, and the actual detection result corresponding to each frame of target point cloud data includes a candidate detection result of at least one candidate object contained in each frame of target point cloud data, and the candidate detection result includes a category and a position of the candidate object, and the pose detection result is used to indicate the pose relationship between two frames of target point cloud data that are adjacent in acquisition time in at least two frames of target point cloud data; based on the pose detection result and the motion model, at least two target detection results of the target object are screened out from each candidate detection result corresponding to the actual detection result of at least two frames of target point cloud data; the target object is to screen and determine at least one candidate object, and at least two target detection results belong to the actual detection results corresponding to target point cloud data of different frames; based on the at least two target detection results corresponding to the target object, a trajectory of the target object is generated. By adopting the solution of the present invention, relatively comprehensive and reliable target detection can be achieved, and accurate target trajectories can be generated. In particular, problems such as short-term occlusion and disappearance of the target can be solved, so that the target trajectory finally generated is more comprehensive and accurate.

[0015] The above invention content is only an overview of the technical solution of the present invention. In order to more clearly understand the technical means of the present invention, it can be implemented according to the contents of the specification. In order to make the above and other purposes, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] The above and other features, advantages and aspects of the embodiments of the present invention will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. Throughout the accompanying drawings, the same or similar reference numerals represent the same or similar elements. It should be understood that the drawings are schematic and the originals and elements are not necessarily drawn to scale.

[0017] Figure 1 It is a flow chart of a method for generating a target trajectory provided by an embodiment of the present invention;

[0018] Figure 2 is a flow chart of another target trajectory generation method provided by an embodiment of the present invention;

[0019] Figure 3aA schematic diagram of an equidistant cylindrical space provided by an embodiment of the present invention;

[0020] Figure 3b A schematic diagram of the effect of stacking bird's-eye view features corresponding to point cloud data provided by an embodiment of the present invention;

[0021] Figure 3c A schematic diagram of a training process of a target detection model provided by an embodiment of the present invention;

[0022] Figure 4 A schematic diagram of a flow chart of another target trajectory generation method provided by an embodiment of the present invention;

[0023] Figure 5 It is a structural schematic diagram of a target trajectory generating device provided by an embodiment of the present invention;

[0024] Figure 6 It is a structural schematic diagram of an electronic device for implementing a target trajectory generation method provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0025] Embodiments of the present invention will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present invention are shown in the accompanying drawings, it should be understood that the present invention can be implemented in various forms and should not be construed as being limited to the embodiments described herein, but rather these embodiments are provided for a more thorough and complete understanding of the present invention. It should be understood that the drawings and embodiments of the present invention are only for exemplary purposes and are not intended to limit the scope of protection of the present invention. It should be understood that the various steps described in the method implementation of the present invention can be performed in different orders and / or performed in parallel. In addition, the method implementation may include additional steps and / or omit the steps shown. The scope of the present invention is not limited in this respect.

[0026] The term "including" and its variations used herein are open inclusions, i.e., "including but not limited to". The term "based on" means "based at least in part on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". The relevant definitions of other terms will be given in the following description.

[0027] It should be noted that the concepts of "first", "second", etc. mentioned in the present invention are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units. It should be noted that the modifications of "one" and "multiple" mentioned in the present invention are illustrative rather than restrictive. Those skilled in the art should understand that unless otherwise clearly indicated in the context, it should be understood as "one or more". The names of the messages or information exchanged between multiple devices in the embodiments of the present invention are for illustrative purposes only and are not used to limit the scope of these messages or information.

[0028] Figure 1 The present invention provides a flow chart of a method for generating a target trajectory. The present invention is applicable to the case where the trajectory of a target object is constructed and generated. The method can be executed by a target trajectory generating device. The target trajectory generating device can be implemented in the form of software and / or hardware and is generally integrated in any electronic device with network communication function. The electronic device can be a mobile terminal, a PC or a server, etc. Figure 1 As shown, the target trajectory generation method of the embodiment of the present invention may include the following process:

[0029] S101, input at least two frames of target point cloud data into a target detection model, and output the pose detection results and the actual detection results corresponding to the at least two frames of target point cloud data through the target detection model. Wherein, the at least two frames of target point cloud data are point cloud data collected continuously in the acquisition time, and the actual detection results corresponding to each frame of target point cloud data include the candidate detection results of at least one candidate object contained in each frame of target point cloud data, and the candidate detection results include the category and position of the candidate object. The pose detection result is used to indicate the pose relationship between two frames of target point cloud data adjacent to each other in the acquisition time in at least two frames of target point cloud data. The target detection model is used to perform data analysis on at least two frames of point cloud data that are continuous in the acquisition time, and output the category detection results and position detection results of the objects in each frame of point cloud data and the pose relationship between two adjacent frames of point cloud data.

[0030] Point cloud data refers to a collection of points in three-dimensional space. A frame of point cloud data contains the three-dimensional coordinate information and reflection intensity information of multiple different signal reflection points. Different reflection points constitute different objects, such as trees, cars, pedestrians, etc. At the same time, each frame of point cloud data will also have its own corresponding collection time. For example, the time information when the vehicle-mounted laser radar collects a frame of point cloud data is the collection time corresponding to the frame of point cloud data.

[0031] At least two frames of target point cloud data input to the target detection model are point cloud data collected continuously in terms of collection time. The continuous collection time may mean that the collection time corresponding to each frame of point cloud data is in an increasing state. For example, if point cloud data is collected every 1 second, and three frames of target point cloud data are input to the target detection model, then in terms of the collection time corresponding to the three frames of target point cloud data input, the first frame is 1 second earlier than the second frame, and the second frame is 1 second earlier than the third frame.

[0032] The target detection model can be used to parse data of at least two frames of point cloud data that are continuous in acquisition time, and the output includes the category detection results and position detection results of the object in each frame of point cloud data and the pose relationship between two adjacent frames of point cloud data. Among them, the category detection result of the object is used to reflect the category to which the detected object belongs, and the position detection result of the object is used to reflect the position information of the detected object in the point cloud data of the frame to which it belongs. The pose relationship can be used to reflect the position change relationship between corresponding points in two adjacent frames of point cloud data. Specifically, because the point cloud data is continuous in acquisition time, there must be a certain position change relationship between the corresponding points corresponding to the same object in the two frames of point cloud data obtained, and this position change relationship can be described using the pose relationship.

[0033] Specifically, at least two frames of target point cloud data can be input into the target detection model, and the target detection model outputs the pose detection results and the actual detection results corresponding to the at least two frames of target point cloud data. The pose detection result can indicate the pose relationship between two adjacent frames of point cloud data in the at least two frames of target point cloud data input into the target detection model. The actual detection results corresponding to each frame of target point cloud data include the candidate detection results of at least one candidate object contained in each frame of target point cloud data, and the candidate detection results include the category and position of the candidate object.

[0034] Exemplarily, taking the input of two frames of target point cloud data into the target detection model for processing as an example, the target detection model can output the pose detection results between the two frames of point cloud data, the actual detection results corresponding to the first frame of target point cloud data, and the actual detection results corresponding to the second frame of target point cloud data. Among them, the actual detection results corresponding to the first frame of target point cloud data include the candidate detection results of each candidate object contained in the first frame of target point cloud data. Assuming that there are three candidate objects, then the actual detection results corresponding to the first frame of target point cloud data include the candidate detection results corresponding to candidate object a1, the candidate detection results corresponding to candidate object a2, and the candidate detection results corresponding to candidate object a3; and the candidate detection results corresponding to each candidate object include the category and position of the candidate object. By analogy, the actual detection results corresponding to the second frame of target point cloud data also contain the same type of specific related information, which will not be described in detail here.

[0035] It should be noted that the pose detection result output by the target detection model indicates the pose relationship between two frames of target point cloud data that are adjacent in acquisition time in at least two frames of target point cloud data. If the input target point cloud data has only two frames, then there is one pose detection result output, which is the pose detection result between the two frames of target point cloud data. If the input target point cloud data has three frames, then there are two pose detection results output, one is the pose detection result between the first and second frames of target point cloud data, and the other is the pose detection result between the second and third frames of target point cloud data. By analogy, the same principle content will not be repeated here.

[0036] S102. Based on the posture detection results and the motion model, at least two target detection results of the target object are screened out from the candidate detection results corresponding to the actual detection results of at least two frames of target point cloud data; the target object is determined by screening at least one candidate object, and at least two target detection results belong to the actual detection results corresponding to the target point cloud data of different frames.

[0037] Since the actual detection results corresponding to each frame of target point cloud data output by the target detection model all contain candidate detection results of each candidate object in the point cloud data of this frame, and the candidate objects between different frames may essentially belong to the same object, it is possible to determine the relevant candidate detection results belonging to the same object from the candidate detection results corresponding to each frame of point cloud data, and use them as the target detection results corresponding to the same object.

[0038] Specifically, at least two target detection results of the target object can be screened out from each candidate detection result corresponding to the actual detection result of at least two frames of target point cloud data based on the pose detection result and the motion model. The target object is the actual detection result corresponding to the target point cloud data of different frames after screening and determination of at least one candidate object, and the at least two target detection results determined belong to.

[0039] Exemplarily, taking the actual detection results corresponding to two frames of target point cloud data as an example, it is assumed that the actual detection results corresponding to the first frame of target point cloud data include three candidate objects, and the actual detection results corresponding to the second frame of target point cloud data include four candidate objects, that is: the actual detection results corresponding to the first frame of target point cloud data include candidate detection results corresponding to candidate object a1, candidate detection results corresponding to candidate object a2, and candidate detection results corresponding to candidate object a3; the actual detection results corresponding to the second frame of target point cloud data include candidate detection results corresponding to candidate object b1, candidate detection results corresponding to candidate object b2, candidate detection results corresponding to candidate object b3, and candidate detection results corresponding to candidate object b4. Then, for each candidate object corresponding to these different frames, there are different candidate objects that are essentially the same object. For example, based on the pose detection results and the motion model, it is determined that candidate object a1 and candidate object b2 are essentially the same object (specifically, it can be a tree), and candidate object a2 and candidate object b3 are essentially the same object (specifically, it can be a pedestrian). Correspondingly, the candidate detection results corresponding to candidate object a1 and the candidate detection results corresponding to candidate object b2 are a set of target detection results for the "tree" object; the candidate detection results corresponding to candidate object a2 and the candidate detection results corresponding to candidate object b3 are another set of target detection results for the "pedestrian" object. By analogy, the target detection results belonging to the same object can be determined from the candidate detection results corresponding to each frame of point cloud data.

[0040] S103: Generate a trajectory of the target object based on at least two target detection results corresponding to the target object.

[0041] Since at least two target detection results corresponding to the determined target object are candidate detection results corresponding to point cloud data of different frames, that is to say, each target detection result obtained reflects the category information and position information of the target object in the point cloud data of the frame to which it belongs, and the point cloud data of different frames themselves have different acquisition times, the trajectory of the target object can be generated based on the position information of the target object in at least two target detection results and the acquisition time of the point cloud data associated with at least two target detection results.

[0042] The technical solution provided by the present invention inputs at least two frames of target point cloud data into a target detection model, and outputs the pose detection results and the actual detection results corresponding to the at least two frames of target point cloud data through the target detection model; based on the pose detection results and the motion model, at least two target detection results of the target object are screened out from the candidate detection results corresponding to the actual detection results of at least two frames of target point cloud data; the target object is to screen and determine at least one candidate object, and at least two target detection results belong to the actual detection results corresponding to the target point cloud data of different frames; based on the at least two target detection results corresponding to the target object, the trajectory of the target object is generated. The solution of the present invention can achieve more comprehensive and reliable target detection and generate accurate target trajectories, especially can solve the problems of short-term occlusion and disappearance of the target, so that the target trajectory finally generated is more comprehensive and accurate.

[0043] Figure 2 This is a flow chart of another method for generating target trajectories provided by an embodiment of the present invention. The technical solution of this embodiment further optimizes the training process of the target detection model in the above embodiment on the basis of the technical solution of the above embodiment. This embodiment can be combined with various optional solutions in one or more of the above embodiments. Figure 2 As shown, the target trajectory generation method of the embodiment of the present invention may include the following process:

[0044] S201. Acquire at least two consecutive frames of historical point cloud data within a historical time period.

[0045] Specifically, the vehicle-mounted laser radar can be used to collect at least two consecutive frames of historical point cloud data in different scenes within a historical time period. For example, the acquired historical point cloud data can be expressed as (x, y, z, r, timestamp), wherein x, y, z respectively reflect the coordinate values ​​of the signal reflection point contained in the point cloud data in the three-dimensional space coordinate system, r is the reflection intensity of the signal reflection point, and timestamp is the timestamp of signal collection, which is essentially the collection time corresponding to the frame of historical point cloud data.

[0046] S202, determining pre-annotation results corresponding to at least two frames of historical point cloud data, the pre-annotation results including historical category detection results and historical position detection results corresponding to the objects in each frame of historical point cloud data and a historical posture relationship between two adjacent frames of historical point cloud data.

[0047] Specifically, for each frame of historical point cloud data obtained, the category result and location information of each object contained in the historical point cloud data can be annotated, and the historical category detection result and historical location detection result corresponding to each object in each frame of historical point cloud data can be obtained. At the same time, the historical posture relationship between two adjacent frames of historical point cloud data in at least two frames of historical point cloud data is also annotated. And all these annotation information are used together as the pre-annotation results corresponding to at least two frames of historical point cloud data.

[0048] As an optional but non-limiting implementation, the process of determining the historical pose relationship between two adjacent frames of historical point cloud data may include the following steps A1-A4:

[0049] Step A1: Determine the transformation matrix of corresponding points in two adjacent frames of historical point cloud data by the nearest neighbor method.

[0050] Since the acquisition time of two adjacent frames of historical point cloud data is continuous, the transformation matrices between adjacent historical point cloud data are similar. Specifically, the random sampling method can be used to pre-process the acquired at least two frames of historical point cloud data to reduce the relevant signal reflection points of unnecessary objects in the historical point cloud data. For example, it can be to remove the signal reflection points related to the road surface in the historical point cloud data, thereby reducing the number of signal reflection points in the historical point cloud data and reducing the calculation amount of the solution.

[0051] Then the corresponding points P = {p1, p2, ..., p n} and Q={q1,q2,...,q n}, where p1, p2, ..., p n are points in the historical point cloud data of the earlier acquisition time, q1, q2, ..., q n is the point in the historical point cloud data of the frame with the later acquisition time. Point p1 and point q1 are a set of corresponding points, point p2 and point q2 are a set of corresponding points, ..., point p n and dot q n is a set of corresponding points. For these different groups of corresponding points, the transformation matrix (R, t) can be determined based on the following formula:

[0052] q i =Rp i +t;

[0053] Among them, R is the rotation matrix; t is the translation vector between two corresponding points; i is the subscript of each group of corresponding points, that is, i∈[1,n].

[0054] It should be noted that the corresponding points of each group determined by the nearest neighbor method can be considered as a rough correspondence, which is not precise enough. Therefore, further processing can be performed in subsequent solutions to determine more precise corresponding points of each group.

[0055] Step A2: Update the historical point cloud data of two adjacent frames based on the transformation matrix to determine new corresponding points.

[0056] Specifically, according to the calculated transformation matrix (R, t), each point in the frame of historical point cloud data collected later in two adjacent frames of historical point cloud data can be updated, that is, the corresponding point Q above is updated to obtain a new point Q update = {q 11 ,q 22 ,...,q nn}, you can use Q update =RP+t for calculation.

[0057] Step A3: Based on the new corresponding points, the transformation matrix is ​​cyclically optimized until the convergence condition is met to obtain the optimal transformation matrix; the convergence condition is that the error of the corresponding points in two adjacent frames of historical point cloud data reaches a set threshold.

[0058] Specifically, the updated corresponding point Q update = {q 11 ,q 22 ,...,q nn} and P = {p1,p2,...,p n}Continue to solve a new round of transformation matrix, so as to realize the cyclic optimization of the transformation matrix. Similarly, repeat the above steps until the convergence condition is met and the optimal transformation matrix is ​​obtained; the convergence condition is that the error of the corresponding points in the historical point cloud data of two adjacent frames reaches the set threshold. That is, let the corresponding points P and Q update When the error between them reaches the set threshold, the optimal transformation matrix can be obtained.

[0059] Step A4: Use the optimal transformation matrix as the historical pose relationship between two adjacent frames of historical point cloud data.

[0060] After the optimal transformation matrix is ​​determined, the optimal transformation matrix can be used as the historical posture relationship between two adjacent frames of historical point cloud data.

[0061] S203: Input at least two frames of historical point cloud data and pre-labeling results into the target detection model.

[0062] Specifically, at least two frames of historical point cloud data, the detection results and position information of each object annotation contained in each frame of historical point cloud data, and the historical pose relationship between two adjacent frames of historical point cloud data in at least two frames of historical point cloud data can be input into the target detection model to train the target detection model.

[0063] S204, fusing the rasterized historical point cloud data of each frame based on the time series information corresponding to each frame of historical point cloud data through the target detection model to obtain 3D point cloud features of each frame of historical point cloud data.

[0064] Among them, the timing information corresponding to each frame of historical point cloud data is the acquisition time corresponding to each frame of historical point cloud data. Rasterization is to process each frame of historical point cloud data to convert each frame of historical point cloud data into a three-dimensional cylindrical grid for representation. Specifically, after knowing the timing information of each frame of historical point cloud data, the rasterized historical point cloud data of each frame can be fused based on the timing information corresponding to each frame of historical point cloud data through the target detection model to obtain the 3D point cloud features of each frame of historical point cloud data.

[0065] As an optional but non-limiting implementation method, the process of rasterizing each frame of historical point cloud data may include the following steps B1-B3:

[0066] Step B1: Based on the historical posture relationship and the time deviation information between each frame of historical point cloud data, each frame of historical point cloud data is converted to the same coordinate system to align the historical point cloud data of each frame.

[0067] The time deviation information is the deviation in the acquisition time of each frame of historical point cloud data relative to the earliest acquisition time of the frame of historical point cloud data. Specifically, the acquisition time of different frames of historical point cloud data minus the acquisition time corresponding to the earliest acquisition time of the frame of historical point cloud data can be obtained to obtain the time deviation information between the frames of historical point cloud data.

[0068] Specifically, based on the historical pose relationship corresponding to adjacent historical point cloud data and the time deviation information between each frame of historical point cloud data, each frame of historical point cloud data can be converted to the same coordinate system. In the same coordinate system, each frame of historical point cloud data can be expressed as (x i ,y i ,z i ,r i ,t_offset i ), where x i ,y i ,z i They respectively reflect the coordinate values ​​of the signal reflection points contained in the historical point cloud data of this frame in the same coordinate system after conversion, r iis the reflection intensity of the signal reflection point, t_offset i The time deviation information corresponding to the historical point cloud data of this frame. Converting the historical point cloud information of each frame to the same coordinate system not only realizes the alignment of the historical point cloud data of each frame, but also can intuitively reflect the difference and progression of the historical point cloud data of adjacent frames in time sequence, and can also intuitively see the specific situation of the point cloud data of each frame, which is more conducive to the fusion of rasterized point cloud data in the subsequent scheme.

[0069] Step B2: Divide each frame of historical point cloud data based on equally spaced cylindrical spaces so that the points in the point cloud data are distributed in each cylindrical space.

[0070] The equidistant cylindrical space means that the distances between the cylinders on the cross section are equal. Specifically, for a frame of historical point cloud data, the equidistant cylindrical space is used to divide the frame of historical point cloud data so that each point in the frame of historical point cloud data can be distributed in the corresponding cylindrical space. For example, Figure 3a A schematic diagram of an equidistant column space provided by an embodiment of the present invention, such as Figure 3a As shown, the equally spaced cylindrical space is composed of 4*4 cylinders, and the length and width of each cylinder are the same. Based on such a cylindrical space, the historical point cloud data of this frame can be divided so that each point in the historical point cloud data of this frame can be distributed correspondingly in these 4*4 cylinders. Similarly, the same principle is performed on each frame of historical point cloud data to achieve the spatial division of each frame of historical point cloud data. It should be noted that even if the historical point cloud data of different frames are divided, it is necessary to ensure that the length and width of each cylinder are the same during the division, that is, the division is carried out according to the same standard cylindrical space. Thereby, the historical point cloud data of each frame can be rasterized.

[0071] It should be noted that for these 4*4 cylinders, the number of points distributed in each cylinder may not be consistent, which is specifically determined by the points contained in the divided frame of historical point cloud data. At the same time, the number of cylinder spaces used to divide the historical point cloud data is not limited here, and it needs to be determined based on the number and range of points in the historical point cloud data. If there are some cylinders in the cylinder space that have no points distributed inside after the division is completed, such cylinders are empty cylinders. For empty cylinders, they can be filled with 0 in subsequent solution calculations. Correspondingly, a cylinder with points distributed inside is a non-empty cylinder.

[0072] Step B3: Based on the center point of each cylinder, the points corresponding to each frame of point cloud data and belonging to the same cylinder space are dimensionally increased.

[0073] For each frame of historical point cloud data, each point in each frame of historical point cloud data can be dimensioned to combine local residual information with the point cloud information to achieve feature enhancement of the point cloud data. Specifically, for a frame of historical point cloud data, the offset between each point in the frame of point cloud data and the center point of the cylinder to which it belongs can be used as a new feature dimension and added to the data corresponding to each point to achieve dimension enhancement of each point.

[0074] For example, continue as Figure 3a As shown, Figure 3a Taking the distribution of each point in the lower left corner of the cylinder as an example, first determine the coordinates of the center point of the lower left corner of the cylinder as (x c ,y c ,z c ), then determine the offset of each point in the cylinder relative to the center point, and increase the dimension of each point based on the obtained offset. After the increase in dimension, each point is expressed as (x i ,y i ,z i ,r i ,t_offset i ,x c ,y c ,z c ,x p ,y p ), where x p The reflected signal reflection point (x i ,y i ,z i ) relative to the center point (x c ,y c ,z c ) is the coordinate difference in the X direction of the coordinate system; p The reflected signal reflection point (x i ,y i ,z i ) relative to the center point (x c ,y c ,z c ) in the Y direction of the coordinate system. In other words, x p ,y pIt essentially reflects the offset between the signal reflection point and the center point of the cylinder to which it belongs. After adding the offset from the center point of the cylinder, each point has the characteristics of 10 dimensions, which means that the dimensionality of the data has been increased. By analogy, the points distributed in other cylinders are processed in the same way, that is, all the points distributed in the 4*4 cylinders are processed to increase the dimensionality of each point in the historical point cloud data of this frame. By the same principle, each frame of historical point cloud data is operated in the same way to increase the dimensionality of each frame of historical point cloud data.

[0075] After dimensionality increase, the feature tensor corresponding to each frame of historical point cloud data is (D, P, N), where D represents the number of feature dimensions of each point after rasterization, N represents the number of points distributed in each non-empty cylinder, and P represents the number of all non-empty cylinders. It should be noted that an upper limit value can be set for the parameter N. If the number of points distributed in the non-empty cylinder exceeds the upper limit value of N, only N points are taken to downsample the point cloud data, thereby reducing the complexity of the data and the amount of calculation.

[0076] As an optional but non-limiting implementation method, the rasterized historical point cloud data of each frame is fused based on the time series information corresponding to each frame of historical point cloud data through the target detection model to obtain the 3D point cloud features of each frame of historical point cloud data, which may include the following steps C1-C2:

[0077] Step C1: The rasterized historical point cloud data of each frame is processed by the target detection model based on the sparse convolutional neural network to obtain the bird's-eye view features corresponding to each frame of historical point cloud data under the bird's-eye view perspective.

[0078] Specifically, in order to obtain the feature tensor (D, P, N), the rasterized historical point cloud data of each frame is processed through the target detection model based on the sparse convolutional neural network to obtain the bird's-eye view features corresponding to each frame of historical point cloud data under the bird's-eye view (Bird's Eye View).

[0079] Step C2: Based on the time series information corresponding to each frame of historical point cloud data, the target detection model stacks the bird's-eye view features corresponding to each frame of historical point cloud data in the time dimension to obtain the 3D point cloud features of each frame of historical point cloud data.

[0080] Specifically, since the time series information can reflect the time series relationship of each frame of historical point cloud data, the bird's-eye view features corresponding to each frame of historical point cloud data can be stacked in the time dimension, and the degree to which each bird's-eye view feature needs to be translated during the stacking process can be determined based on the time series information. Then, the backbone neural network of the target detection model is used to extract features to obtain the 3D point cloud features of each frame of historical point cloud data. For example, the backbone neural network can be a pointpi l lar backbone neural network.

[0081] For example, take the stacking of the bird's-eye view features of two frames of historical point cloud data as an example, Figure 3b As shown, Figure 3b A schematic diagram of the effect of stacking bird's-eye view features corresponding to point cloud data provided by an embodiment of the present invention, Figure 3b The orange column on the upper left is T k-1 The bird’s-eye view features corresponding to the frame historical point cloud data, and the corresponding acquisition time is T k-1 time; Figure 3b The blue column on the lower left is T k The bird’s-eye view features corresponding to the frame historical point cloud data, and the corresponding acquisition time is T k Based on the timing information between the two frames of point cloud data, it can be determined that the two need to be staggered by a cylindrical space during the stacking process. After stacking the two in the time dimension, the 3D BEV feature can be obtained, that is, Figure 3b The stacking effect on the right side.

[0082] As an optional but non-limiting implementation, before using the backbone neural network module of the target detection model to extract features, it also includes:

[0083] After stacking each bird's-eye view feature in the time dimension, the RGB feature information corresponding to each frame of historical point cloud data can be added to the obtained 3DBEV feature that integrates time information. Among them, the RGB feature information can be obtained by performing 2D convolution extraction based on the RGB image corresponding to each frame of historical point cloud data actually captured. After adding the corresponding RGB feature information, the backbone neural network module of the target detection model is used to extract features.

[0084] S205. The 3D target detection head based on the target detection network performs 3D target detection and posture relationship prediction on the 3D point cloud features of each frame of historical point cloud data.

[0085] Among them, the 3D target detection head includes a classification network and a regression network, which can perform 3D target detection on the 3D point cloud features of each frame of historical point cloud data, that is, the category and position of the object contained in each frame of historical point cloud data can be detected. Specifically, the classification network can be used to obtain the category and confidence of each object contained in each frame of historical point cloud data. The regression network can be used to obtain the position and size of the 3D bounding box of each object contained in each frame of historical point cloud data. Among them, the 3D bounding box can be represented by a seven-dimensional vector (x, y, z, w, l, h, θ), where x, y, z represent the geometric center coordinates of the 3D bounding box; w, l, h represent the width, length, and height of the 3D bounding box; θ represents the heading angle of the 3D bounding box. At the same time, the pose relationship between two adjacent frames of historical point cloud data can also be predicted.

[0086] S206: Determine a loss function of the target detection model based on the model prediction results and pre-labeling results output by the target detection model.

[0087] Specifically, the model prediction results output by the target detection model include the category detection results and position detection results of each object contained in each frame of historical point cloud data, as well as the predicted pose relationship between two adjacent frames of historical point cloud data. Based on the model prediction results and pre-labeling results output by the target detection model, the loss function of the target detection model can be determined. Three loss functions are used in the training process of the target detection model, namely, using the classification loss function Focal Loss as the loss function corresponding to the classification network; using the Smooth L1 loss function as the loss function corresponding to the regression network; and using the cosine similarity of the matrix to construct the loss function of the pose relationship, where the loss function L1 of the pose relationship pose It can be the following formula:

[0088]

[0089] Among them, R gt is the rotation matrix in the optimal transformation matrix in the pre-labeled result; t gt is the translation vector in the optimal transformation matrix in the pre-annotated result; R pre is the rotation matrix in the optimal transformation matrix in the model prediction result; t pre is the translation vector in the optimal transformation matrix in the model prediction result; m is the number of frames of historical point cloud data input in the target model training process. It should be noted that the number of frames of historical point cloud data input in the target model training process is not the more the better, and needs to be determined based on the actual situation of model training, for example, it can be two consecutive frames or three consecutive frames.

[0090] S207. When the loss function meets the convergence condition, the training ends and the target detection model is obtained.

[0091] Specifically, the convergence condition can be that the values ​​of each loss function decrease and tend to be stable, or the number of model training times exceeds a set threshold. When the loss function meets the convergence condition, the training ends and the target detection model is obtained. Figure 3c As shown, Figure 3c A schematic diagram of a training process of a target detection model provided by an embodiment of the present invention, Figure 3c In the process, three frames of continuous historical point cloud data are input, and finally the 3D object detection head can be used to output the confidence detection results corresponding to the objects in each frame of historical point cloud data (that is, the category of the object can be determined), the 3D box detection results (that is, the position of the object can be determined), and the pose relationship prediction results between adjacent frames of point cloud data.

[0092] S208. Input at least two frames of target point cloud data into a target detection model, and output, through the target detection model, a pose detection result and actual detection results corresponding to each of the at least two frames of target point cloud data, wherein the at least two frames of target point cloud data are point cloud data collected continuously in acquisition time, and the actual detection result corresponding to each frame of target point cloud data includes a candidate detection result of at least one candidate object contained in each frame of target point cloud data, the candidate detection result includes a category and a position of the candidate object, and the pose detection result is used to indicate a pose relationship between two frames of target point cloud data that are adjacent in acquisition time in the at least two frames of target point cloud data.

[0093] S209. Based on the posture detection results and the motion model, at least two target detection results of the target object are screened out from the candidate detection results corresponding to the actual detection results of at least two frames of target point cloud data; the target object is determined by screening at least one candidate object, and at least two target detection results belong to the actual detection results corresponding to the target point cloud data of different frames.

[0094] S210: Generate a trajectory of the target object based on at least two target detection results corresponding to the target object.

[0095] The technical solution provided by the present invention inputs at least two frames of target point cloud data into a target detection model, and outputs the pose detection results and the actual detection results corresponding to the at least two frames of target point cloud data through the target detection model; based on the pose detection results and the motion model, at least two target detection results of the target object are screened out from the candidate detection results corresponding to the actual detection results of at least two frames of target point cloud data; the target object is to screen and determine at least one candidate object, and at least two target detection results belong to the actual detection results corresponding to the target point cloud data of different frames; based on the at least two target detection results corresponding to the target object, the trajectory of the target object is generated. The solution of the present invention can achieve more comprehensive and reliable target detection and generate accurate target trajectories, especially can solve the problems of short-term occlusion and disappearance of the target, so that the target trajectory finally generated is more comprehensive and accurate.

[0096] Figure 4 A flowchart of another target trajectory generation method provided by an embodiment of the present invention is provided. The technical solution of this embodiment is based on the technical solution of the above embodiment. The technical solution of this embodiment further optimizes the process of selecting at least two target detection results of the target object from each candidate detection result corresponding to the actual detection result of at least two frames of target point cloud data based on the posture detection results and motion model in the above embodiment. This embodiment can be combined with each optional solution in one or more of the above embodiments. Figure 4 As shown, the target trajectory generation method of the embodiment of the present invention may include the following process:

[0097] S401. Input at least two frames of target point cloud data into a target detection model, and output a pose detection result and actual detection results corresponding to each of the at least two frames of target point cloud data through the target detection model, wherein the at least two frames of target point cloud data are point cloud data collected continuously in acquisition time, and the actual detection result corresponding to each frame of target point cloud data includes a candidate detection result of at least one candidate object contained in each frame of target point cloud data, and the candidate detection result includes a category and a position of the candidate object. The pose detection result is used to indicate a pose relationship between two frames of target point cloud data that are adjacent in acquisition time in at least two frames of target point cloud data. The target detection model is used to perform data analysis on at least two frames of point cloud data that are continuous in acquisition time, and outputs a category detection result and a position detection result of an object in each frame of point cloud data and a pose relationship between two adjacent frames of point cloud data.

[0098] S402. Determine a first actual detection result and a second actual detection result based on actual detection results corresponding to at least two frames of target point cloud data, wherein the first actual detection result is the actual detection result corresponding to the current frame point cloud data, and the second actual detection result is the actual detection result corresponding to the next frame point cloud data; the current frame point cloud data and the next frame point cloud data are two adjacent frames of point cloud data, and the corresponding acquisition time of the current frame point cloud data is earlier than the corresponding acquisition time of the next frame point cloud data.

[0099] Specifically, since what is input to the target detection model is at least two frames of target point cloud data collected continuously in the acquisition time, and the target detection model outputs the pose detection result and the actual detection results corresponding to at least two frames of target point cloud data. Then, for these actual detection results obtained, the first actual detection result and the second actual detection result can be determined therefrom. Among them, the first actual detection result and the second actual detection result are essentially the actual detection results corresponding to two adjacent frames of target point cloud data, that is, the first actual detection result is the actual detection result corresponding to the current frame point cloud data, the second actual detection result is the actual detection result corresponding to the next frame point cloud data, and the corresponding acquisition time of the current frame point cloud data is earlier than the corresponding acquisition time of the next frame point cloud data.

[0100] Exemplarily, it is assumed that from the actual detection results output from the target detection model, the first actual detection result is determined to be T k The actual detection result corresponding to the frame target point cloud data, the second actual detection result is T k+1 The actual detection result corresponding to the frame target point cloud data. Then T k Frame target point cloud data and T k+1 The frame target point cloud data is two adjacent frames of point cloud data, and T k The acquisition time corresponding to the frame target point cloud data must be earlier than T k+1 The acquisition time corresponding to the frame target point cloud data.

[0101] S403: Determine the next moment corresponding to the next frame of point cloud data.

[0102] After the next frame of point cloud data is determined, the next moment corresponding to the next frame of point cloud data can be determined. k+1 The frame target point cloud data is the next frame point cloud data, and the next moment corresponding to it is T k+1 time.

[0103] S404: Based on the pose detection result, the current frame point cloud data and the next frame point cloud data are converted to the same coordinate system.

[0104] Specifically, based on the pose detection result output by the target detection model, the coordinates of the points in the current frame point cloud data and the next frame point cloud data can be transformed so that the two frames of point cloud data are converted to the same coordinate system.

[0105] S405. Track each object in the current frame point cloud data, and predict the prediction results of each object at the next moment based on the motion model according to the state parameters corresponding to each object in the current frame point cloud data; the prediction results include the category prediction results and position prediction results corresponding to each object at the next moment.

[0106] Among them, the motion model can predict the state information of the object at the next moment through the various state information of the object at the current moment. For example, the motion model can be a 3D Kalman filter model. The state parameters corresponding to each object in the current frame point cloud data can be determined based on the actual detection results corresponding to the current frame point cloud data, specifically based on the candidate detection results corresponding to each candidate object in the actual detection results. These state parameters include the category corresponding to the object, and the position and size of the 3D bounding box.

[0107] Specifically, each object in the current frame point cloud data can be tracked, and the 3D Kalman filter model can be used to predict the prediction results of each object at the next moment based on the state parameters corresponding to each object in the current frame point cloud data; the obtained prediction results include the category prediction results and position prediction results corresponding to each object at the next moment, wherein the category prediction result is the category corresponding to each object, and the position prediction result is the predicted position and size of the 3D bounding box corresponding to each object.

[0108] For example, T k+1 The prediction of each target object at time T is taken as an example to illustrate, provided that k The frame target point cloud data is the current frame point cloud data, and the adjacent T k+1 The frame target point cloud data is the next frame point cloud data, and T k T corresponding to the frame target point cloud data k The time is the current time, T k+1 T corresponding to the frame target point cloud data k+1 The moment is the next moment. Specifically, first, for the output of the target detection model about T k The actual detection results corresponding to the frame target point cloud data can determine T kThe categories, positions and sizes of 3D bounding boxes of the objects in the frame. For example, suppose the position and size of the 3D bounding box of object e1, the position and size of the 3D bounding box of object f1, and the position and size of the 3D bounding box of object g1 can be obtained. According to the positions and sizes of the 3D bounding boxes corresponding to the objects, it can be determined that the objects e1, f1 and g1 are respectively in the current T k Then, in the process of using the 3D Kalman filter model for prediction, the state parameters corresponding to the time instant T are calculated based on the state parameters of objects e1, f1, and g1 at the current time instant T. k The state parameters corresponding to the time are used to predict the three objects at T k+1 The state parameters corresponding to the time instant are predicted as follows: k+1 The position and size of the 3D bounding box corresponding to the moment, and the object f1 at T k+1 The position and size of the 3D bounding box corresponding to the moment, and the position of the object g1 at T k+1 The position and size of the 3D bounding box corresponding to the moment.

[0109] S406: Perform similarity matching between the predicted result and each candidate detection result corresponding to the second actual detection result to obtain a similarity matching result.

[0110] Specifically, the state parameters of each object in the obtained prediction results can be converted to the same coordinate system where the current frame point cloud data and the next frame point cloud data are located. That is to say, at this time, the coordinate system not only has the actual detection results corresponding to the two frames of point cloud data output by the target detection model, specifically: the actual detection results corresponding to the current frame point cloud data (ie: the first actual detection results) and the actual detection results corresponding to the next frame point cloud data (ie: the second actual detection results); at the same time, the coordinate system also has the prediction results corresponding to the next moment obtained after predicting each object in the current frame point cloud data with the help of the 3D Kalman filter model. It should be noted that the time corresponding to the prediction result and the next moment corresponding to the next frame point cloud data are essentially the same moment. For example, they both correspond to T k+1 Then, in the coordinate system, the prediction result can be matched with each candidate detection result corresponding to the second actual detection result by similarity. Specifically, 3D Intersection over Union (3D I ntersect ion over Union) can be used to calculate the intersection ratio), and then the Hungarian algorithm can be used to match the target and the trajectory to obtain a similarity matching result.

[0111] S407 . Based on the similarity matching result, associate the candidate detection results for describing the same target object in the first actual detection result and the second actual detection result to screen out at least two target detection results for describing the same target object.

[0112] Specifically, based on the similarity matching results, it can be determined which object contained in the second actual detection result can be matched with which object in the predicted result. If they can match, it means that the two essentially describe the same object at the same time, and then it can be determined that the object in the second actual detection result can correspond to an object in the first actual detection result. Then, the candidate detection results for describing the same target object in the first actual detection result and the second actual detection result can be associated to screen out at least two target detection results for describing the same target object.

[0113] For example, if a 3D Kalman filter model is used for T k After predicting the three objects (object e1, object f1, and object g1) contained in the frame target point cloud data, we can get T k+1 The prediction results corresponding to the three objects at the moment are the prediction results of object e1, object f1 and object g1. And the premise is that the target detection model has also output T k+1 The candidate detection results of each candidate object contained in the frame target point cloud data, for example, are the candidate detection results of candidate object e2, the candidate detection results of candidate object f2, the candidate detection results of candidate object g2, and the candidate detection results of candidate object h2. Then, in the process of similarity matching the predicted result with each candidate detection result corresponding to the second actual detection result, the essence is to determine which objects can be successfully matched among the "related objects of the predicted result (object e1, object f1, and object g1)" and the "related objects of the second actual detection result (candidate object e2, candidate object f2, candidate object g2, and candidate object h2)". Two objects that can be successfully matched indicate that the two objects are essentially describing the same object. For example, if after matching, it is determined that the predicted result of object e1 can match the candidate detection result corresponding to candidate object f2, it means that both object e1 and candidate object f2 essentially describe the object in T k+1 Then, since the prediction result corresponding to object e1 is derived from the prediction result of T k The object e1 contained in the frame target point cloud data is predicted to be k+1 The situation at the moment, then, with the help of this intermediate bridge, we can determine the "T k The frame target point cloud data contains objects e1" and "T k+1The candidate objects f2'' contained in the frame target point cloud data are essentially the same object (e.g., a pedestrian), that is, the target object; then, the candidate detection results for describing the same target object in the first actual detection result and the second actual detection result can be associated to screen out at least two target detection results for describing the same target object. For example, screen out T k Frame target point cloud data and T k+1 In the frame target point cloud data, which ones are describing the detection results of pedestrians, which ones are describing the detection results of trees, and which ones are describing the detection results of vehicles.

[0114] S408: Generate a trajectory of the target object based on at least two target detection results corresponding to the target object.

[0115] For example, through the previous steps, we can determine T k Frame target point cloud data and T k+1 Which ones in the frame target point cloud data are describing the detection results of pedestrians, that is, at least two target detection results corresponding to the target object are obtained, and then the trajectory of the pedestrian can be generated based on such at least two target detection results.

[0116] As an optional but non-limiting implementation method, after obtaining the similarity matching result, the target trajectory generation method also includes: if the similarity matching result exceeds the matching threshold, the match is successful, and the motion model is updated based on the generated trajectory of the target object; if the similarity matching result does not exceed the matching threshold, the match is unsuccessful, and the actual detection results and the prediction results are retained until the preset number of life cycles are reached and then they are cleaned up.

[0117] Specifically, if the similarity matching result exceeds the matching threshold, the match is successful, and the motion model is updated based on the generated trajectory of the target object; if the similarity matching result does not exceed the matching threshold, the match is unsuccessful, and the actual detection results and the predicted results are retained. For example, in the matching process, some objects in the predicted results cannot be matched to any candidate object in the actual detection results. Such objects may not be collected during the actual point cloud data collection process and disappear, so these objects in the predicted results cannot be matched. For another example, there may be some candidate objects in the second actual detection result that cannot be matched to any of the related objects in the predicted results. Such candidate objects may be newly collected objects in the actual collection scene. When collecting the actual scene, the frame of point cloud data with an earlier collection time naturally cannot collect the information of the newly appeared object, so the related objects of the predicted results naturally do not include this new object, and the subsequent solutions naturally cannot match. For such unsuccessful matching objects, the relevant content of the actual detection results and the predicted results can be retained until the preset number of life cycles are reached. For example, after matching, it is found that T k There is an object s in the frame target point cloud data that is completely unrelated to T k+1 If any object in the target point cloud data of the frame is found, the detection results related to the object s can be retained, and then the detection results can be continued with the Tth frame. k+2 Frame target point cloud data is matched and associated, or with T k+3 The frame target point cloud data is matched and associated. Once it can be matched with the object in the subsequent target point cloud data collected later, the trajectory of the object s can be generated. Although the relevant information of the object s is blocked at a certain moment, the trajectory of the object s can still be generated as a whole. This method can solve the problems of short-term occlusion and disappearance of the target, making the target detection result more stable and reliable, and generating the target trajectory at the same time.

[0118] The technical solution provided by the present invention inputs at least two frames of target point cloud data into a target detection model, and outputs the pose detection results and the actual detection results corresponding to the at least two frames of target point cloud data through the target detection model; based on the pose detection results and the motion model, at least two target detection results of the target object are screened out from the candidate detection results corresponding to the actual detection results of at least two frames of target point cloud data; the target object is to screen and determine at least one candidate object, and at least two target detection results belong to the actual detection results corresponding to the target point cloud data of different frames; based on the at least two target detection results corresponding to the target object, the trajectory of the target object is generated. The solution of the present invention can achieve more comprehensive and reliable target detection and generate accurate target trajectories, especially can solve the problems of short-term occlusion and disappearance of the target, so that the target trajectory finally generated is more comprehensive and accurate.

[0119] Figure 5 This is a schematic diagram of the structure of a target trajectory generation device provided by an embodiment of the present invention. The embodiment of the present invention is applicable to the case of constructing and generating the trajectory of a target object. The target trajectory generation device can be implemented in the form of software and / or hardware, and is generally integrated on any electronic device with network communication function, which can be a mobile terminal, PC or server, etc. Figure 5 As shown, the target trajectory generation device of the embodiment of the present invention may include a detection module 510, a target detection result determination module 520 and a trajectory generation module 530. Among them:

[0120] A detection module 510 is used to input at least two frames of target point cloud data into a target detection model, and output a pose detection result and an actual detection result corresponding to each of the at least two frames of target point cloud data through the target detection model, wherein the at least two frames of target point cloud data are point cloud data collected continuously in acquisition time, and the actual detection result corresponding to each frame of the target point cloud data includes a candidate detection result of at least one candidate object contained in each frame of the target point cloud data, and the candidate detection result includes a category and a position of the candidate object, and the pose detection result is used to indicate a pose relationship between two frames of target point cloud data that are adjacent in acquisition time in at least two frames of target point cloud data, and the target detection model is used to perform data analysis on at least two frames of point cloud data that are continuous in acquisition time, and output a category detection result and a position detection result of an object in each frame of point cloud data and a pose relationship between two adjacent frames of point cloud data;

[0121] The target detection result determination module 520 is used to screen out at least two target detection results of the target object from each candidate detection result corresponding to the actual detection result of at least two frames of target point cloud data based on the pose detection result and the motion model; the target object is the actual detection result corresponding to the target point cloud data of different frames after screening and determination of the at least one candidate object;

[0122] The trajectory generating module 530 is configured to generate a trajectory of the target object based on the at least two target detection results corresponding to the target object.

[0123] As an optional but non-limiting implementation method, the target trajectory generation device also includes: a target detection model training module, which is used to obtain at least two consecutive frames of historical point cloud data within a historical time period; determine the pre-annotation results corresponding to the at least two frames of historical point cloud data, and the pre-annotation results include the historical category detection results and historical position detection results corresponding to the objects in each frame of historical point cloud data and the historical posture relationship between two adjacent frames of historical point cloud data; input the at least two frames of historical point cloud data and the pre-annotation results into the target detection model; through the target detection model, the rasterized frames of historical point cloud data are fused based on the timing information corresponding to each frame of historical point cloud data to obtain the 3D point cloud features of each frame of historical point cloud data; based on the 3D target detection head of the target detection network, 3D target detection and posture relationship prediction are performed on the 3D point cloud features of each frame of historical point cloud data; based on the model prediction results output by the target detection model and the pre-annotation results, the loss function of the target detection model is determined; when the loss function meets the convergence condition, the training ends and the target detection model is obtained.

[0124] As an optional but non-limiting implementation method, the target trajectory generation device also includes: a historical posture relationship determination module, which is used to determine the transformation matrix of the corresponding points in the two adjacent frames of historical point cloud data by the nearest neighbor method; based on the transformation matrix, the two adjacent frames of historical point cloud data are updated to determine new corresponding points; based on the new corresponding points, the transformation matrix is ​​cyclically optimized until the convergence condition is met to obtain the optimal transformation matrix; the convergence condition is that the error of the corresponding points in the two adjacent frames of historical point cloud data reaches a set threshold; and the optimal transformation matrix is ​​used as the historical posture relationship of the two adjacent frames of historical point cloud data.

[0125] As an optional but non-limiting implementation method, the target trajectory generation device also includes: a point cloud data rasterization module, which is used to convert each frame of historical point cloud data into the same coordinate system based on the historical posture relationship and the time deviation information between each frame of historical point cloud data, so as to align each frame of historical point cloud data; divide each frame of historical point cloud data based on equally spaced cylindrical spaces so that the points in the point cloud data are distributed in each cylindrical space; and increase the dimensions of the points corresponding to each frame of point cloud data and belonging to the same cylindrical space based on the center point of each cylinder.

[0126] As an optional but non-limiting implementation method, the target detection model training module is used to fuse the rasterized historical point cloud data of each frame based on the timing information corresponding to each frame of historical point cloud data through the target detection model to obtain 3D point cloud features of each frame of historical point cloud data, specifically including: processing the rasterized historical point cloud data of each frame based on a sparse convolutional neural network through the target detection model to obtain the bird's-eye view features corresponding to each frame of historical point cloud data from a bird's-eye view perspective; stacking the bird's-eye view features corresponding to each frame of historical point cloud data in the time dimension based on the timing information corresponding to each frame of historical point cloud data through the target detection model to obtain the 3D point cloud features of each frame of historical point cloud data.

[0127] As an optional but non-limiting implementation, the target detection result determination module 520 includes an actual detection result determination unit, a next moment determination unit, a conversion unit, a tracking prediction unit, a matching unit and an association unit. Among them: the actual detection result determination unit is used to determine the first actual detection result and the second actual detection result based on the actual detection results corresponding to the at least two frames of target point cloud data, wherein the first actual detection result is the actual detection result corresponding to the current frame point cloud data, and the second actual detection result is the actual detection result corresponding to the next frame point cloud data; the current frame point cloud data and the next frame point cloud data are two adjacent frames of point cloud data, and the corresponding acquisition time of the current frame point cloud data is earlier than the corresponding acquisition time of the next frame point cloud data; the next moment determination unit is used to determine the next moment corresponding to the next frame point cloud data; the conversion unit is used to convert the current frame point cloud data and the next frame point cloud data to the same coordinate system based on the posture detection result. The invention further comprises the following steps: a tracking prediction unit, which is used to track each object in the current frame point cloud data, and predict the prediction result of each object at the next moment based on the motion model according to the state parameters corresponding to each object in the current frame point cloud data; the prediction result includes the category prediction result and the position prediction result corresponding to each object at the next moment; a matching unit, which is used to perform similarity matching between the prediction result and each candidate detection result corresponding to the second actual detection result to obtain a similarity matching result; an association unit, which is used to associate the candidate detection results for describing the same target object in the first actual detection result and the second actual detection result based on the similarity matching result, so as to screen out at least two target detection results for describing the same target object.

[0128] As an optional but non-limiting implementation, the target trajectory generation device further includes: a life cycle management unit, which is used to, if the similarity matching result exceeds the matching threshold, the matching is successful, and the motion model is updated based on the generated trajectory of the target object; if the similarity matching result does not exceed the matching threshold, the matching is unsuccessful, and the actual detection result and the prediction result are retained until a preset number of life cycles are reached.

[0129] The target trajectory generation device provided in the embodiment of the present invention can be used to execute the target trajectory generation method, and has the corresponding functional modules and beneficial effects of executing the target trajectory generation method.

[0130] It is worth noting that the various units and modules included in the above-mentioned device are only divided according to functional logic, but are not limited to the above-mentioned division, as long as the corresponding functions can be achieved; in addition, the specific names of the functional units are only for the convenience of distinguishing each other, and are not used to limit the protection scope of the embodiments of the present invention.

[0131] Figure 6 A schematic diagram of the structure of an electronic device for implementing a target trajectory generation method provided by an embodiment of the present invention. Figure 6 , which shows a schematic diagram of the structure of an electronic device 610 suitable for implementing an embodiment of the present invention. The terminal device in the embodiment of the present invention may include, but is not limited to, mobile terminals such as mobile phones, notebook computers, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Figure 6 The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.

[0132] like Figure 6 As shown, the electronic device 610 includes at least one processor 611, and a memory connected to the at least one processor 611 in communication, such as a read-only memory (ROM) 612, a random access memory (RAM) 613, etc., wherein the memory stores a computer program that can be executed by at least one processor, and the processor 611 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 612 or the computer program loaded from the storage unit 618 to the random access memory (RAM) 613. In the (RAM) 613, various programs and data required for the operation of the electronic device 610 can also be stored. The processor 611, (ROM) 612 and (RAM) 613 are connected to each other via a bus 614. An input / output (I / O) interface 615 is also connected to the bus 614.

[0133] A number of components in the electronic device 610 are connected to the I / O interface 615, including: an input unit 616, such as a keyboard, a mouse, etc.; an output unit 617, such as various types of displays, speakers, etc.; a storage unit 618, such as a disk, an optical disk, etc.; and a communication unit 619, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 619 allows the electronic device 610 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0134] The processor 611 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the processor 611 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The processor 611 executes the target trajectory generation method provided in any embodiment of the present invention.

[0135] In particular, according to an embodiment of the present invention, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present invention includes a computer program product, which includes a computer program carried on a non-transitory computer-readable medium, and the computer program includes a program code for executing the target trajectory generation method shown in the flowchart. When the computer program is executed by a processing device, the above functions defined in the target trajectory generation method of the embodiment of the present invention are executed.

[0136] The names of the messages or information exchanged between multiple devices in the embodiments of the present invention are only used for illustrative purposes, and are not used to limit the scope of these messages or information.

[0137] The electronic device provided in the embodiment of the present invention and the target trajectory generation method provided in the above embodiment belong to the same inventive concept. For technical details not fully described in this embodiment, reference can be made to the above embodiment, and this embodiment has the same beneficial effects as the above embodiment.

[0138] An embodiment of the present invention provides a computer storage medium on which a computer program is stored. When the program is executed by a processor, the target trajectory generation method provided by the above embodiment is implemented.

[0139] It should be noted that the computer-readable medium of the present invention can be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media can include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present invention, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device. In the present invention, a computer-readable signal medium can include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable program code. This propagated data signal can take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. The computer readable signal medium may also be any computer readable medium other than a computer readable storage medium, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer readable medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

[0140] Computer program code for performing the operations of the present invention may be written in one or more programming languages ​​or a combination thereof, including but not limited to object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0141] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present invention. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0142] The units involved in the embodiments of the present invention may be implemented in software or in hardware. The name of the unit does not limit the unit itself in some cases. The functions described above may be performed at least in part by one or more hardware logic components. For example, non-limitingly, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), system on chip (SOCs), complex programmable logic devices (CPLDs), and the like.

[0143] In the context of the present invention, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0144] The above description is only a preferred embodiment of the present invention and an explanation of the technical principles used. Those skilled in the art should understand that the disclosure scope involved in the present invention is not limited to the technical solution formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above disclosed concept. For example, the above features are replaced with the technical features with similar functions disclosed in the present invention (but not limited to) by each other.

[0145] In addition, although each operation is described in a specific order, this should not be construed as requiring these operations to be performed in the specific order shown or in a sequential order. Under certain circumstances, multitasking and parallel processing may be advantageous. Similarly, although some specific implementation details are included in the above discussion, these should not be construed as limiting the scope of the present invention. Some features described in the context of a separate embodiment can also be implemented in a single embodiment in combination. On the contrary, the various features described in the context of a single embodiment can also be implemented in multiple embodiments individually or in any suitable sub-combination mode.

[0146] Although the subject matter has been described in language specific to structural features and / or methodological logical actions, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. On the contrary, the specific features and actions described above are merely example forms of implementing the claims.

Claims

1. A target trajectory generation method, characterized in that: The method comprises: Input at least two frames of target point cloud data into a target detection model, and output a pose detection result and an actual detection result corresponding to each of the at least two frames of target point cloud data through the target detection model, wherein the at least two frames of target point cloud data are point cloud data collected continuously in acquisition time, and the actual detection result corresponding to each frame of the target point cloud data includes a candidate detection result of at least one candidate object contained in each frame of the target point cloud data, and the candidate detection result includes a category and a position of the candidate object, and the pose detection result is used to indicate a pose relationship between two frames of target point cloud data that are adjacent in acquisition time in at least two frames of target point cloud data, and the target detection model is used to perform data analysis on at least two frames of point cloud data that are continuous in acquisition time, and output a category detection result and a position detection result of an object in each frame of point cloud data and a pose relationship between two adjacent frames of point cloud data; Based on the pose detection result and the motion model, at least two target detection results of the target object are screened out from each candidate detection result corresponding to the actual detection result of at least two frames of target point cloud data; the target object is screened and determined for the at least one candidate object, and the at least two target detection results belong to the actual detection results corresponding to the target point cloud data of different frames; A trajectory of the target object is generated based on the at least two target detection results corresponding to the target object.

2. The method according to claim 1, characterized in that The training process of the target detection model includes: Obtain at least two consecutive frames of historical point cloud data within a historical time period; Determine pre-annotation results corresponding to the at least two frames of historical point cloud data, wherein the pre-annotation results include historical category detection results and historical position detection results corresponding to the objects in each frame of historical point cloud data and a historical posture relationship between two adjacent frames of historical point cloud data; Inputting the at least two frames of historical point cloud data and the pre-annotation result into the target detection model; The target detection model is used to fuse the rasterized historical point cloud data of each frame based on the time series information corresponding to the historical point cloud data of each frame to obtain the 3D point cloud features of the historical point cloud data of each frame; The 3D target detection head based on the target detection network performs 3D target detection and posture relationship prediction on the 3D point cloud features of each frame of historical point cloud data; Determining a loss function of the target detection model based on a model prediction result output by the target detection model and the pre-labeling result; The training ends when the loss function meets the convergence condition, and the target detection model is obtained.

3. The method according to claim 2, characterized in that The process of determining the historical pose relationship between two adjacent frames of historical point cloud data includes: Determine the transformation matrix of corresponding points in the two adjacent frames of historical point cloud data by the nearest neighbor method; Based on the transformation matrix, the two adjacent frames of historical point cloud data are updated to determine new corresponding points; The transformation matrix is ​​cyclically optimized based on the new corresponding points until a convergence condition is met to obtain an optimal transformation matrix; the convergence condition is that the error of the corresponding points in the two adjacent frames of historical point cloud data reaches a set threshold; The optimal transformation matrix is ​​used as the historical pose relationship between the two adjacent frames of historical point cloud data.

4. The method according to claim 3, characterized in that The process of rasterizing each frame of historical point cloud data includes: Based on the historical posture relationship and the time deviation information between each frame of historical point cloud data, each frame of historical point cloud data is converted to the same coordinate system to align each frame of historical point cloud data; The historical point cloud data of each frame is divided based on the equally spaced cylindrical space, so that the points in the point cloud data are distributed in each cylindrical space; Based on the center point of each cylinder, the points in the same cylinder space corresponding to each frame of point cloud data are increased in dimension.

5. The method according to claim 1, characterized in that The target detection model fuses the rasterized historical point cloud data of each frame based on the time sequence information corresponding to each frame of historical point cloud data to obtain the 3D point cloud features of each frame of historical point cloud data, including: The rasterized historical point cloud data of each frame is processed by the target detection model based on the sparse convolutional neural network to obtain the bird's-eye view features corresponding to each frame of historical point cloud data under the bird's-eye view perspective; Based on the timing information corresponding to each frame of historical point cloud data, the target detection model stacks the bird's-eye view features corresponding to each frame of historical point cloud data in the time dimension to obtain the 3D point cloud features of each frame of historical point cloud data.

6. The method according to claim 1, characterized in that The step of selecting at least two target detection results of the target object from candidate detection results corresponding to actual detection results of at least two frames of target point cloud data based on the pose detection result and the motion model includes: A first actual detection result and a second actual detection result are determined based on actual detection results corresponding to the at least two frames of target point cloud data, wherein the first actual detection result is the actual detection result corresponding to the current frame of point cloud data, and the second actual detection result is the actual detection result corresponding to the next frame of point cloud data; the current frame of point cloud data and the next frame of point cloud data are two adjacent frames of point cloud data, and the corresponding acquisition time of the current frame of point cloud data is earlier than the corresponding acquisition time of the next frame of point cloud data; Determine a next moment corresponding to the next frame of point cloud data; Based on the pose detection result, converting the current frame point cloud data and the next frame point cloud data into the same coordinate system; Tracking each object in the current frame point cloud data, and predicting the prediction result of each object at the next moment based on the motion model according to the state parameters corresponding to each object in the current frame point cloud data; the prediction result includes the category prediction result and the position prediction result corresponding to each object at the next moment; Performing similarity matching between the prediction result and each candidate detection result corresponding to the second actual detection result to obtain a similarity matching result; Based on the similarity matching result, candidate detection results for describing the same target object in the first actual detection result and the second actual detection result are associated to screen out at least two target detection results for describing the same target object.

7. The method according to claim 6, characterized in that After obtaining the similarity matching result, the method further includes: If the similarity matching result exceeds the matching threshold, the match is successful, and the motion model is updated based on the generated trajectory of the target object; If the similarity matching result does not exceed the matching threshold, the matching is unsuccessful, and the actual detection result and the prediction result are retained until the life cycle of the preset number of times is reached and then they are cleared.

8. A target trajectory generating device, characterized in that: The device comprises: A detection module, used for inputting at least two frames of target point cloud data into a target detection model, and outputting a pose detection result and an actual detection result corresponding to each of the at least two frames of target point cloud data through the target detection model, wherein the at least two frames of target point cloud data are point cloud data collected continuously in acquisition time, and the actual detection result corresponding to each frame of the target point cloud data includes a candidate detection result of at least one candidate object contained in each frame of the target point cloud data, and the candidate detection result includes a category and a position of the candidate object, and the pose detection result is used to indicate a pose relationship between two frames of target point cloud data that are adjacent in acquisition time in at least two frames of target point cloud data, and the target detection model is used to perform data analysis on at least two frames of point cloud data that are continuous in acquisition time, and output a category detection result and a position detection result of an object in each frame of point cloud data and a pose relationship between two adjacent frames of point cloud data; A target detection result determination module is used to screen out at least two target detection results of a target object from each candidate detection result corresponding to the actual detection result of at least two frames of target point cloud data based on the pose detection result and the motion model; the target object is the actual detection result corresponding to the target point cloud data of different frames after screening and determination of the at least one candidate object; The trajectory generation module is used to generate a trajectory of the target object based on the at least two target detection results corresponding to the target object.

9. An electronic device, characterized in that: The electronic device comprises: one or more processors; a storage device for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors implement the target trajectory generation method as described in any one of claims 1-7.

10. A storage medium containing computer executable instructions, characterized in that: The computer executable instructions are used to execute the target trajectory generation method as described in any one of claims 1-7 when executed by a computer processor.

Images