Point cloud matching method and device, equipment and storage medium

Abstract

The invention provides a point cloud matching method and device, equipment and a storage medium, and relates to the technical field of automatic driving, in particular to the technical fields of automatic driving perception, data annotation and the like. According to the specific implementation scheme, obstacle point clouds at multiple moments are obtained; constructing a factor graph by using the obstacle point clouds, the obstacle poses and the obstacle bounding boxes at the multiple moments; the factor graph is optimized; and matching the obstacle point clouds at the multiple moments based on the optimized factor graph. The point cloud matching accuracy of the target obstacle can be improved.

Description

Technical Field

[0001] The present disclosure relates to the field of autonomous driving technology, and in particular to technical fields such as autonomous driving perception and data labeling. Background Art

[0002] LiDAR is a common sensor for autonomous vehicles, but due to reasons such as field of view obstruction and sparse sampling of LiDAR, it can only obtain incomplete and sparse point clouds on one side of the target obstacle. In scenarios such as autonomous driving perception, data annotation, and digital asset production, in order to obtain the densest and most complete point cloud possible, it is necessary to match and fuse point clouds for the same obstacle at multiple times (or multiple frames) to obtain a complete point cloud for the obstacle. How to match point clouds for the same obstacle at different times (or different frames) is a technical problem that needs to be solved. Summary of the invention

[0003] The present disclosure provides a point cloud matching method, apparatus, device and storage medium.

[0004] According to one aspect of the present disclosure, a point cloud matching method is provided, comprising:

[0005] Obtain obstacle point clouds at multiple times;

[0006] Using the obstacle point clouds, obstacle poses, and obstacle bounding boxes at the multiple moments, a factor graph is constructed;

[0007] Optimizing the factor graph;

[0008] Based on the optimized factor graph, the obstacle point clouds at multiple moments are matched.

[0009] According to another aspect of the present disclosure, there is provided a point cloud matching device, comprising:

[0010] Point cloud acquisition module, used to obtain obstacle point clouds at multiple times;

[0011] A factor graph construction module is used to construct a factor graph using the obstacle point clouds, obstacle poses and obstacle bounding boxes at the multiple moments;

[0012] A factor graph optimization module, used for optimizing the factor graph;

[0013] The matching module is used to match the obstacle point clouds at the multiple moments based on the optimized factor graph.

[0014] According to another aspect of the present disclosure, there is provided an electronic device, comprising:

[0015] at least one processor; and

[0016] a memory communicatively connected to the at least one processor; wherein,

[0017] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute any method in the embodiments of the present disclosure.

[0018] According to another aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable the computer to execute any method according to the embodiments of the present disclosure.

[0019] According to another aspect of the present disclosure, a computer program product is provided, including a computer program, which implements any method according to the embodiments of the present disclosure when executed by a processor.

[0020] When performing point cloud matching, the present disclosure uses the obstacle bounding box as a constraint factor, so that the obstacle bounding box can be used to assist in point cloud matching, thereby improving the accuracy of point cloud matching.

[0021] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure.

[0023] Figure 1 is a schematic flow chart of a point cloud matching method proposed in an embodiment of the present disclosure;

[0024] Figure 2 It is a schematic diagram of the factor graph optimization principle;

[0025] Figure 3 is an example 1 of a factor graph constructed according to an embodiment of the present disclosure;

[0026] Figure 4 is Example 2 of a factor graph constructed according to an embodiment of the present disclosure;

[0027] Figure 5 is a structural schematic diagram of a point cloud matching device 500 according to an embodiment of the present disclosure;

[0028] Figure 6 A schematic block diagram of an example electronic device 600 that may be used to implement embodiments of the present disclosure is shown. DETAILED DESCRIPTION

[0029] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope of the present disclosure. Similarly, for the sake of clarity and conciseness, the description of well-known functions and structures is omitted in the following description.

[0030] The “and / or” in the embodiments of the present disclosure indicates that three relationships may exist. For example, A and / or B may indicate the following three situations: A exists alone, A and B exist at the same time, and B exists alone. The term “at least one” herein indicates any combination of at least two of any one or more of a plurality of types. For example, at least one of A, B, and C may indicate any one or more elements selected from the set consisting of A, B, and C. The terms “first” and “second” herein refer to and distinguish between multiple similar technical terms, and do not mean to limit the order or to limit the meaning to only two. For example, the first feature and the second feature refer to two types / two features. The first feature may be one or more, and the second feature may also be one or more.

[0031] LiDAR is a common sensor for autonomous vehicles, but due to reasons such as field of view obstruction and sparse sampling of LiDAR, it can only obtain incomplete and sparse point clouds on one side of the target obstacle. In scenarios such as autonomous driving perception, data annotation, and digital asset production, in order to obtain the densest and most complete point cloud possible, it is necessary to match and fuse multiple frames of vehicle incomplete point clouds. At present, there are some point cloud matching algorithms that are used to match point clouds for the same obstacle at different times (or different frames).

[0032] The main purpose of the point cloud matching algorithm is to find a transformation matrix to determine the spatial relationship between the point clouds of the same obstacle at different times (or different frames), so as to compare, align and merge the point cloud data of the same obstacle in three-dimensional space to generate complete point cloud data. Point cloud matching algorithms are widely used in computer vision, robot navigation and 3D modeling. The following are several common point cloud matching algorithms:

[0033] (1) Iterative Closest Point (ICP) algorithm. The ICP algorithm is an iterative least squares optimization algorithm that aligns two sets of point clouds by continuously matching the nearest neighbor points between the two point clouds and calculating the rotation and translation matrices (the rotation and translation matrices can form the pose transformation matrices between point clouds at different times). The advantage of this algorithm is that it is simple and easy to implement and is suitable for point cloud data that has been nearly aligned, but it is sensitive to the initial position and is prone to falling into a local optimal solution.

[0034] (2) Normal Distributions Transform (NDT) algorithm. The NDT algorithm divides the point cloud space into grids, builds a normal distribution model in each grid, and aligns the point cloud based on a probability model. The NDT algorithm performs well in noisy environments and is often used in SLAM systems, but it has a large amount of calculations and is suitable for more regular environments.

[0035] (3) Feature-based matching algorithm. This method first extracts features from the point cloud, such as Fast Point Feature Histograms (FPFH), Signature of Histograms of Orientations (SHOT) and other feature description factors, and then matches based on feature points. This method has good robustness to the local structure of the point cloud and is suitable for situations with large posture changes, but the computational complexity of feature extraction is large.

[0036] (4) Deep learning methods. In recent years, deep learning-based point cloud matching algorithms, such as PointNet, Dynamic Graph CNN (DGCNN), and Fast Global Registration (FGR), have been developed to extract point cloud features and perform global matching through neural networks. These algorithms can learn the complex geometric structure of point clouds and are highly robust to noise and missing data.

[0037] Although these algorithms have slightly different principles, they are all based on matching based on the geometric features of point clouds. Therefore, when the geometric features of point clouds are degraded, these algorithms do not work well.

[0038] Based on this, an embodiment of the present disclosure proposes a point cloud matching method. Figure 1 is a schematic flow chart of a point cloud matching method proposed in an embodiment of the present disclosure, comprising:

[0039] S110, obtaining obstacle point clouds at multiple times;

[0040] S120, constructing a factor graph using the obstacle point clouds, obstacle poses, and obstacle bounding boxes at the multiple moments;

[0041] S130, optimizing the factor graph;

[0042] S140: Matching the obstacle point clouds at the multiple moments based on the optimized factor graph.

[0043] In some examples, step S110 includes:

[0044] Using the vehicle's LiDAR system, point clouds are acquired at multiple adjacent moments.

[0045] Based on the point clouds respectively acquired at multiple adjacent moments, obstacle point clouds of the same obstacle at multiple moments are determined.

[0046] Among them, the ego vehicle can also be called the main vehicle, and the ego vehicle / main vehicle can be an autonomous driving vehicle.

[0047] In some examples, the laser radar system of the vehicle collects point clouds of the surrounding environment at multiple consecutive moments. For example, point clouds are collected at multiple moments such as time k (or the moment corresponding to the kth frame), time k+1 (or the moment corresponding to the k+1th frame), and time k+2 (or the moment corresponding to the k+2th frame); there is a time sequence between time k, time k+1, and time k+2, for example, time k is the earliest moment. Among them, the point cloud collected at time k is called the kth frame point cloud, the point cloud collected at time k+1 is called the k+1th frame point cloud, the point cloud collected at time k+2 is called the k+2th frame point cloud, and so on.

[0048] In some examples, a detection model (such as a 3D detection model) is used to divide the point cloud. The 3D detection model is a neural network model used for target detection in three-dimensional space. It can detect and locate objects from 3D point clouds or other three-dimensional data. In one embodiment of the present disclosure, the 3D detection model is used to identify the point clouds of dynamic obstacles in the point clouds acquired at multiple times based on the 3D bounding box. Afterwards, the point clouds of the same obstacle are determined from the point clouds acquired at multiple times.

[0049] In some examples, a Multi-Object Tracking (MOT) algorithm is used to identify obstacle point clouds for the same obstacle. The MOT algorithm is an algorithm for tracking multiple targets in video or point cloud data. It can continuously detect and identify multiple targets in a scene in time series data, and assign a unique ID to each target to maintain target consistency across frames. The disclosed embodiment uses the MOT algorithm to associate dynamic obstacles in adjacent frames, obtain a continuous correspondence between dynamic obstacles between frames, and thereby determine the obstacle point cloud for the same obstacle in the factor graph, creating conditions for constructing the factor graph.

[0050] For example, the point cloud at time k includes obstacle point clouds corresponding to obstacle 1, obstacle 2, and obstacle 3; the dynamic point cloud at time k+1 includes obstacle point clouds corresponding to obstacle 1, obstacle 2, and obstacle 4. Based on the MOT algorithm, the obstacle point cloud corresponding to obstacle 1 in the point cloud at time k and the obstacle point cloud corresponding to obstacle 1 in the point cloud at time k+1 can be determined, and a unique ID is assigned to these two obstacle point clouds, such as ID=1; the obstacle point cloud corresponding to obstacle 2 in the point cloud at time k and the obstacle point cloud corresponding to obstacle 2 in the point cloud at time k+1 can also be determined, and a unique ID is assigned to these two obstacle point clouds, such as ID=2.

[0051] The point cloud matching algorithm proposed in the embodiment of the present disclosure has the main goal of finding a transformation matrix, which contains information about the rotation matrix R and the translation vector t, so that under the transformation of this transformation matrix, for the point cloud of the same obstacle, the point cloud at the earlier moment in adjacent moments is aligned as much as possible with the point cloud at the later moment in adjacent moments.

[0052] In some embodiments, based on the information of the same obstacle at different times, the above step S120 can be used to construct a factor graph. In the factor graph, the obstacle point cloud of the same obstacle at multiple times is used to form a point cloud geometric feature constraint for the transformation matrix; and the obstacle point cloud and the obstacle bounding box are used to form a bounding box constraint for the transformation matrix. Based on the above point cloud geometric feature constraint and bounding box constraint, the factor graph is optimized to achieve matching of the point clouds of the same obstacle at different times, which can improve the accuracy of point cloud matching.

[0053] The principle of Factor Graph Optimization (FGO) is based on graph theory and probabilistic graph models, and is mainly used to deal with large-scale nonlinear optimization problems. Its core idea is to use factor graphs to describe the relationships and constraints between multiple variables, and then use nonlinear optimization algorithms to solve each variable. Factor graphs are composed of two basic concepts: nodes and edges. Figure 2 It is a schematic diagram of the factor graph optimization principle, such as Figure 2 As shown in the figure, both circles and ovals represent nodes. The nodes represented by circles are variable nodes, i.e. nodes that need to be solved, such as the position of the vehicle; the nodes represented by ovals are constraint nodes, representing observation constraints, such as common road signs. The lines connecting the nodes represent edges, which represent the constraints between the nodes they connect.

[0054] When optimizing the factor graph, the embodiment of the present disclosure may adopt a nonlinear optimization method to iterate the values ​​of the nodes in the factor graph multiple times until a convergence condition is met; the convergence condition includes at least one of the following:

[0055] (1) The number of iterations is greater than or equal to the threshold;

[0056] (2) The values ​​of the adjusted nodes satisfy the constraints.

[0057] The nonlinear optimization algorithm includes at least one of the following: gradient descent algorithm, Newton algorithm, quasi-Newton algorithm, etc. The optimization direction may also be referred to as a search direction. When the optimization algorithm is a gradient descent algorithm, the optimization direction or the search direction may be a negative gradient direction or a gradient direction, and the step size may be an adjustment amplitude.

[0058] By using the above nonlinear optimization method to iteratively optimize the factor graph, the speed of factor graph optimization can be improved, thereby improving the speed and efficiency of point cloud matching.

[0059] The optimization process includes multiple iterations. During each iteration, the values ​​in the nodes are adjusted, and calculations are performed based on the adjusted nodes to determine whether the values ​​of the adjusted nodes meet the constraints. If not, the next iteration is performed, that is, the values ​​in the nodes are adjusted again, and it is determined again whether the values ​​of the adjusted nodes meet the constraints. This iteration is repeated until the convergence conditions are met, and the optimization process ends.

[0060] When the optimization process ends, if the values ​​of the nodes meet the constraints, the optimization of the factor graph is considered successful; if the number of iterations reaches the threshold but the constraints are still not met, the optimization of the factor graph is considered to have failed.

[0061] The disclosed embodiment utilizes obstacle point clouds, obstacle postures, and obstacle bounding boxes at multiple times to construct and optimize a factor graph, and achieves matching of obstacle point clouds at different times based on the optimized factor graph; it is able to combine the geometric constraints between obstacle point clouds at different times with the detection constraints of the 3D detection module to achieve stable matching of obstacle point clouds.

[0062] In some examples, constructing the factor graph includes:

[0063] Constructing multiple nodes in the factor graph, the multiple nodes include: obstacle point clouds, obstacle poses, and obstacle bounding boxes at each moment;

[0064] Constructing constraints between nodes, the constraints including: a first constraint between obstacle point clouds and obstacle poses at adjacent moments, and constraints between at least two of the obstacle point clouds, obstacle poses, and obstacle bounding boxes;

[0065] A factor graph is determined based on the plurality of nodes and the constraint condition.

[0066] The disclosed embodiment utilizes static point clouds, obstacle point clouds, obstacle poses and ego-vehicle poses at various moments to construct a factor graph, and locates the ego-vehicle based on the factor graph; it is capable of comprehensively utilizing the geometric feature constraints of the point clouds (corresponding to the first constraint condition) and the bounding box constraints (corresponding to the second constraint condition) to perform point cloud matching, thereby improving the accuracy of point cloud matching.

[0067] Figure 3 is an example 1 of a factor graph constructed according to an embodiment of the present disclosure. Figure 3 As shown, the nodes in the factor graph include the following:

[0068] Obstacle point cloud, obstacle pose and obstacle bounding box

[0069] The obstacle point cloud (P k ), obstacle posture (T k ) and obstacle bounding box (x k );

[0070] The obstacle point cloud (P k+1 ), obstacle posture (T k+1 ) and obstacle bounding box (x k+1 ).

[0071] The factor graph shows nodes at two adjacent moments. In some embodiments, the factor graph includes nodes at multiple consecutive moments, and there are multiple groups of adjacent moments in the multiple moments. For example, the factor graph is formed between the node at moment k and the node at moment k+1, the factor graph is formed between the node at moment k+1 and the node at moment k+2, and so on; in one example, when performing point cloud matching calculations, the factor graph formed by each group of nodes at adjacent moments is optimized. Figure 3 This is only an example and does not represent a limitation on the number of corresponding moments of nodes in the factor graph. In addition, the symbols representing each node in this embodiment are only examples and are not intended to limit the present disclosure.

[0072] like Figure 3 As shown, the constraints in the factor graph include the first constraint and the constraints between at least two items among the obstacle point cloud, obstacle pose and obstacle bounding box. Figure 3 e in 1 ) can represent the constraint factor between the obstacle point cloud and obstacle pose at adjacent moments, including the obstacle point cloud and obstacle pose at the previous moment in the adjacent moments, and the obstacle point cloud and obstacle pose at the subsequent moment in the adjacent moments. The first constraint condition can be called a geometric feature constraint, and the constraint condition between at least two of the obstacle point cloud, obstacle pose and obstacle bounding box can be called a bounding box constraint, which is as follows:

[0073] (I) Point cloud geometric feature constraints

[0074] In some examples, constructing a first constraint condition includes:

[0075] Based on the obstacle poses at adjacent moments, determine the first pose transformation matrix of the obstacle point cloud at adjacent moments;

[0076] The position transformation matrix is ​​used to transform the obstacle point cloud at the previous moment in the adjacent moments to obtain a first intermediate value;

[0077] A first constraint condition is constructed, where the first constraint condition includes: a difference value between the first intermediate value and an obstacle point cloud at a later time in adjacent time moments is less than or equal to a first threshold value.

[0078] Taking the adjacent moments as moment k and moment k+1 as an example, moment k is the earlier moment and moment k+1 is the later moment. In some examples, the first constraint condition is expressed as:

[0079]

[0080] Among them, e 1 Approaching 0;

[0081] P k represents the obstacle point cloud at time k, P k+1 Represents the obstacle point cloud at time k+1;

[0082] T k represents the obstacle position at time k, T k+1 represents the obstacle position at time k+1;

[0083] T k+1,k represents the first pose transformation matrix of the obstacle point cloud at adjacent moments, that is, the pose transformation matrix of the obstacle point cloud (or obstacle) from moment k to moment k+1; T k+1,k According to T k and T k+1 Sure;

[0084] ||.|| 2 represents the square of the second-order norm;

[0085] i represents the identity of a point in the point cloud.

[0086] As in the above formula (1), the first intermediate value is equal to T k+1,k P k ; The square of the second-order norm of the difference between the first intermediate value and the obstacle point cloud at the subsequent moment approaches 0, which means that the difference between the first intermediate value and the obstacle point cloud at the subsequent moment is less than or equal to the first threshold.

[0087] It can be seen that the first constraint condition uses the obstacle point clouds of the same obstacle at adjacent times to establish geometric constraints between the obstacle point clouds at adjacent times, and uses it as a constraint condition for matching point clouds at different times.

[0088] (II) Bounding box constraints

[0089] like Figure 3 In the above, the constraint conditions between at least two items of the obstacle point cloud, obstacle pose and obstacle bounding box include a second constraint condition (such as Figure 3 e in 2 ) and the third constraint (such as Figure 3 e in 3 ). The second constraint condition is the constraint condition between the obstacle bounding box and the obstacle point cloud at the same time, which can be called the bounding box constraint of the same frame; the third constraint condition is the constraint condition between the obstacle bounding box and the obstacle point cloud at different times, which can be called the bounding box constraint between different frames. The following are detailed respectively:

[0090] 1. Bounding box constraints of the same frame:

[0091] In some examples, the second constraint condition includes: at the same time, each point in the obstacle point cloud is within the obstacle bounding box.

[0092] In some examples, the second constraint is expressed as:

[0093]

[0094] Among them, e 2 Approaching 0;

[0095] x k Represents the obstacle bounding box at time k;

[0096] represents the i-th point in the obstacle point cloud at time k, such as including the three-dimensional coordinates of the point, the coordinate system of the three-dimensional coordinates is a coordinate system with the center point of the point cloud as the origin;

[0097] Indicates that the coordinate system of the i-th point in the obstacle point cloud at time k is converted to a coordinate system with the center point of the obstacle bounding box as the origin;

[0098] Represents the distance between the i-th point in the k-th frame point cloud and the center point of the obstacle bounding box;

[0099] S represents the size of the obstacle bounding box, including three values ​​representing length, width, and height;

[0100] 0.5S represents the distance between the obstacle bounding box and the center point;

[0101] Ideally, the i-th point in the k-th frame point cloud is within the obstacle bounding box, that is, If the i-th point in the k-th frame point cloud is outside the obstacle bounding box, then

[0102] The principle of setting the second constraint is to find The points outside the obstacle bounding box are selected and the exceeding parts are adjusted to be close to 0.

[0103] 2. Bounding box constraints between different frames:

[0104] like Figure 3 As shown, the third constraint condition includes:

[0105] The constraint conditions between the obstacle pose and obstacle bounding box at the previous moment, and the obstacle pose and obstacle point cloud at the later moment, and / or the constraint conditions between the obstacle pose and obstacle point cloud at the previous moment, and the obstacle pose and obstacle bounding box at the later moment.

[0106] In one example, the third constraint condition is constructed, including:

[0107] Based on the obstacle poses at adjacent moments, determine the first pose transformation matrix and the second pose transformation matrix of the obstacle point cloud at adjacent moments;

[0108] Using the first pose transformation matrix, the obstacle bounding box at the previous moment is transformed to obtain a first transformed bounding box; each point in the obstacle point cloud at the subsequent moment is within the range of the first transformed bounding box; and / or,

[0109] The obstacle bounding box at the later time is transformed by using the second posture transformation matrix to obtain a second transformed bounding box; each point in the obstacle point cloud at the previous time is within the range of the second transformed bounding box.

[0110] like Figure 3 In the example shown, the third constraint condition includes at least one of the following two parts:

[0111] T k , T k+1 、x k and P k+1 The constraints between them are expressed as follows:

[0112] T k , T k+1 、x k+1 and P kThe constraints between them can be expressed by formula (3-2).

[0113] Specifically include the following parts:

[0114] Part I:

[0115]

[0116] Among them, e 3-1 Approaching 0;

[0117] T k represents the obstacle position at time k, T k+1 represents the obstacle position at time k+1;

[0118] T k+1,k T represents the first pose transformation matrix of the obstacle point cloud, that is, the pose transformation matrix of the obstacle point cloud (or obstacle) from time k to time k+1; k+1,k According to T k and T k+1 Sure;

[0119] x k Represents the obstacle bounding box at time k;

[0120] represents the i-th point in the obstacle point cloud at time k+1;

[0121] i represents the identity of a point in the point cloud.

[0122] As shown in formula (3-1), using T k+1,k x k Transform, that is, get the first transformation bounding box T k+1,k x k ; In an ideal state, the bounding box obtained after transformation (i.e., the first transformed bounding box T k+1,k x k ) should be able to enclose the obstacle point cloud at time k+1, that is, each point in the obstacle point cloud at time k+1 is within the range of the first transformation bounding box, that is, If the i-th point in the point cloud at time k+1 is outside the first transformed bounding box, then

[0123] if but

[0124] if but:

[0125]

[0126] The principle of setting the third constraint is to find The points outside the first transformation bounding box are adjusted so that the exceeding parts are close to 0.

[0127] Part II:

[0128]

[0129] Among them, e 3-2 Approaching 0;

[0130] T k represents the obstacle position at time k, T k+1 represents the obstacle position at time k+1;

[0131] T kk+1 T represents the second pose transformation matrix of the obstacle point cloud, that is, the pose transformation matrix of the obstacle point cloud (or obstacle) from time k+1 to time k; k,k+1 According to T k and T k+1 Determine, the second pose transformation matrix T k,k+1 is the first pose transformation matrix T k+1,k The inverse matrix of

[0132] x k+1 Represents the obstacle bounding box at time k+1;

[0133] represents the i-th point in the obstacle point cloud at time k;

[0134] i represents the identity of a point in the point cloud.

[0135] As in formula (3-2), using T k,k+1 x k+1 Transform, and get the second transformation bounding box T k,k+1 x k+1 ; In an ideal state, the bounding box obtained after transformation (i.e., the second transformed bounding box T k,k+1 x k+1 ) should be able to enclose the obstacle point cloud at time k, that is, each point in the obstacle point cloud at time k is within the range of the second transformation bounding box, that is, If the i-th point in the point cloud at the k-th time is outside the second transformed bounding box, then

[0136] if but

[0137] if but:

[0138]

[0139] The principle of setting the third constraint is to find The points outside the obstacle bounding box are selected and the exceeding parts are adjusted to be close to 0.

[0140] It can be seen that the second constraint and the third constraint use the relationship between the obstacle point cloud and the obstacle bounding box to establish the constraint between the obstacle point cloud and the obstacle bounding box; in this disclosure, the constraint between the obstacle point cloud and the obstacle bounding box at the same time is called the second constraint, and the constraint between the obstacle point cloud and the obstacle bounding box at different times is called the third constraint. The embodiment of the present disclosure combines the second constraint, the third constraint and the first constraint, and can use the obstacle bounding box as a matching factor to improve the accuracy of point cloud matching.

[0141] In addition to the above nodes and constraints, the factor graph proposed in the embodiment of the present disclosure may also include other nodes and constraints. In some embodiments, the nodes in the factor graph may also include: obstacle speeds at various moments; the constraints in the factor graph may also include at least one of the fourth constraint and the fifth constraint; wherein the fourth constraint includes the constraint between obstacle speeds at adjacent moments, and the fifth constraint includes the constraint between obstacle posture and obstacle speed at adjacent moments.

[0142] Figure 4 This is Example 2 of a factor graph constructed according to an embodiment of the present disclosure. Figure 4 As shown, the nodes in the factor graph may also include:

[0143] The obstacle speed (v k ) and the obstacle velocity at time k+1 (v k+1 ). The aforementioned symbols are only examples and are not intended to limit the present disclosure.

[0144] like Figure 4 As shown, the constraints in the factor graph may also include a fourth constraint and / or a fifth constraint. In some examples, the fourth constraint includes: the difference value of the obstacle speed at adjacent moments is less than or equal to the second threshold.

[0145] Taking the adjacent moments as moment k and moment k+1 as an example, moment k is the earlier moment and moment k+1 is the later moment. In some examples, the fourth constraint condition is expressed as:

[0146] e 4 =v k+1 -v k (4)

[0147] Among them, e4 Approaching the zero vector;

[0148] v k+1 represents the speed of the obstacle at time k+1, v k Represents the speed of the obstacle at the dynamic moment k; the speed can be a three-dimensional vector, and the three elements in the three-dimensional vector respectively represent the speed of the dynamic obstacle in the directions of the three coordinate axes.

[0149] As shown in the above formula (4), the difference between the speed of the dynamic obstacle at time k+1 and the speed at time k approaches the zero vector, which means that the difference in the speed of the dynamic obstacle at adjacent times is less than.

[0150] In some examples, the fifth constraint condition is constructed in the following manner:

[0151] Determine a third posture transformation matrix based on obstacle speeds at adjacent moments and time intervals between adjacent moments;

[0152] Based on the third posture transformation matrix, transform the posture of the obstacle at a previous moment in the adjacent moments to obtain a second intermediate value;

[0153] A fifth constraint condition is constructed, and the fifth constraint condition includes: a difference value between the second intermediate value and the obstacle posture at a later moment in adjacent moments is less than or equal to a third threshold.

[0154] like Figure 4 As shown, taking the adjacent moments as moment k and moment k+1 as an example, moment k is the earlier moment and moment k+1 is the later moment. In some examples, the fifth constraint condition is expressed as:

[0155] e 5 =(T k+1 ) -1 exp(v k △t)T k (5)

[0156] Among them, e 5 Approaching the identity matrix;

[0157] T k represents the obstacle position at time k, T k+1 represents the obstacle posture at time k+1; the obstacle posture can be the posture in the world coordinate system;

[0158] v k represents the obstacle speed at time k (approximately the obstacle speed at time k+1);

[0159] △t represents the time interval between two adjacent moments;

[0160] exp(), which means mapping the vector in the brackets to a matrix;

[0161] As shown in equation (5), the third pose transformation matrix is ​​equal to exp(v k △t); Based on the third posture transformation matrix, the obstacle posture at the previous moment is transformed to obtain the second intermediate value; the second intermediate value is equal to exp(v k Ideally, the second intermediate value is equal to the obstacle posture at the later time; therefore, in equation (5), the product of the inverse matrix of the obstacle posture at the later time and the second intermediate value approaches the unit matrix, which means that the difference between the second intermediate value and the obstacle posture at the later time is less than or equal to the third threshold.

[0162] The fourth and fifth constraints are constraints based on kinematic principles. The fourth and fifth constraints are combined with the aforementioned constraints to jointly constrain the optimization of the factor graph, which can improve the accuracy and robustness of point cloud matching.

[0163] The above describes a method for laser radar positioning of a self-vehicle using a factor graph optimization method. The disclosed embodiment can also set corresponding weights for each constraint condition, and optimize the factor graph based on each weight.

[0164] For example, setting the overall constraints of the factor graph is:

[0165] E=w 1 e 1 +w 2 e 5 +w 3-1 e 3-1 +w 3-2 e 3-2 +w 4 e 4 +w 5 e 5 (6)

[0166] Among them, w 1 、w 2 、w 3-1 and w 3-2 、w 4 、w 5 They are the weights of the first constraint, the second constraint, the third constraint, the fourth constraint, and the fifth constraint. By setting the weights of each constraint, different importance levels can be set for different constraints, so that in the factor graph optimization process, optimization can be performed with emphasis according to different importance levels.

[0167] The following is an exemplary description of multiple iterations:

[0168] by Figure 4 Take the factor graph shown as an example, Figure 4 In the figure, the nodes represented by circles are constraint nodes, and the nodes represented by ovals are variable nodes. During the factor graph optimization process, the values ​​of constraint nodes remain unchanged, and the values ​​of variable nodes are adjusted in each iteration, so that the nodes after adjustment finally meet the constraints between nodes.

[0169] After constructing the factor graph, first determine the fixed values ​​of each constraint node and the initial value of each variable node. In one example, the initial value of the obstacle pose is determined by the orientation and center point position of the bounding box detected by the model, the obstacle speed can be determined by observation or empirical value, and the initial value of the obstacle bounding box is determined by the orientation and center point position of the detection box detected by the model.

[0170] In the first iteration, adjust the initial values ​​of each variable node (such as Figure 4 The obstacle speed, obstacle posture, and obstacle bounding box in the algorithm are used to determine whether the first to fifth constraint conditions are satisfied between the nodes using the adjusted variable nodes and constraint nodes; if all or part of the constraints are not satisfied, the second iteration process is entered;

[0171] In the second iteration, the values ​​of each variable node (the values ​​adjusted in the first iteration) are further adjusted, and the adjusted variable nodes and constraint nodes are used to determine whether the first to fifth constraint conditions are satisfied between the nodes; if all or part of the constraints are not satisfied, the third iteration is entered;

[0172] Until the variable nodes and constraint nodes after adjustment meet the first to fifth constraints, the factor graph optimization is considered successful. Based on the obstacle pose (including direction and position) after the last adjustment, the matching relationship of the obstacle point cloud at different times can be determined, thereby achieving the matching of the obstacle point cloud. If the number of iterations reaches the preset number threshold, and the nodes still do not meet all the constraints from the first to the fifth constraints, the factor graph optimization is considered to have failed.

[0173] In the factor graph optimization process, the numerical values ​​corresponding to multiple variable nodes can be adjusted based on a nonlinear optimization algorithm to obtain multiple adjusted values ​​of variable nodes, which can include: determining the optimization direction and step size corresponding to multiple variable nodes based on the nonlinear optimization algorithm and multiple constraints in the factor graph; and obtaining the values ​​of the optimized variable nodes based on the optimization direction and step size. The nonlinear optimization algorithm includes at least one of the following: a gradient descent algorithm, a Newton algorithm, a quasi-Newton algorithm, etc. The optimization direction can also be referred to as a search direction. When the optimization algorithm is a gradient descent algorithm, the optimization direction or the search direction can be a negative gradient direction or a gradient direction, and the step size can be the adjustment amplitude.

[0174] Whether a constraint is satisfied can be determined based on the threshold corresponding to the constraint. Among them, one or more thresholds corresponding to each constraint can be set according to the actual situation, and one or more thresholds corresponding to one or more constraints can be the same or different, which is not limited here. In addition, setting different weights for constraints can be reflected in different settings of the thresholds corresponding to the constraints; if the threshold corresponding to a constraint is set more strictly, it means that the weight of the constraint is higher; if the threshold corresponding to a constraint is set more loosely, it means that the weight of the constraint is lower.

[0175] The disclosed embodiment also provides a point cloud matching device. Figure 5 is a schematic diagram of the structure of a point cloud matching device 500 according to an embodiment of the present disclosure, comprising:

[0176] Point cloud acquisition module 510, used to acquire obstacle point clouds at multiple moments;

[0177] A factor graph construction module 520, configured to construct a factor graph using the obstacle point clouds, obstacle poses and obstacle bounding boxes at the multiple moments;

[0178] A factor graph optimization module 530, used to optimize the factor graph;

[0179] The matching module 540 is used to match the obstacle point clouds at the multiple moments based on the optimized factor graph.

[0180] In some implementations, the factor graph construction module 520 is used to:

[0181] Construct multiple nodes in the factor graph, the multiple nodes include: obstacle point clouds, obstacle poses and obstacle bounding boxes at each moment;

[0182] Constructing constraints between nodes, the constraints including: a first constraint between obstacle point clouds and obstacle poses at adjacent moments, and constraints between at least two of the obstacle point clouds, obstacle poses, and obstacle bounding boxes;

[0183] The factor graph is determined based on the plurality of nodes and the constraint condition.

[0184] In some implementations, the factor graph construction module 520 is used to:

[0185] Based on the obstacle poses at adjacent moments, determine the first pose transformation matrix of the obstacle point cloud at adjacent moments;

[0186] Using the first pose transformation matrix, transform the obstacle point cloud at the previous moment in the adjacent moments to obtain a first intermediate value;

[0187] A first constraint condition is constructed, where the first constraint condition includes: a difference value between the first intermediate value and an obstacle point cloud at a later time in adjacent time moments is less than or equal to a first threshold value.

[0188] In some implementations, the constraints between at least two of the obstacle point cloud, the obstacle pose, and the obstacle bounding box include:

[0189] At least one of the second constraint and the third constraint; wherein,

[0190] The second constraint condition is a constraint condition between the obstacle bounding box and the obstacle point cloud at the same time;

[0191] The third constraint condition includes: the constraint condition between the obstacle pose and obstacle bounding box at the previous moment in adjacent moments, and the obstacle pose and obstacle point cloud at the subsequent moment in adjacent moments, and / or the constraint condition between the obstacle pose and obstacle point cloud at the previous moment in adjacent moments, and the obstacle pose and obstacle bounding box at the subsequent moment in adjacent moments.

[0192] In some implementations, the second constraint condition includes: at the same time, each point in the obstacle point cloud is within the obstacle bounding box.

[0193] In some implementations, the factor graph construction module 520 is used to:

[0194] Based on the obstacle poses at adjacent moments, determine the first pose transformation matrix and the second pose transformation matrix of the obstacle point cloud at adjacent moments;

[0195] The obstacle bounding box at the previous moment is transformed using the first pose transformation matrix to obtain a first transformed bounding box; each point in the obstacle point cloud at the subsequent moment is within the range of the first transformed bounding box; and / or,

[0196] The obstacle bounding box at the later time is transformed by using the second posture transformation matrix to obtain a second transformed bounding box; each point in the obstacle point cloud at the previous time is within the range of the second transformed bounding box.

[0197] In some embodiments, the node further includes: obstacle speed at each moment;

[0198] The constraint condition also includes at least one of the fourth constraint condition and the fifth constraint condition; wherein,

[0199] The fourth constraint condition includes a constraint condition between obstacle speeds at adjacent moments;

[0200] The fifth constraint condition includes a constraint condition between the obstacle posture and obstacle speed at adjacent moments.

[0201] In some implementations, the fourth constraint condition includes: a difference value of obstacle speeds at adjacent moments is less than or equal to a second threshold.

[0202] In some implementations, the factor graph construction module 520 is used to:

[0203] Determine a third posture transformation matrix based on obstacle speeds at adjacent moments and time intervals between adjacent moments;

[0204] Based on the third posture transformation matrix, transform the posture of the obstacle at a previous moment in the adjacent moments to obtain a second intermediate value;

[0205] A fifth constraint condition is constructed, and the fifth constraint condition includes: a difference value between the second intermediate value and the obstacle posture at a later moment in adjacent moments is less than or equal to a third threshold.

[0206] In some implementations, the point cloud acquisition module 510 is used to:

[0207] Using the vehicle's LiDAR system, point clouds are acquired at multiple adjacent moments.

[0208] Based on the point clouds respectively acquired at multiple adjacent moments, obstacle point clouds of the same obstacle at the multiple moments are determined.

[0209] In some implementations, the factor graph optimization module 530 is configured to:

[0210] A nonlinear optimization method is used to iterate the values ​​of the nodes in the factor graph for multiple times until a convergence condition is satisfied; the convergence condition includes at least one of the following:

[0211] The number of iterations is greater than or equal to the threshold;

[0212] The adjusted node values ​​satisfy the constraints.

[0213] In some implementations, the factor graph optimization module 530 is configured to:

[0214] Set corresponding weights for the first constraint, the second constraint, the third constraint, the fourth constraint and the fifth constraint respectively;

[0215] Based on the various weights, the factor graph is optimized.

[0216] For the description of specific functions and examples of each module and submodule of the device in the embodiment of the present disclosure, reference can be made to the relevant description of the corresponding steps in the above method embodiment, which will not be repeated here.

[0217] In the technical solution disclosed in the present invention, the acquisition, storage and application of personal information of users involved are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0218] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium and a computer program product.

[0219] Figure 6 A schematic block diagram of an example electronic device 600 that can be used to implement an embodiment of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0220] like Figure 6 As shown, the device 600 includes a computing unit 601, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 602 or a computer program loaded from a storage unit 608 into a random access memory (RAM) 603. In the RAM 603, various programs and data required for the operation of the device 600 can also be stored. The computing unit 601, the ROM 602, and the RAM 603 are connected to each other via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0221] A number of components in the device 600 are connected to the I / O interface 605, including: an input unit 606, such as a keyboard, a mouse, etc.; an output unit 607, such as various types of displays, speakers, etc.; a storage unit 608, such as a disk, an optical disk, etc.; and a communication unit 609, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 609 allows the device 600 to exchange data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0222] The computing unit 601 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 601 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 601 performs the various methods and processes described above, such as detection methods. For example, in some embodiments, the detection method may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 608. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 600 via ROM 602 and / or communication unit 609. When the computer program is loaded into RAM 603 and executed by the computing unit 601, one or more steps of the detection method described above may be performed. Alternatively, in other embodiments, the computing unit 601 may be configured to perform the detection method in any other appropriate manner (e.g., by means of firmware).

[0223] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0224] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0225] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0226] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0227] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0228] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship with each other. The server may be a cloud server, a server of a distributed system, or a server combined with a blockchain.

[0229] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0230] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the principles of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A point cloud matching method, comprising: Obtain obstacle point clouds at multiple times; Constructing a factor graph using the obstacle point clouds, obstacle poses, and obstacle bounding boxes at the plurality of moments; Optimizing the factor graph; Based on the optimized factor graph, the obstacle point clouds at the multiple moments are matched.

2. The method according to claim 1, wherein: The step of constructing a factor graph by using the obstacle point clouds, obstacle poses and obstacle bounding boxes at the plurality of moments includes: Constructing a plurality of nodes in the factor graph, the plurality of nodes comprising: the obstacle point cloud, the obstacle pose and the obstacle bounding box at each moment; Constructing constraints between the nodes, the constraints including: a first constraint between the obstacle point cloud and the obstacle pose at adjacent moments, and a constraint between at least two of the obstacle point cloud, the obstacle pose and the obstacle bounding box; The factor graph is determined based on the plurality of nodes and the constraints.

3. The method according to claim 2, wherein: Constructing the first constraint condition includes: Based on the obstacle pose at adjacent moments, determining a first pose transformation matrix of the obstacle point cloud at the adjacent moments; Using the first posture transformation matrix, transforming the obstacle point cloud at a previous moment in the adjacent moments to obtain a first intermediate value; The first constraint condition is constructed, and the first constraint condition includes: a difference value between the first intermediate value and the obstacle point cloud at a later moment in the adjacent moments is less than or equal to a first threshold.

4. The method according to claim 2 or 3, wherein: The constraint conditions between at least two of the obstacle point cloud, the obstacle pose and the obstacle bounding box include: At least one of the second constraint and the third constraint; wherein, The second constraint condition is a constraint condition between the obstacle bounding box and the obstacle point cloud at the same time; The third constraint condition includes: the constraint condition between the obstacle pose and obstacle bounding box at the previous moment in the adjacent moments, and the obstacle pose and obstacle point cloud at the subsequent moment in the adjacent moments, and / or the constraint condition between the obstacle pose and obstacle point cloud at the previous moment in the adjacent moments, and the obstacle pose and obstacle bounding box at the subsequent moment in the adjacent moments.

5. The method according to claim 4, wherein: The second constraint condition includes: at the same time, each point in the obstacle point cloud is within the obstacle bounding box.

6. The method according to claim 4, wherein: Constructing the third constraint condition includes: Based on the obstacle poses at adjacent moments, determining a first pose transformation matrix and a second pose transformation matrix of the obstacle point cloud at the adjacent moments; The obstacle bounding box at the previous moment is transformed using the first posture transformation matrix to obtain a first transformed bounding box; each point in the obstacle point cloud at the subsequent moment is within the range of the first transformed bounding box; and / or, The obstacle bounding box at the later time is transformed by using the second posture transformation matrix to obtain a second transformed bounding box; each point in the obstacle point cloud at the previous time is within the range of the second transformed bounding box.

7. The method according to any one of claims 2 to 6, wherein: The node also includes: the speed of the obstacle at each moment; The constraint condition also includes at least one of the fourth constraint condition and the fifth constraint condition; wherein, The fourth constraint condition includes a constraint condition between the obstacle speeds at adjacent moments; The fifth constraint condition includes a constraint condition between the obstacle posture and the obstacle speed at the adjacent moments.

8. The method according to claim 7, wherein: The fourth constraint condition includes: a difference value of the obstacle speed at adjacent moments is less than or equal to a second threshold.

9. The method according to claim 7 or 8, wherein: Constructing the fifth constraint condition includes: Determine a third posture transformation matrix based on the obstacle speeds at the adjacent moments and the time intervals between the adjacent moments; Based on the third posture transformation matrix, transform the posture of the obstacle at a previous moment in the adjacent moments to obtain a second intermediate value; The fifth constraint condition is constructed, and the fifth constraint condition includes: a difference value between the second intermediate value and the obstacle posture at a later moment in the adjacent moments is less than or equal to a third threshold.

10. The method according to any one of claims 1 to 9, wherein: The step of obtaining obstacle point clouds at multiple moments includes: Using the vehicle's LiDAR system, point clouds are acquired at multiple adjacent moments. Based on the point clouds respectively acquired at a plurality of adjacent moments, the obstacle point clouds of the same obstacle at the plurality of moments are determined.

11. The method according to any one of claims 7 to 9, wherein: The step of optimizing the factor graph comprises: A nonlinear optimization method is used to iterate the values ​​of the nodes in the factor graph for multiple times until a convergence condition is satisfied; the convergence condition includes at least one of the following: The number of iterations is greater than or equal to the threshold; The adjusted value of the node satisfies the constraint condition.

12. The method according to claim 11, wherein: The step of optimizing the factor graph comprises: Set corresponding weights for the first constraint condition, the second constraint condition, the third constraint condition, the fourth constraint condition, and the fifth constraint condition respectively; Based on the respective weights, the factor graph is optimized.

13. A point cloud matching device, comprising: Point cloud acquisition module, used to obtain obstacle point clouds at multiple times; A factor graph construction module, used to construct a factor graph using the obstacle point clouds, obstacle poses and obstacle bounding boxes at the plurality of moments; A factor graph optimization module, used for optimizing the factor graph; The matching module is used to match the obstacle point clouds at the multiple moments based on the optimized factor graph.

14. The device according to claim 13, wherein: The factor graph building module is used to: Constructing a plurality of nodes in the factor graph, the plurality of nodes comprising: the obstacle point cloud, the obstacle pose and the obstacle bounding box at each moment; Constructing constraints between the nodes, the constraints including: a first constraint between the obstacle point cloud and the obstacle pose at adjacent moments, and a constraint between at least two of the obstacle point cloud, the obstacle pose and the obstacle bounding box; The factor graph is determined based on the plurality of nodes and the constraints.

15. The device according to claim 14, wherein: The factor graph building module is used to: Based on the obstacle pose at adjacent moments, determining a first pose transformation matrix of the obstacle point cloud at the adjacent moments; Using the first posture transformation matrix, transforming the obstacle point cloud at a previous moment in the adjacent moments to obtain a first intermediate value; The first constraint condition is constructed, and the first constraint condition includes: a difference value between the first intermediate value and the obstacle point cloud at a later moment in the adjacent moments is less than or equal to a first threshold.

16. The device according to claim 14 or 15, wherein: The constraint conditions between at least two items of the obstacle point cloud, the obstacle pose and the obstacle bounding box include: At least one of the second constraint and the third constraint; wherein, The second constraint condition is a constraint condition between the obstacle bounding box and the obstacle point cloud at the same time; The third constraint condition includes: the constraint condition between the obstacle pose and obstacle bounding box at the previous moment in the adjacent moments, and the obstacle pose and obstacle point cloud at the subsequent moment in the adjacent moments, and / or the constraint condition between the obstacle pose and obstacle point cloud at the previous moment in the adjacent moments, and the obstacle pose and obstacle bounding box at the subsequent moment in the adjacent moments.

17. The device according to claim 16, wherein: The second constraint condition includes: at the same time, each point in the obstacle point cloud is within the obstacle bounding box.

18. The device according to claim 16, wherein: The factor graph building module is used to: Based on the obstacle poses at adjacent moments, determining a first pose transformation matrix and a second pose transformation matrix of the obstacle point cloud at the adjacent moments; The obstacle bounding box at the previous moment is transformed using the first posture transformation matrix to obtain a first transformed bounding box; each point in the obstacle point cloud at the subsequent moment is within the range of the first transformed bounding box; and / or, The obstacle bounding box at the later time is transformed by using the second posture transformation matrix to obtain a second transformed bounding box; each point in the obstacle point cloud at the previous time is within the range of the second transformed bounding box.

19. The device according to any one of claims 14 to 18, wherein: The node also includes: the speed of the obstacle at each moment; The constraint condition also includes at least one of the fourth constraint condition and the fifth constraint condition; wherein, The fourth constraint condition includes a constraint condition between the obstacle speeds at adjacent moments; The fifth constraint condition includes a constraint condition between the obstacle posture and the obstacle speed at the adjacent moments.

20. The device according to claim 19, wherein The fourth constraint condition includes: a difference value of the obstacle speed at adjacent moments is less than or equal to a second threshold.

21. The device according to claim 19 or 20, wherein: The factor graph building module is used to: Determine a third posture transformation matrix based on the obstacle speeds at the adjacent moments and the time intervals between the adjacent moments; Based on the third posture transformation matrix, transform the posture of the obstacle at a previous moment in the adjacent moments to obtain a second intermediate value; The fifth constraint condition is constructed, and the fifth constraint condition includes: a difference value between the second intermediate value and the obstacle posture at a later moment in the adjacent moments is less than or equal to a third threshold.

22. The device according to any one of claims 13 to 21, wherein: The point cloud acquisition module is used to: Using the vehicle's LiDAR system, point clouds are acquired at multiple adjacent moments. Based on the point clouds respectively acquired at a plurality of adjacent moments, the obstacle point clouds of the same obstacle at the plurality of moments are determined.

23. The device according to any one of claims 19 to 21, wherein: The factor graph optimization module is used to: A nonlinear optimization method is used to iterate the values ​​of the nodes in the factor graph for multiple times until a convergence condition is satisfied; the convergence condition includes at least one of the following: The number of iterations is greater than or equal to the threshold; The adjusted value of the node satisfies the constraint condition.

24. The device according to claim 23, wherein: The factor graph optimization module is used to: Set corresponding weights for the first constraint condition, the second constraint condition, the third constraint condition, the fourth constraint condition, and the fifth constraint condition respectively; Based on the respective weights, the factor graph is optimized.

25. An electronic device comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 12.

26. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1-12.

27. A computer program product comprising a computer program, which, when executed by a processor, implements the method according to any one of claims 1 to 12.

Images