Vehicle extrinsic parameter calibration method, device, equipment and storage medium

Abstract

This disclosure relates to a method, apparatus, device, and storage medium for vehicle extrinsic parameter calibration. The method includes acquiring a target point cloud based on an onboard LiDAR, wherein the target point cloud is formed by stitching together the poses of multiple frames of original point clouds after motion distortion compensation; acquiring a target image based on an onboard camera; constructing target constraints based on the target point cloud, target image, and original extrinsic parameters; and optimizing the target constraints using a nonlinear optimization algorithm to obtain the vehicle extrinsic parameters. This disclosure, by forming a target point cloud through pose stitching of multiple frames of original point clouds after motion distortion compensation, solves the problem that sparse single-frame LiDAR point clouds can only extract some basic geometric features, leading to inaccurate LiDAR point cloud and camera registration, thus improving the accuracy and safety of extrinsic parameter calibration for autonomous vehicles.

Description

Technical Field

[0001] This disclosure relates to the field of autonomous driving technology, and in particular to a method, apparatus, device, and storage medium for calibrating vehicle external parameters. Background Technology

[0002] In the field of autonomous driving technology, mechanical LiDAR and cameras are usually used simultaneously for mapping and localization. LiDAR can provide relatively accurate scene structure information, while cameras can provide rich scene texture information. By fusing the two sensors, sufficient scene structure and texture information can be obtained, which greatly helps in mapping and localization.

[0003] However, to fuse the outputs of the two sensors, the first problem to solve is the coordinate system registration of the two sensors, that is, how to transform the outputs of the two sensors into the same coordinate system. In practice, the coordinate system of the first frame of data output by the LiDAR is generally taken as the world coordinate system, and the camera output is transformed into the world coordinate system (i.e., the LiDAR coordinate system). Therefore, the accuracy of the relative pose (i.e., camera extrinsic parameters) between the mechanical LiDAR and the camera has a significant impact on the fusion effect between the two sensors.

[0004] Current mainstream methods for extrinsic parameter calibration between LiDAR and cameras can be divided into calibration-plate-based methods and target-less methods. Calibration-plate-based methods require pre-deploying a specific texture calibration plate within a fixed scene before data acquisition. Successful extrinsic parameter calibration depends heavily on the placement of the calibration plate and the rationality of the texture design, thus hindering its application in open road scenarios for autonomous vehicles. Target-less methods typically use only a single frame of LiDAR point cloud and camera registration. A single frame of LiDAR point cloud is sparse, yielding only basic geometric features and making it difficult to extract higher-level spatial location information, which negatively impacts the accuracy of extrinsic parameter calibration for autonomous vehicles. Summary of the Invention

[0005] To solve the above-mentioned technical problems, or at least partially solve them, this disclosure provides a method, apparatus, equipment, and storage medium for calibrating vehicle external parameters.

[0006] In a first aspect, embodiments of this disclosure provide a method for calibrating vehicle extrinsic parameters, including:

[0007] Acquire a target point cloud based on vehicle-mounted LiDAR, wherein the target point cloud is formed by stitching together the poses of multiple frames of original point clouds after motion distortion compensation.

[0008] Acquire target images based on vehicle-mounted cameras;

[0009] Target constraints are constructed based on the target point cloud, the target image, and the original extrinsic parameters.

[0010] The target constraints are optimized using a nonlinear optimization algorithm to obtain the vehicle's external parameters.

[0011] Secondly, embodiments of this disclosure provide a vehicle external parameter calibration device, comprising:

[0012] The first acquisition module is used to acquire a target point cloud based on a vehicle-mounted lidar, wherein the target point cloud is formed by stitching together the poses of multiple frames of original point clouds after motion distortion compensation.

[0013] The second acquisition module is used to acquire target images based on the vehicle-mounted camera;

[0014] The constraint construction module is used to construct target constraint conditions based on the target point cloud, the target image, and the original extrinsic parameters.

[0015] The extrinsic parameter acquisition module is used to optimize the target constraint conditions based on a nonlinear optimization algorithm to obtain the vehicle's extrinsic parameters.

[0016] Thirdly, embodiments of this disclosure provide an electronic device, the electronic device comprising: a processor; a memory for storing executable instructions of the processor; the processor being configured to read the executable instructions from the memory and execute the instructions to implement the vehicle extrinsic parameter calibration method provided in embodiments of this disclosure.

[0017] Fourthly, embodiments of this disclosure provide a computer-readable storage medium having a computer program stored thereon, the program being executed by a processor as described in embodiments of this disclosure for vehicle extrinsic parameter calibration.

[0018] The technical solution provided in this disclosure does not employ an extrinsic parameter calibration method based on a calibration board, thus improving the flexibility and practicality of extrinsic parameter calibration. It obtains a target point cloud formed by stitching together the poses of multiple frames of original point clouds after motion distortion compensation based on vehicle-mounted LiDAR. Then, it constructs target constraints based on the target point cloud, target image, and original extrinsic parameters, and optimizes the target constraints through a nonlinear optimization algorithm to obtain vehicle extrinsic parameters. This solves the problem that the single-frame LiDAR point cloud is relatively sparse, from which only some very basic geometric features can be extracted, leading to inaccurate LiDAR point cloud and camera registration, thereby improving the accuracy and safety of extrinsic parameter calibration for autonomous vehicles.

[0019] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0020] The accompanying drawings, which are incorporated in and form a part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure.

[0021] To more clearly illustrate the technical solutions in the embodiments of this disclosure or the prior art, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0022] Figure 1 A schematic diagram illustrating a vehicle external parameter calibration process provided in this embodiment of the disclosure;

[0023] Figure 2 A schematic diagram of a process for acquiring a target point cloud provided in an embodiment of this disclosure;

[0024] Figure 3 (a) in the figure is a distortion-free original image provided in an embodiment of this disclosure.

[0025] Figure 3 (b) in this disclosure is a curb mask provided in an embodiment of the present disclosure.

[0026] Figure 3 (c) is a roadside density distribution map provided in an embodiment of this disclosure.

[0027] Figure 4 (a) in the figure is a schematic diagram of the central axis of the cylinder.

[0028] Figure 4 (b) is a schematic diagram of the cross-section of the first type of cylindrical column along the axis.

[0029] Figure 4 (c) in the figure is a schematic diagram of the cross section of the second type of cylindrical column along the axis.

[0030] Figure 4 (d) in the figure is a schematic diagram of the cross section of the third type of cylindrical column along the axis.

[0031] Figure 5 This is a schematic diagram of the structure of a vehicle external parameter calibration device provided in an embodiment of the present disclosure;

[0032] Figure 6 This is a schematic diagram of an electronic device structure provided in an embodiment of the present disclosure; Detailed Implementation

[0033] To better understand the above-mentioned objectives, features, and advantages of this disclosure, the solutions disclosed herein will be further described below. It should be noted that, unless otherwise specified, the embodiments and features described herein can be combined with each other.

[0034] Numerous specific details are set forth in the following description in order to provide a full understanding of this disclosure, but this disclosure may also be implemented in other ways different from those described herein; obviously, the embodiments in the specification are only some, and not all, of the embodiments of this disclosure.

[0035] There are two main methods for extrinsic parameter calibration of autonomous vehicles: The first is the targert-based calibration method. This method involves placing one or more calibration boards with specific textures (e.g., QR codes) in an area detectable by both sensors within a fixed scene. Specific points on the calibration boards (e.g., the corner points of the QR code) are then found in the camera images and LiDAR point clouds using manual selection or digital image processing methods. This identifies corresponding points in multiple camera images and LiDAR point clouds, transforming the problem into a Proof-of-Number (PNP) problem to solve for the relative extrinsic parameters between the two sensors. This method requires pre-positioning the specific textured calibration boards within the scene before data acquisition. The success of extrinsic parameter calibration depends heavily on the placement of the calibration boards and the rationality of the texture design; therefore, this method is impractical in open road scenarios for autonomous driving applications.

[0036] The second type is the extrinsic calibration method that does not use a calibration board. This method constructs constraints based on the original image and a single-frame LiDAR point cloud. For example, it extracts line segments from the camera image and the single-frame LiDAR point cloud respectively to construct a minimum reprojection error constraint; or it projects the LiDAR point cloud onto the camera plane based on the camera model to construct a maximum cross-correlation coefficient constraint. This type of method typically uses only a single-frame LiDAR point cloud for camera registration. A single-frame LiDAR point cloud is relatively sparse, from which only some basic geometric features can be extracted. It is difficult to extract higher-level spatial location information, which is detrimental to the accuracy of extrinsic calibration for autonomous vehicles, affecting their safety and reliability.

[0037] To address the above issues, this disclosure provides a method for calibrating vehicle extrinsic parameters, which will be described in detail below.

[0038] Figure 1 This is a flowchart illustrating a vehicle extrinsic parameter calibration method provided in an embodiment of this disclosure. This method can be executed by a vehicle extrinsic parameter calibration device, which can be implemented using software and / or hardware, and is generally integrated into an electronic device. For example... Figure 1 As shown, the method mainly includes the following steps S102 to S108:

[0039] Step S102: Obtain the target point cloud based on the vehicle-mounted lidar.

[0040] In one embodiment, reference Figure 2 As shown, the acquisition of target point cloud based on vehicle-mounted LiDAR includes: acquiring multiple frames of original point cloud collected by vehicle-mounted LiDAR, and acquiring the pose and velocity state of each frame of the original point cloud based on the LiDAR odometry method.

[0041] Each frame of the original point cloud is subjected to motion distortion compensation based on its velocity state. The poses of each frame of the original point cloud after motion distortion compensation are then stitched together to obtain a dense laser point cloud, which is the target point cloud.

[0042] For example, multiple frames of original point clouds collected by the vehicle-mounted LiDAR are acquired, and the multiple frames of original point clouds are input into the LiDAR Otometric Calculation (LIO algorithm) frame by frame. The pose and velocity state of each frame of original point cloud are output. Motion distortion compensation is performed on each frame of original point cloud according to the velocity state. The multiple frames of original point clouds after motion distortion compensation are stitched together frame by frame according to the pose to obtain the target point cloud.

[0043] Step S104: Acquire the target image based on the vehicle-mounted camera.

[0044] In one embodiment, acquiring the target image based on the vehicle-mounted camera includes: acquiring the original image captured by the vehicle-mounted camera, performing distortion correction processing on the original image, and obtaining the target image.

[0045] For example, all frames of original images captured by the vehicle camera are obtained, and the single frame original image and camera intrinsic parameters are input into the distortion image algorithm to obtain the distortion-free original image, i.e., the target image.

[0046] Step S106: Construct target constraints based on the target point cloud, the target image, and the original extrinsic parameters.

[0047] In one embodiment, constructing target constraints based on the target point cloud, the target image, and the original extrinsic parameters includes: obtaining a first target density distribution map based on the target image; obtaining a target plane based on the target point cloud; obtaining a first target point cloud based on the target plane; and constructing a first constraint based on the first target density distribution map, the first target point cloud, and the original extrinsic parameters, wherein the target constraints include the first constraint.

[0048] In one embodiment, obtaining a first target density distribution map based on the target image includes: performing semantic segmentation on the target image based on a semantic segmentation algorithm to obtain a first target mask; performing dilation processing on the first target mask and assigning values ​​to obtain the first target density distribution map.

[0049] For example, the primary target could be a curb, sidewalk, etc., and semantic segmentation algorithms could include DeepLab, SegNet, etc. Taking a curb as an example, refer to... Figure 3 As shown, Figure 3 In the image (a), the original image after distortion correction is shown. Figure 3 (b) in the diagram represents the curb mask. Figure 3 (c) in the figure is the roadside density distribution map. The original image after distortion removal is input into the semantic segmentation algorithm for semantic segmentation to obtain the roadside mask with roadside semantics. The pixels in the roadside mask are all assigned a value of 255. The roadside mask after the assignment is continuously dilated 7 times. After the first dilation, the pixel is assigned a value of 127. After the second dilation, the pixel is assigned a value of 63. And so on. The pixels after the 7 dilations are assigned values ​​of 127, 63, 31, 15, 7, 3, and 1 respectively. Finally, a two-dimensional roadside density distribution map is obtained.

[0050] In one embodiment, obtaining a target plane based on the target point cloud includes: dividing the target point cloud into an octree mesh to obtain octree nodes; fitting the first point cloud points within the octree nodes using a random sampling consensus algorithm to obtain a fitting plane and the number of first point cloud points in the fitting plane; and obtaining a fusion plane and a target fitting plane based on the octree nodes and the fitting plane, wherein the target plane includes the fusion plane and the target fitting plane.

[0051] In one embodiment, the octree node includes a first octree node and a second octree node. Obtaining a fusion plane and a target fitting plane based on the octree node and the fitting plane includes: traversing the second octree nodes adjacent to the first octree node; determining the first octree node whose angle between the normal vectors of the fitting planes corresponding to the first and second octree nodes is less than a first threshold, and whose distance from the point cloud center of the first octree node to the fitting plane corresponding to the second octree node is less than a second threshold, and whose distance from the point cloud center of the second octree node to the fitting plane corresponding to the first octree node is less than a second threshold, as a target octree node; merging the point clouds of the target octree node and the point clouds of the second octree node to obtain a fusion plane; and determining the target fitting plane based on the fitting plane corresponding to the target octree node.

[0052] For example, multiple 0.4m*0.4m*0.4m grids are pre-set, and the dense laser point cloud (target point cloud) is divided into these 0.4m*0.4m*0.4m grids. Each 0.4m*0.4m*0.4m grid is used as an octree node. For all first point cloud points within each octree node, a Random Sampling Consensus (RANSAC) algorithm is used for plane fitting, outputting the fitted plane, its parametric equations, and the number of first point cloud points within the fitted plane. The ratio of the number of first point cloud points in the fitted plane to the number of first point cloud points in the octree node is calculated. The set of first point cloud points in octree nodes with a ratio greater than 80% constitutes the fitted plane. For octree nodes with a ratio less than 80%, the point cloud formed by all first point cloud points within the octree node is obtained. The center coordinates of the grid containing the point cloud formed by all first point cloud points within the octree node are used as the dividing point, and the octree node's grid is divided into 8 equal sub-nodes. The aforementioned plane fitting is performed on each sub-node. This process is repeated 3-5 times until the fitting process stops, yielding all fitted planes. After determining the fitted planes, for each first octree node, all spatially adjacent second octree nodes are traversed one by one. Octrees whose normal vectors to the fitted planes corresponding to the first and second octree nodes are less than 10°, and whose distances from the point cloud center of the first octree node to its corresponding fitted plane of the second octree node are less than 0.01m, and whose distances from the point cloud center of the second octree node to its corresponding fitted plane of the first octree node are both less than 0.01m, are designated as target octree nodes. The point clouds of the target octree nodes and the second octree nodes are merged to obtain a fusion plane. The fitted plane corresponding to the target octree node is then defined as the target fitted plane. The set of the fusion plane and the target fitted plane is defined as the target plane, meaning the target plane includes both the fusion plane and the target fitted plane.

[0053] This invention proposes a novel method for searching planes in point clouds using an octree search algorithm. Compared to traditional octree-based search algorithms, this invention incorporates an octree node fusion step. For irregularly shaped planes, traditional octree search algorithms will fit the plane into multiple rectangles, while this algorithm can preserve the original shape of the plane.

[0054] In one embodiment, obtaining a first target point cloud based on the target plane includes: filtering the target plane to obtain candidate road surface point clouds; obtaining a bird's-eye view height map of the road surface point cloud based on the candidate road surface point clouds; detecting the bird's-eye view height map of the road surface point cloud based on an edge detection algorithm to obtain the edges of the bird's-eye view height map of the road surface point cloud; and projecting the pixels of the edges of the bird's-eye view height map of the road surface point cloud onto the target point cloud to obtain the first target point cloud.

[0055] For example, the target planes obtained above are filtered, and the point clouds corresponding to target planes whose normals and Z-axis angles are less than 10° are merged to form candidate road surface point clouds. Then, the eigenvector corresponding to the smallest eigenvalue of the coordinate covariance matrix of the candidate road surface point clouds is used as the normal vector of the candidate road surface. The height of each point cloud in the candidate road surface point cloud along the bird's-eye view direction of the road surface normal vector is calculated, and the height is linearly mapped to the range (0, 255) to form a bird's-eye view height map. Specifically, let the road surface normal vector be (nx, ny, nz), and the coordinates of a point cloud in the point cloud be (X, Y, Z), then the bird's-eye view height of that point cloud is nx*Y - ny*X. Then, an edge detection algorithm is used to detect edges on the bird's-eye view height map, extracting the edges. The pixels of the edges of the bird's-eye view height map are projected onto the dense laser point cloud, and the first point cloud point in the dense laser point cloud corresponding to the edge pixels is extracted to form a 3D curb point cloud.

[0056] This invention proposes a novel method for extracting roadside point clouds. Compared with traditional algorithms that fit roadside parameters, this invention does not require any specific shape or orientation of the roadside, and can extract any roadside point cloud, thus reducing the difficulty of extracting roadside point clouds and broadening the applicability of roadside point cloud extraction.

[0057] In one embodiment, constructing the first constraint condition based on the first target density distribution map, the first target point cloud, and the original extrinsic parameters includes: projecting a first point cloud point in the first target point cloud onto the first target density distribution map to obtain a first pixel point of the first point cloud point on the first target density distribution map; obtaining a first pixel brightness of the first point cloud point on the first target density distribution map based on the coordinates of the first pixel point; and constructing the first constraint condition based on the first pixel brightness.

[0058] For example, taking the roadside as the first target, the original extrinsic parameters are generally known or manually determined. These original extrinsic parameters can be the original extrinsic parameters of the vehicle-mounted camera, the calibration extrinsic parameters of the vehicle, or the original extrinsic parameters of the camera. Taking the original extrinsic parameters of the camera as an example, the original extrinsic parameters are (R,t), where rotation R and displacement t are used to project all roadside point cloud points into a two-dimensional roadside density distribution map Q, obtaining the coordinates Xi of the roadside point cloud points, and the coordinates xi of the first pixel point on the roadside density distribution map Q. Based on the coordinates xi of the first pixel point, the first pixel brightness pi of the roadside point cloud point on the roadside density distribution map Q is obtained. The first constraint condition is constructed based on the first pixel brightness pi. The expression for the coordinates xi of the first pixel point is: xi = f(Xi,R,t), the expression for the first pixel brightness is: pi = Q(xi), and the expression for the first constraint condition is:

[0059]

[0060] Where (R,t) are the original extrinsic parameters, Xi is the coordinate of the i-th first point cloud point, xi is the coordinate of the first pixel point, pi is the brightness of the first pixel, and n is the number of first point cloud points in the first target point cloud. This is the first constraint condition.

[0061] In one embodiment, the step of constructing target constraints based on the target point cloud, the target image, and the original extrinsic parameters further includes: preprocessing the target image to obtain a two-dimensional line segment diagram of the target image; obtaining a three-dimensional line segment diagram based on the target plane; constructing a first matching point pair based on the original extrinsic parameters, the two-dimensional line segment diagram, and the three-dimensional line segment diagram; and constructing a second constraint based on the first matching point pair, wherein the target constraints include the second constraint.

[0062] In one embodiment, obtaining a three-dimensional line segment diagram based on the target plane includes: obtaining the normal vectors between intersecting target planes; taking the intersection lines between target planes where the angle between the normal vectors of the intersecting target planes is greater than a third threshold as planar intersection lines, and all the planar intersection lines constitute the three-dimensional line segment diagram.

[0063] For example, based on the above embodiments, after obtaining the target plane, any target plane is selected, and for the octree node corresponding to the target plane, other octree nodes adjacent to the octree node corresponding to the target plane in space are traversed to obtain the angle between the normals of the target planes corresponding to the two octree nodes. If the angle is greater than 50°, the intersection line of the two target planes is determined to be the plane intersection line, and all the plane intersection lines constitute the three-dimensional line segment diagram.

[0064] In one embodiment, preprocessing the target image to obtain a two-dimensional line segment map of the target image includes: inputting the target image into an edge detection algorithm to obtain an edge map of the target image; and inputting the edge map into a Hough algorithm to obtain the two-dimensional line segment map of the target image.

[0065] For example, the edge detection algorithm can be the Canny edge detection algorithm, and the Hough algorithm can be the rough line segment detection algorithm. The target image is input into the Canny edge detection algorithm to obtain the edge map of the target image. The edge map of the target image is input into the rough line segment detection algorithm to obtain multiple first line segments corresponding to the edge map. All the first line segments constitute a two-dimensional line segment map of the target image.

[0066] In one embodiment, constructing a first matching point pair based on the original extrinsic parameters, the two-dimensional line segment diagram, and the three-dimensional line segment diagram includes: projecting a second point cloud from the three-dimensional line segment diagram onto the two-dimensional line segment diagram based on the original extrinsic parameters to obtain a second pixel of the second point cloud on the two-dimensional line segment diagram;

[0067] Determine the first vertical distance between the second pixel and all first line segments in the two-dimensional line segment diagram, determine the smallest first vertical distance as the first distance, and determine the second pixel whose first distance is less than a first preset threshold as the target second pixel.

[0068] The third pixel point is determined as the target third pixel point based on the foot of the perpendicular from the first line segment corresponding to the first distance to the target second pixel point.

[0069] Match the second point cloud corresponding to the second pixel of the target with the third pixel of the target to construct the first matching point pair.

[0070] In one embodiment, constructing the second constraint based on the first matching point pair includes: determining the first reprojection error based on the coordinates of the second target pixel and the third target pixel in the first matching point pair; and constructing the second constraint based on the first reprojection error.

[0071] For example, the range of the first preset threshold can be between 1 and 10 pixels, specifically 2 pixels, 4 pixels and 9 pixels. In the 3D line segment diagram, for all planar intersections, a second point cloud point is taken at intervals of 0.01m to obtain the set of second point cloud points, i.e., all second point cloud points. Based on the original extrinsic parameters, each second point cloud point in the set is projected onto the 2D line segment diagram to obtain the second pixel point of the second point cloud point on the 2D line segment diagram. The first perpendicular distance from each second pixel point to all first line segments in the 2D line segment diagram is calculated. The smallest first perpendicular distance is found among all the first perpendicular distances and determined as the first distance. Second pixel points with a first distance less than 10 pixels are determined as target second pixel points. Then, the third pixel point corresponding to the perpendicular foot of the target second pixel point on the first line segment corresponding to the first distance is determined as the target third pixel point. The second point cloud point corresponding to the target second pixel point is matched with the target third pixel point to construct the first matching point pair. The construction method of each first matching point pair is repeated to obtain the first matching point pair set. Here, the 2D line segment diagram includes multiple first line segments, and the first line segment is composed of third pixel points.

[0072] The first projection error is determined based on the coordinates of the second and third pixels of the target at the first matching point; the second constraint is constructed based on the first projection error.

[0073] The expression for the coordinates of the second pixel is:

[0074] ;

[0075] The expression for the first projection error is:

[0076] =

[0077] The expression for the second constraint is:

[0078]

[0079] Where (R,t) are the original extrinsic parameters. Let i be the coordinates of the i-th second point cloud point. The coordinates of the second pixel of the target The coordinates of the third pixel of the target are First projection error, N The number of first matching pairs Second constraint.

[0080] In one embodiment, the step of constructing target constraints based on the target point cloud, the target image, and the original extrinsic parameters further includes: performing side-view processing on the target point cloud to obtain a three-dimensional side-view map; constructing a second matching point pair based on the original extrinsic parameters, the two-dimensional line segment map, and the three-dimensional side-view map; and constructing a third constraint based on the second matching point pair, wherein the target constraints include the third constraint.

[0081] In one embodiment, the step of performing side-view processing on the target point cloud to obtain a three-dimensional side-view map includes: filtering the target point cloud to obtain candidate second target point clouds; performing cylinder radius consistency verification and cylinder orientation completeness verification on the candidate second target point clouds, and determining the candidate second target point cloud that simultaneously satisfies the cylinder radius consistency verification and cylinder orientation completeness verification as the second target point cloud; and obtaining the three-dimensional side-view map based on the second target point cloud and the initial pose of the autonomous vehicle.

[0082] For example, the second target can be a cylinder, a utility pole, a tree trunk, etc.; taking a cylinder as an example, refer to... Figure 4 As shown, Figure 4 (a) in the diagram is a schematic diagram of the central axis of the cylinder. Figure 4 (b) in the diagram is a cross-sectional view of the first type of cylindrical column along the axis. Figure 4 (c) in the diagram is a cross-sectional view of the second type of cylindrical column along the axis. Figure 4(d) in the diagram is a cross-sectional view of the third type of cylindrical column along the axis. The specific process for filtering the target point cloud to obtain candidate second target point clouds is as follows: A 3D mesh is created according to a preset size of 0.1m*0.1m*0.1m, dividing the target point cloud into various meshes of the preset size; for each mesh, the eigenvalues ​​and eigenvectors of the covariance matrix of the target point cloud coordinates within the mesh are calculated. If the largest eigenvalue is greater than 10 times the largest of the other two eigenvalues, then that portion of the point cloud in the mesh is considered a candidate cylindrical point cloud; otherwise, refer to... Figure 4 As can be seen from (b), the cylindrical surface of the candidate cylinder is incomplete, and the point cloud within the grid does not constitute the point cloud of the candidate cylinder.

[0083] The candidate second target point clouds are subjected to cylinder radius consistency checks and cylinder orientation completeness checks. Candidate second target point clouds that simultaneously satisfy both checks are determined as the second target point clouds. Specifically, for each candidate cylindrical point cloud, the eigenvector corresponding to the largest eigenvalue of the covariance moment of its internal point cloud coordinates is taken as the orientation vector of the candidate cylindrical point cloud. The center of the candidate cylindrical point cloud is obtained as the cylinder center point, and the central axis of the candidate cylindrical point cloud is calculated based on the cylinder center point. After determining the central axis of the candidate cylindrical point cloud, cylinder radius consistency checks and cylinder orientation completeness checks are performed respectively.

[0084] The detailed process of cylinder radius consistency verification is as follows: Calculate the distance from each point in the candidate cylinder point cloud to the central axis of the candidate cylinder point cloud, sort the distances from smallest to largest, and if the maximum distance from a point in the point cloud to the central axis of the candidate cylinder is more than 10 times the minimum distance, refer to... Figure 4 From (c), we can see that the radius of the candidate cylindrical point cloud does not converge, meaning that the candidate cylindrical point cloud within this grid does not constitute a cylinder. Otherwise, the candidate cylindrical point cloud within this grid is considered to constitute a cylinder.

[0085] The completeness check for cylindrical orientation is performed as follows: Using the central axis of the candidate cylindrical point cloud as the normal, take its cross-section and calculate the angles between all point cloud points and the central axis. Divide the point cloud into bins every 10° along the central axis and place the point cloud points with angles to the central axis into those bins. If more than half of the bins contain the corresponding point cloud points, refer to... Figure 4 From (d) we can see that if the radius of the candidate cylindrical point cloud is convergent and the angle of the point cloud is relatively average, then the candidate cylindrical point cloud in the grid is considered to constitute a cylinder; otherwise, it is considered not to constitute a cylinder.

[0086] After simultaneously satisfying the cylinder radius consistency check and cylinder orientation completeness check, the candidate cylindrical point cloud is determined as the cylindrical point cloud. Then, based on the initial pose of the autonomous vehicle, the cylindrical point cloud is observed from the initial pose direction of the autonomous vehicle to obtain the three-dimensional side view of the cylindrical point cloud. All cylindrical point clouds are calculated in sequence to obtain all three-dimensional side view, and all three-dimensional side view are constructed into a three-dimensional side view map.

[0087] This invention proposes a novel method for extracting cylinders from point clouds. Compared to the traditional method using the RANSAC algorithm, this method performs cylinder radius consistency checks and cylinder orientation completeness checks, avoiding the false detection problem of the traditional RANSAC algorithm.

[0088] In one embodiment, constructing the second matching point pair based on the original extrinsic parameters, the two-dimensional line segment diagram, and the three-dimensional side view diagram includes:

[0089] Based on the original extrinsic parameters, the third point cloud in the three-dimensional side view map is projected onto the two-dimensional line segment map to obtain the fourth pixel of the third point cloud on the two-dimensional line segment map.

[0090] Determine the second vertical distance between the fourth pixel and all first line segments in the two-dimensional line segment diagram, determine the smallest second vertical distance as the second distance, and determine the fourth pixel whose second distance is less than the second preset threshold as the target fourth pixel.

[0091] The third pixel point is determined as the expected third pixel point based on the foot of the perpendicular from the target fourth pixel point on the first line segment corresponding to the second distance;

[0092] The third point cloud corresponding to the target fourth pixel is matched with the expected third pixel to construct a second matching point pair.

[0093] In one embodiment, constructing the third constraint based on the second matching point pair includes: determining a second projection error based on the coordinates of the target fourth pixel in the second matching point pair and the coordinates of the desired third pixel; and constructing the third constraint based on the second projection error.

[0094] For example, the range of the first preset threshold can be between 1 and 10 pixels, specifically 3 pixels, 7 pixels, and 8 pixels. In all sidelines of the 3D sideline view, a third point cloud is taken at intervals of 0.01m to obtain a set of third point cloud points, i.e., all third point cloud points. Based on the original extrinsic parameters, each third point cloud point in the set is projected onto a 2D line segment diagram to obtain the fourth pixel point of the third point cloud point on the 2D line segment diagram. The second perpendicular distance from each fourth pixel point to all first line segments in the 2D line segment diagram is calculated. The smallest second perpendicular distance is found among all second perpendicular distances and determined as the second distance. Fourth pixels with a second distance less than 10 pixels are determined as target fourth pixels. The third pixel point corresponding to the perpendicular foot of the target fourth pixel point on the first line segment corresponding to the second distance is determined as the desired third pixel point. The third point cloud point corresponding to the target fourth pixel point and the desired third pixel point are used to construct a second matching point pair. The construction method for each second matching point pair is repeated to obtain a set of second matching point pairs.

[0095] The second projection error is determined based on the coordinates of the fourth pixel of the target and the coordinates of the expected third pixel of the second matching point; the third constraint condition is constructed based on the second projection error.

[0096] The expression for the coordinates of the fourth pixel of the target is:

[0097] ;

[0098] The expression for the second projection error is:

[0099] =

[0100] The expression for the third constraint is:

[0101]

[0102] Where (R,t) are the original extrinsic parameters. Let j be the world coordinates of the third point cloud. The coordinates of the fourth pixel of the target are The coordinates of the desired third pixel are: The second projection error is denoted by m, and m is the number of the second matching point pairs. The third constraint.

[0103] Step S108: Optimize the target constraint conditions based on a nonlinear optimization algorithm to obtain the vehicle's external parameters.

[0104] For example, the nonlinear optimization algorithm includes the Gauss-Newton algorithm, the Levenberg-Marquardt algorithm, etc., and the objective constraint includes at least one of the first constraint, the second constraint, and the third constraint. The objective constraint also includes a fourth constraint. The vehicle extrinsic parameters can be camera extrinsic parameters or calibration extrinsic parameters of the vehicle camera. The values ​​of the vehicle extrinsic parameters are obtained according to actual needs.

[0105] In one embodiment, the first constraint condition is optimized based on a nonlinear optimization algorithm to obtain the vehicle's extrinsic parameters.

[0106] For example, the first constraint condition

[0107] The Gauss-Newton algorithm is used for optimization to obtain the extrinsic parameters of the first camera, which are then used as the extrinsic parameters of the vehicle.

[0108] In one embodiment, the second constraint condition is optimized based on a nonlinear optimization algorithm to obtain the vehicle's extrinsic parameters.

[0109] For example, the second constraint condition

[0110] The data is input into the Levenberg-Marquardt algorithm for optimization to obtain the extrinsic parameters of the second camera, which are then used as the extrinsic parameters of the vehicle.

[0111] In one embodiment, the third constraint condition is optimized based on a nonlinear optimization algorithm to obtain the vehicle's extrinsic parameters.

[0112] For example, the third constraint condition

[0113] The data is input into the Levenberg-Marquardt or Gauss-Newton algorithm for optimization to obtain the extrinsic parameters of the third camera, which are then used as the extrinsic parameters of the vehicle.

[0114] In one embodiment, the first constraint, the second constraint, and the third constraint are input into a nonlinear optimization algorithm for optimization to obtain the vehicle's extrinsic parameters.

[0115] For example, the first, second, and third constraints are weighted to obtain the overall optimization objective. ,Right now ,Will The data is input into a nonlinear optimization algorithm for optimization to obtain the extrinsic parameters of the fourth camera, which are then used as the extrinsic parameters of the vehicle.

[0116] As mentioned above, the vehicle extrinsic parameter calibration method provided in this disclosure optimizes the vehicle extrinsic parameters and improves their accuracy by constructing at least one of the first constraint condition, the second constraint condition, and the third constraint condition, thereby improving the safety of vehicle driving.

[0117] In summary, the vehicle extrinsic parameter calibration method provided in this embodiment does not require the use of a calibration board-based extrinsic parameter calibration method, thus improving the applicability of autonomous driving extrinsic parameter calibration. Furthermore, by extracting semantic features from stitched multi-frame LiDAR point clouds and constructing target constraints with single-frame target images, it overcomes the problem that single-frame LiDAR point clouds can only extract a small number of geometric features and have insufficient point cloud information. The target constraints further improve the accuracy of vehicle extrinsic parameters and help ensure the safety of vehicle operation.

[0118] Corresponding to the vehicle extrinsic parameter calibration method provided in this disclosure, this disclosure also provides a vehicle extrinsic parameter calibration device. Figure 4 This is a schematic diagram of a vehicle extrinsic parameter calibration device provided in an embodiment of the present disclosure. The device can be implemented by software and / or hardware, and is generally integrated into an electronic device, such as... Figure 4 As shown, the vehicle external parameter calibration device includes:

[0119] The first acquisition module is used to acquire a target point cloud based on a vehicle-mounted lidar, wherein the target point cloud is formed by stitching together the poses of multiple frames of original point clouds after motion distortion compensation.

[0120] The second acquisition module is used to acquire target images based on the vehicle-mounted camera;

[0121] The constraint construction module is used to construct target constraints based on the target point cloud, the target image, and the original extrinsic parameters.

[0122] The extrinsic parameter acquisition module is used to optimize the target constraint conditions based on a nonlinear optimization algorithm to obtain the vehicle's extrinsic parameters.

[0123] In one embodiment, the first acquisition module includes a first sub-acquisition module, which is used to acquire multiple frames of original point clouds collected by the vehicle-mounted lidar, and acquire the pose and velocity state of each frame of the original point cloud based on the laser odometry method; perform motion distortion compensation on each frame of the original point cloud according to the velocity state, and stitch the poses of the multiple frames of the original point cloud after motion distortion compensation to obtain the target point cloud.

[0124] In one embodiment, the second acquisition module includes a second sub-acquisition module, which is used to acquire the original image captured by the vehicle-mounted camera, perform distortion correction processing on the original image, and obtain the target image.

[0125] In one embodiment, the constraint construction module includes a first constraint construction module, which is used to obtain a first target density distribution map based on the target image; obtain a target plane based on the target point cloud; obtain a first target point cloud based on the target plane; and construct a first constraint condition based on the first target density distribution map, the first target point cloud, and the original extrinsic parameters.

[0126] In one embodiment, the first constraint construction module includes a target plane acquisition module, which is used to acquire a target plane based on the target point cloud.

[0127] In one embodiment, the target plane acquisition module includes a target plane acquisition submodule, which is used to divide the target point cloud into an octree mesh to obtain octree nodes; fit the first point cloud points within the octree nodes using a random sampling consensus algorithm to obtain a fitting plane and the number of first point cloud points in the fitting plane; and obtain a fusion plane and a target fitting plane based on the octree nodes and the fitting plane, wherein the target plane includes the fusion plane and the target fitting plane.

[0128] In one embodiment, the target plane acquisition submodule includes a fusion plane and a target fitting plane acquisition module, which is used to obtain the fusion plane and the target fitting plane based on the octree nodes and the fitting plane.

[0129] In one embodiment, the fusion plane and target fitting plane acquisition module includes a fusion plane and target fitting plane acquisition submodule. This submodule is used to traverse the second octree nodes adjacent to the first octree node, determine the first octree nodes whose angle between the normal vectors of the fitting planes corresponding to the first and second octree nodes is less than a first threshold, whose distance from the point cloud center of the first octree node to the fitting plane corresponding to the second octree node is less than a second threshold, and whose distance from the point cloud center of the second octree node to the fitting plane corresponding to the first octree node is less than a second threshold, and whose distance from the point cloud center of the second octree node to the fitting plane corresponding to the first octree node is less than a second threshold, as target octree nodes. The point clouds of the target octree nodes and the point clouds of the second octree nodes are merged to obtain a fusion plane; the target fitting plane is determined based on the fitting plane corresponding to the target octree node.

[0130] In one embodiment, the first constraint construction module includes a density distribution map acquisition module, which is used to acquire a first target density distribution map based on the target image.

[0131] In one embodiment, the density distribution map acquisition module includes a density distribution map acquisition submodule, which is used to perform semantic segmentation on the target image based on a semantic segmentation algorithm to obtain a first target mask; and to perform dilation processing on the first target mask and assign values ​​to obtain the first target density distribution map.

[0132] In one embodiment, the first constraint construction module includes a first target point cloud acquisition module, which is used to acquire a first target point cloud based on the target plane.

[0133] In one embodiment, the first target point cloud acquisition module includes a first target point cloud acquisition submodule, which is used to filter the target plane to acquire candidate road surface point clouds; acquire a bird's-eye view height map of the road surface point cloud based on the candidate road surface point clouds; detect the bird's-eye view height map of the road surface point cloud based on an edge detection algorithm to acquire the edges of the bird's-eye view height map of the road surface point cloud; and project the pixels of the edges of the bird's-eye view height map of the road surface point cloud onto the target point cloud to obtain the first target point cloud.

[0134] In one embodiment, the first constraint construction module includes a first constraint condition construction first sub-module, which is used to construct the first constraint condition based on the first target density distribution map, the first target point cloud, and the original extrinsic parameters.

[0135] In one embodiment, the first constraint construction submodule includes a second constraint construction submodule, which is used to project a first point cloud point in the first target point cloud onto a first target density distribution map based on the original extrinsic parameters, to obtain a first pixel of the first point cloud point on the first target density distribution map; to obtain a first pixel brightness of the first point cloud point on the first target density distribution map based on the coordinates of the first pixel; and to construct the first constraint based on the first pixel brightness, wherein the coordinate expression of the first pixel is:

[0136] The expression for the brightness of the first pixel is:

[0137] The expression for the first constraint is:

[0138]

[0139] Where (R,t) are the original extrinsic parameters, Xi is the world coordinate of the i-th first point cloud point, xi is the coordinate of the first pixel, pi is the brightness of the first pixel, and n is the number of first point cloud points in the first target point cloud. This is the first constraint condition.

[0140] In one embodiment, the constraint construction module includes a second constraint construction module, which is used to preprocess the target image to obtain a two-dimensional line segment diagram of the target image; obtain a three-dimensional line segment diagram based on the target plane; construct a first matching point pair based on the original extrinsic parameters, the two-dimensional line segment diagram, and the three-dimensional line segment diagram; and construct a second constraint condition based on the first matching point pair.

[0141] In one embodiment, the second constraint construction module includes a two-dimensional line segment diagram acquisition module, which is used to preprocess the target image to obtain a two-dimensional line segment diagram of the target image.

[0142] In one embodiment, the two-dimensional line segment map acquisition module includes a two-dimensional line segment map acquisition submodule, which is used to input the target image into an edge detection algorithm to acquire an edge map of the target image; and input the edge map into a Hough algorithm to acquire the two-dimensional line segment map of the target image.

[0143] In one embodiment, the second constraint construction module includes a three-dimensional line segment diagram acquisition module, which is used to acquire a three-dimensional line segment diagram based on the target plane.

[0144] In one embodiment, the three-dimensional line segment diagram acquisition module includes a three-dimensional line segment diagram acquisition submodule, which is used to acquire the normal vectors between intersecting target planes; the intersection lines between target planes whose angle between the normal vectors between intersecting target planes is greater than a third threshold are taken as planar intersection lines, and all the planar intersection lines constitute the three-dimensional line segment diagram.

[0145] In one embodiment, the second constraint construction module includes a first matching point pair construction module, which is used to construct a first matching point pair based on the original extrinsic parameters, the two-dimensional line segment diagram, and the three-dimensional line segment diagram.

[0146] In one embodiment, the first matching point pair construction module includes a first matching point pair construction submodule. The first matching point pair construction submodule is used to project the second point cloud points in the three-dimensional line segment diagram onto the two-dimensional line segment diagram based on the original extrinsic parameters to obtain the second pixel points of the second point cloud points on the two-dimensional line segment diagram; determine the first vertical distance between the second pixel points and all first line segments in the two-dimensional line segment diagram, determine the smallest first vertical distance as the first distance, and determine the second pixel points whose first distance is less than a first preset threshold as target second pixel points; determine the third pixel point corresponding to the foot of the perpendicular of the target second pixel point on the first line segment corresponding to the first distance as the target third pixel point; and match the second point cloud points corresponding to the target second pixel points with the target third pixel points to construct a first matching point pair.

[0147] In one embodiment, the second constraint construction module includes a second constraint construction submodule, which is used to determine the first reprojection error based on the coordinates of the target second pixel and the target third pixel in the first matching point pair; and to construct the second constraint based on the first reprojection error, wherein the expression for the coordinates of the target second pixel is: ;

[0148] The expression for the first projection error is:

[0149] =

[0150] The expression for the second constraint is:

[0151]

[0152] Where (R,t) are the original extrinsic parameters. Let i be the world coordinates of the i-th second point cloud point. The coordinates of the second pixel of the target The coordinates of the third pixel of the target are First projection error, N The number of first matching pairs Second constraint.

[0153] In one embodiment, the constraint construction module includes a third constraint construction module, which is used to perform side view processing on the target point cloud to obtain a three-dimensional side view map;

[0154] Based on the original extrinsic parameters, the two-dimensional line segment diagram, and the three-dimensional side view diagram, a second matching point pair is constructed; based on the second matching point pair, a third constraint condition is constructed.

[0155] In one embodiment, the third constraint construction module includes a three-dimensional side view map acquisition module, which is used to perform side view processing on the target point cloud to obtain a three-dimensional side view map.

[0156] In one embodiment, the three-dimensional side view map acquisition module includes a three-dimensional side view map acquisition submodule, which is used to filter the target point cloud to obtain candidate second target point clouds; perform cylinder radius consistency verification and cylinder direction completeness verification on the candidate second target point clouds, and determine the candidate second target point clouds that simultaneously satisfy the cylinder radius consistency verification and cylinder direction completeness verification as the second target point cloud; and acquire the three-dimensional side view map based on the second target point cloud and the initial pose of the autonomous vehicle.

[0157] In one embodiment, the third constraint construction module includes a first matching point pair construction submodule, and the second matching point pair construction module is used to construct a second matching point pair based on the original extrinsic parameters, the two-dimensional line segment diagram, and the three-dimensional side view diagram.

[0158] In one embodiment, the second matching point pair construction module includes a second matching point pair construction submodule. The second matching point pair construction submodule is used to: project a third point cloud from the three-dimensional side view map onto the two-dimensional line segment map based on the original extrinsic parameters to obtain a fourth pixel on the two-dimensional line segment map; determine a second perpendicular distance between the fourth pixel and all first line segments in the two-dimensional line segment map, determine the smallest second perpendicular distance as the second distance, and determine the fourth pixel whose second distance is less than a second preset threshold as a target fourth pixel; determine the third pixel corresponding to the foot of the perpendicular from the target fourth pixel on the first line segment corresponding to the second distance as a desired third pixel; and match the third point cloud corresponding to the target fourth pixel with the desired third pixel to construct a second matching point pair.

[0159] In one embodiment, the third constraint construction module includes a third constraint construction submodule, which is used to determine a second projection error based on the coordinates of the target fourth pixel point in the second matching point pair and the coordinates of the desired third pixel point; and to construct the third constraint based on the second projection error, wherein the expression for the coordinates of the target fourth pixel point is: ;

[0160] The expression for the second projection error is:

[0161] =

[0162] The expression for the third constraint is:

[0163]

[0164] Where (R,t) are the original extrinsic parameters. Let j be the world coordinates of the third point cloud. The coordinates of the fourth pixel of the target are The coordinates of the desired third pixel are: The second projection error is denoted by m, and m is the number of the second matching point pairs. The third constraint.

[0165] The vehicle extrinsic parameter calibration device provided in this disclosure can execute the vehicle extrinsic parameter calibration method provided in any embodiment of this disclosure, and has the corresponding functional modules and beneficial effects of executing the method.

[0166] This disclosure also provides an electronic device, which includes a processor and a memory; wherein the memory is used to store processor-executable instructions; the processor is used to read the executable instructions from the memory and execute the instructions to implement the above-described vehicle extrinsic parameter calibration method.

[0167] This disclosure provides a specific structure of an electronic device, which can be referred to as... Figure 5 The diagram shown is a structural schematic of an electronic device, such as Figure 5 As shown, the electronic device 400 includes one or more processors 401 and memory 402.

[0168] The processor 401 may be a central processing unit (CPU) or other form of processing unit with processing power and / or instruction execution capability, and may control other components in the electronic device 400 to perform desired functions.

[0169] The memory 402 may include one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may include, for example, random access memory (RAM) and / or cache memory. The non-volatile memory may include, for example, read-only memory (ROM), hard disk, flash memory, etc. One or more computer program instructions may be stored on the computer-readable storage medium, and the processor 801 may execute the program instructions to implement the vehicle extrinsic parameter calibration method of the embodiments of this disclosure described above and / or other desired functions. Various contents such as input signals, signal components, and noise components may also be stored in the computer-readable storage medium.

[0170] In one example, the electronic device 400 may also include an input device 403 and an output device 404, which are interconnected via a bus system and / or other forms of connection mechanism (not shown).

[0171] In addition, the input device 403 may also include, for example, a keyboard, a mouse, etc.

[0172] The output device 404 can output various information to the outside, including determined distance information, direction information, etc. The output device 404 may include, for example, a display, a speaker, a printer, and a communication network and its connected remote output devices, etc.

[0173] In addition to the methods and devices described above, embodiments of this disclosure may also be computer program products, including computer program instructions that, when executed by a processor, cause the processor to perform the vehicle extrinsic parameter calibration method provided in embodiments of this disclosure.

[0174] The computer program product can be written in any combination of one or more programming languages ​​to perform the operations of the embodiments of this disclosure. The programming languages ​​include object-oriented programming languages ​​such as Java and C++, as well as conventional procedural programming languages ​​such as C or similar languages. The program code can be executed entirely on a user's computing device, partially on a user's computing device, as a standalone software package, partially on a user's computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0175] Furthermore, embodiments of this disclosure may also be computer-readable storage media storing computer program instructions that, when executed by a processor, cause the processor to perform the vehicle extrinsic parameter calibration method provided in embodiments of this disclosure.

[0176] The computer-readable storage medium may be any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may be, for example, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: an electrical connection having one or more wires, a portable disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof.

[0177] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

Claims

1. A method for calibrating vehicle extrinsic parameters, characterized in that, include: Acquire a target point cloud based on vehicle-mounted LiDAR, wherein the target point cloud is formed by stitching together the poses of multiple frames of original point clouds after motion distortion compensation; Acquire target images based on vehicle-mounted cameras; Constructing target constraints based on the target point cloud, the target image, and the original extrinsic parameters includes: performing semantic segmentation on the target image using a semantic segmentation algorithm to obtain a first target mask; performing dilation processing on the first target mask and assigning values ​​to obtain a first target density distribution map; obtaining a target plane based on the target point cloud; obtaining a first target point cloud based on the target plane; and constructing a first constraint based on the first target density distribution map, the first target point cloud, and the original extrinsic parameters, wherein the target constraint includes the first constraint. The target constraints are optimized using a nonlinear optimization algorithm to obtain the vehicle's external parameters.

2. The method according to claim 1, characterized in that, The step of obtaining the target plane based on the target point cloud includes: The target point cloud is divided into an octree mesh to obtain octree nodes; The first point cloud point within the octree node is fitted using a random sampling consensus algorithm to obtain the fitting plane and the number of the first point cloud points in the fitting plane. Based on the octree nodes and the fitting plane, a fusion plane and a target fitting plane are obtained, wherein the target plane includes the fusion plane and the target fitting plane.

3. The method according to claim 2, characterized in that, The octree node includes a first octree node and a second octree node. Based on the octree node and the fitting plane, a fusion plane and a target fitting plane are obtained, including: Traverse the second octree nodes adjacent to the first octree node, and determine the first octree nodes whose angle between the normal vectors of the fitting planes corresponding to the first octree node and the second octree node is less than a first threshold, and whose distance from the point cloud center of the first octree node to the fitting plane corresponding to the second octree node is less than a second threshold, and whose distance from the point cloud center of the second octree node to the fitting plane corresponding to the first octree node is less than a second threshold, and take the first octree node as the target octree node. Merge the point clouds of the target octree node and the point clouds of the second octree node to obtain a fusion plane. The target fitting plane is determined based on the fitting plane corresponding to the target octree node.

4. The method according to claim 1, characterized in that, The step of obtaining the first target point cloud based on the target plane includes: The target plane is filtered to obtain candidate road surface point clouds; Based on the candidate road surface point cloud, obtain a bird's-eye view height map of the road surface point cloud; The edge of the road surface point cloud bird's-eye view height map is obtained by detecting the edge of the road surface point cloud bird's-eye view height map based on the edge detection algorithm; The first target point cloud is obtained by projecting the pixels at the edge of the bird's-eye view height map of the road surface point cloud onto the target point cloud.

5. The method according to claim 1, characterized in that, The construction of the first constraint condition based on the first target density distribution map, the first target point cloud, and the original extrinsic parameters includes: Based on the original extrinsic parameters, the first point cloud point in the first target point cloud is projected onto the first target density distribution map to obtain the first pixel point of the first point cloud point on the first target density distribution map; Based on the coordinates of the first pixel, the brightness of the first pixel in the first target density distribution map is obtained. The first constraint condition is constructed based on the brightness of the first pixel.

6. The method according to claim 5, characterized in that, The coordinate expression of the first pixel is: The expression for the brightness of the first pixel is: The expression for the first constraint is: Where (R,t) are the original extrinsic parameters, Xi is the coordinate of the i-th first point cloud point, xi is the coordinate of the first pixel point, pi is the brightness of the first pixel, and n is the number of first point cloud points in the first target point cloud. This is the first constraint condition.

7. The method according to any one of claims 1-6, characterized in that, The step of constructing target constraints based on the target point cloud, the target image, and the original extrinsic parameters further includes: The target image is preprocessed to obtain a two-dimensional line segment diagram of the target image; Based on the target plane, obtain a three-dimensional line segment diagram; Based on the original extrinsic parameters, the two-dimensional line segment diagram, and the three-dimensional line segment diagram, a first matching point pair is constructed; A second constraint is constructed based on the first matching point pair, wherein the target constraint further includes the second constraint.

8. The method according to claim 7, characterized in that, The step of obtaining a three-dimensional line segment diagram based on the target plane includes: Obtain the normal vectors between the intersecting target planes; The intersection lines between the target planes whose normal vectors are greater than a third threshold are taken as planar intersection lines, and all the planar intersection lines constitute the three-dimensional line segment diagram.

9. The method according to claim 7, characterized in that, The construction of the first matching point pair based on the original extrinsic parameters, the two-dimensional line segment graph, and the three-dimensional line segment graph includes: Based on the original extrinsic parameters, the second point cloud in the three-dimensional line segment diagram is projected onto the two-dimensional line segment diagram to obtain the second pixel point of the second point cloud in the two-dimensional line segment diagram. Determine the first vertical distance between the second pixel and all first line segments in the two-dimensional line segment diagram, determine the smallest first vertical distance as the first distance, and determine the second pixel whose first distance is less than a first preset threshold as the target second pixel. The third pixel point is determined as the target third pixel point based on the foot of the perpendicular from the first line segment corresponding to the first distance to the target second pixel point. Match the second point cloud corresponding to the second pixel of the target with the third pixel of the target to construct the first matching point pair.

10. The method according to claim 9, characterized in that, The construction of the second constraint based on the first matching point pair includes: The first reprojection error is determined based on the coordinates of the second and third pixels of the target in the first matching point pair; The second constraint is constructed based on the first projection error.

11. The method according to claim 10, characterized in that, The expression for the coordinates of the second pixel of the target is: ; The expression for the first projection error is: = The expression for the second constraint is: Where (R,t) are the original extrinsic parameters. Let i be the coordinates of the i-th second point cloud point. The coordinates of the second pixel of the target The coordinates of the third pixel of the target are First projection error, N The number of first matching pairs Second constraint.

12. The method according to claim 7, characterized in that, The step of constructing target constraints based on the target point cloud, the target image, and the original extrinsic parameters further includes: The target point cloud is processed by side-view processing to obtain a three-dimensional side-view map; Based on the original extrinsic parameters, the two-dimensional line segment diagram, and the three-dimensional side view diagram, a second matching point pair is constructed; A third constraint is constructed based on the second matching point pair, wherein the target constraint further includes the third constraint.

13. The method according to claim 12, characterized in that, The step of performing side-view processing on the target point cloud to obtain a three-dimensional side-view map includes: The target point cloud is filtered to obtain candidate second target point clouds; Perform cylinder radius consistency verification and cylinder direction completeness verification on the candidate second target point cloud, and determine the candidate second target point cloud that simultaneously satisfies the cylinder radius consistency verification and cylinder direction completeness verification as the second target point cloud; Based on the second target point cloud and the initial pose of the autonomous vehicle, the three-dimensional side view map is obtained.

14. The method according to claim 13, characterized in that, The construction of the second matching point pair based on the original extrinsic parameters, the two-dimensional line segment diagram, and the three-dimensional side view diagram includes: Based on the original extrinsic parameters, the third point cloud in the three-dimensional side view map is projected onto the two-dimensional line segment map to obtain the fourth pixel of the third point cloud on the two-dimensional line segment map. Determine the second vertical distance between the fourth pixel and all first line segments in the two-dimensional line segment diagram, determine the smallest second vertical distance as the second distance, and determine the fourth pixel whose second distance is less than the second preset threshold as the target fourth pixel. The third pixel point is determined as the expected third pixel point based on the foot of the perpendicular from the target fourth pixel point on the first line segment corresponding to the second distance; The third point cloud corresponding to the target fourth pixel is matched with the expected third pixel to construct a second matching point pair.

15. The method according to claim 14, characterized in that, The construction of the third constraint based on the second matching point pair includes: The second projection error is determined based on the coordinates of the target fourth pixel and the desired third pixel in the second matching point pair. The third constraint is constructed based on the second projection error.

16. The method according to claim 15, characterized in that, The expression for the coordinates of the fourth pixel of the target is: ; The expression for the second projection error is: = The expression for the third constraint is: Where (R,t) are the original extrinsic parameters. Let j be the coordinates of the third point cloud. The coordinates of the fourth pixel of the target are The coordinates of the desired third pixel are: The second projection error is denoted by m, and m is the number of the second matching point pairs. The third constraint.

17. The method according to claim 7, characterized in that, The target image is preprocessed to obtain a two-dimensional line segment diagram of the target image, including: The target image is input into an edge detection algorithm to obtain an edge map of the target image; The edge map is input into the Hough algorithm to obtain the two-dimensional line segment map of the target image.

18. The method according to claim 12, characterized in that, The optimization of the target constraints based on a nonlinear optimization algorithm to obtain vehicle extrinsic parameters includes: One or more of the first constraint, the second constraint, and the third constraint are input into a nonlinear optimization algorithm to obtain the vehicle's extrinsic parameters.

19. The method according to claim 1, characterized in that, The acquisition of the target point cloud based on the vehicle-mounted lidar includes: The system acquires multiple frames of raw point cloud data collected by the vehicle-mounted lidar, and obtains the pose and velocity state of each frame of the raw point cloud based on the lidar odometry method. Each frame of the original point cloud is subjected to motion distortion compensation based on its velocity state. The poses of multiple frames of the original point cloud after motion distortion compensation are stitched together to obtain the target point cloud.

20. The method according to claim 1, characterized in that, The acquisition of the target image based on the camera includes: The original image captured by the vehicle-mounted camera is acquired, and the original image is subjected to distortion correction processing to obtain the target image.

21. A vehicle external parameter calibration device, characterized in that, include: The first acquisition module is used to acquire a target point cloud based on a vehicle-mounted lidar, wherein the target point cloud is formed by stitching together the poses of multiple frames of original point clouds after motion distortion compensation. The second acquisition module is used to acquire target images based on the vehicle-mounted camera; A constraint construction module is used to construct target constraints based on the target point cloud, the target image, and the original extrinsic parameters, including: performing semantic segmentation on the target image based on a semantic segmentation algorithm to obtain a first target mask; performing dilation processing on the first target mask and assigning values ​​to obtain a first target density distribution map; obtaining a target plane based on the target point cloud; obtaining a first target point cloud based on the target plane; and constructing a first constraint based on the first target density distribution map, the first target point cloud, and the original extrinsic parameters, wherein the target constraint includes the first constraint. The extrinsic parameter acquisition module is used to optimize the target constraint conditions based on a nonlinear optimization algorithm to obtain the vehicle's extrinsic parameters.

22. An electronic device, characterized in that, The electronic device includes: One or more processors; Storage device for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the method as described in any one of claims 1-20.

23. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the method as described in any one of claims 1-20.

Images