External parameter calibration method and device, electronic equipment and storage medium

Abstract

The embodiment of the invention provides an external parameter calibration method and device, electronic equipment and a storage medium, and the method comprises the steps: obtaining image data and point cloud data which are synchronous in time, and enabling the point cloud data to be collected by a sensor; performing edge extraction on the image data to obtain an image edge, and performing edge extraction on the point cloud data to obtain a point cloud edge; based on the to-be-adjusted external parameter, the disturbance range and the step length, generating multiple groups of candidate external parameters of the sensor, and respectively determining the matching degree between the image edge and the point cloud edge corresponding to each group of candidate external parameters; and taking the candidate external parameter with the maximum matching degree as a new external parameter to be adjusted, optimizing and iteratively adjusting the disturbance range and the step length through a dichotomy, and returning to the step of generating the multiple groups of candidate external parameters of the sensor based on the external parameter to be adjusted, the disturbance range and the step length until the matching degree meets a convergence condition. And determining an external parameter calibration result of the sensor according to the current candidate external parameters. Therefore, the calibration process is simplified, and the calibration efficiency is improved.

Description

Technical Field

[0001] This application relates to the field of autonomous driving technology, and in particular to an external parameter calibration method, apparatus, electronic device, and storage medium. Background Technology

[0002] Autonomous driving systems need to fuse data from multiple sensors to perceive the surrounding environment. However, the installation positions and angles of each sensor are different. Therefore, it is necessary to unify the coordinate systems of these sensors through external parameter calibration to ensure that the multi-sensor data can be accurately fused and provide a reliable basis for decision-making.

[0003] In existing technologies, sensor extrinsic parameter calibration can be achieved based on environmental features (such as edges and line features). Specifically, firstly, edges are extracted from point cloud data and image data respectively. Then, feature matching is used to find point cloud edges and image edges that belong to the same physical edge. Based on these matched edges, a mathematical model of the sensor extrinsic parameters is established. Finally, the sensor extrinsic parameters are solved by minimizing the projection error to complete the calibration.

[0004] However, this sensor extrinsic parameter calibration method is likely to result in a cumbersome and inefficient calibration process due to factors such as the reliance on inefficient depth discontinuity judgment for point cloud edge extraction. Summary of the Invention

[0005] This application provides an external parameter calibration method, apparatus, electronic device, and storage medium, aiming to improve the problem of cumbersome and inefficient calibration process of sensor external parameters.

[0006] This application provides an embodiment of an external parameter calibration method, including:

[0007] Acquire time-synchronized image data and point cloud data, wherein the point cloud data is collected by a sensor;

[0008] Edge extraction is performed on the image data to obtain image edges, and edge extraction is performed on the point cloud data to obtain point cloud edges;

[0009] Based on the extrinsic parameters to be adjusted, the perturbation range, and the step size, multiple sets of candidate extrinsic parameters for the sensor are generated, and the matching degree between the image edge and the point cloud edge corresponding to each set of candidate extrinsic parameters is determined respectively.

[0010] The candidate extrinsic parameter with the highest matching degree is taken as the new extrinsic parameter to be adjusted, and the perturbation range and the step size are optimized and iterated by the binary search method. The process of generating multiple sets of candidate extrinsic parameters of the sensor based on the extrinsic parameter to be adjusted, the perturbation range, and the step size is returned until the matching degree meets the convergence condition. The extrinsic parameter calibration result of the sensor is determined according to the current candidate extrinsic parameters.

[0011] As can be seen from the above, the solution provided in this application acquires time-synchronized image data and point cloud data and extracts edges respectively. Then, based on the extrinsic parameters to be adjusted, the perturbation range, and the step size, multiple sets of candidate extrinsic parameters are generated, and the matching degree between the image edge and the point cloud edge corresponding to each set of candidate extrinsic parameters is determined. The extrinsic parameters to be adjusted are then updated with the candidate extrinsic parameters with the highest matching degree. The perturbation range and step size are then optimized iteratively by combining the binary search method until the matching degree converges. This effectively avoids the problem caused by the inefficient depth discontinuity judgment that the point cloud edge extraction relies on in the prior art, simplifies the calibration process, improves the calibration efficiency, and thus improves the cumbersome and inefficient state of the sensor extrinsic parameter calibration process.

[0012] Optionally, the convergence condition includes:

[0013] The difference between the matching degree of the current candidate extrinsic parameter and the matching degree of the candidate extrinsic parameter in the previous round is less than a preset difference; and / or,

[0014] The matching degree corresponding to the current candidate extrinsic parameter is greater than the preset convergence value.

[0015] Optionally, determining the matching degree between the image edge and the point cloud edge corresponding to each group of candidate extrinsic parameters includes:

[0016] Based on each set of candidate extrinsic parameters, a mapping relationship is determined between the image edge and the point cloud edge;

[0017] Based on each mapping relationship, the point cloud edges are mapped to the coordinate system of the image data to determine the corresponding mapped pixels of the point cloud edges in the image data;

[0018] The matching degree between the image edge and the mapped pixel is determined as the matching degree between the image edge and the point cloud edge.

[0019] Optionally, determining the matching degree between the image edge and the mapped pixel includes:

[0020] The pixel values ​​of the mapped pixels are normalized and averaged to obtain the matching degree between the image edge and the mapped pixels.

[0021] Optionally, the image data and the point cloud data include multiple sets, and the step of determining the extrinsic parameter calibration result of the sensor based on the current candidate extrinsic parameters includes:

[0022] The current candidate extrinsic parameters are determined as the reference extrinsic parameters of the sensor;

[0023] Determine the mean and variance of the reference extrinsic parameters corresponding to multiple sets of the image data and the point cloud data;

[0024] If the variance is greater than a preset error value, return to the step of acquiring time-synchronized image data and point cloud data;

[0025] If the variance is less than or equal to the preset error value, determine whether the difference between the average value and the current external parameter of the sensor is greater than the preset fluctuation value. If it is greater, then the average value is used as the external parameter calibration result of the sensor.

[0026] Optionally, the step of extracting edges from the image data to obtain image edges includes:

[0027] Edge extraction is performed on the image data to obtain reference edges;

[0028] For each target pixel in the reference edge, edge diffusion processing is performed on the target pixel based on the pixel values ​​of the neighboring pixels to obtain the fused pixel value of the target pixel, thus obtaining the image edge.

[0029] Optionally, the point cloud data includes an initial point cloud and at least one reference point cloud, and the step of edge extraction from the point cloud data to obtain point cloud edges includes:

[0030] Determine the relative pose of each reference point cloud relative to the initial point cloud;

[0031] Based on the relative pose, the reference point cloud and the initial point cloud are registered and stitched together to obtain a dense point cloud;

[0032] Edge extraction is performed on the dense point cloud to obtain the point cloud edges.

[0033] Optionally, the step of extracting the edges of the dense point cloud to obtain the point cloud edges includes:

[0034] Cluster analysis is performed on the dense point cloud to identify point cloud clusters with a preset shape;

[0035] The point cloud cluster is divided into multiple voxels;

[0036] Based on the adjacency of the voxels, voxels located at the edges are determined as point cloud edges.

[0037] This application also provides an external parameter calibration device, including:

[0038] The acquisition module is used to acquire time-synchronized image data and point cloud data, wherein the point cloud data is collected by a sensor;

[0039] An edge extraction module is used to extract edges from the image data to obtain image edges, and to extract edges from the point cloud data to obtain point cloud edges;

[0040] An initialization module is used to generate multiple sets of candidate extrinsic parameters for the sensor based on the extrinsic parameters to be adjusted, the perturbation range, and the step size, and to determine the matching degree between the image edge and the point cloud edge corresponding to each set of candidate extrinsic parameters.

[0041] The iterative adjustment module is used to take the candidate extrinsic parameter with the largest matching degree as the new extrinsic parameter to be adjusted, and to optimize and iteratively adjust the perturbation range and the step size through a binary search method. It returns to the step of generating multiple sets of candidate extrinsic parameters of the sensor based on the extrinsic parameter to be adjusted, the perturbation range, and the step size, until the matching degree meets the convergence condition. The extrinsic parameter calibration result of the sensor is determined based on the current candidate extrinsic parameters.

[0042] This application also provides an electronic device, including a processor and a memory, wherein:

[0043] Memory, used to store computer programs;

[0044] A processor for executing a program stored in memory to implement the method described in any of the preceding claims.

[0045] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements any of the methods described above. Attached Figure Description

[0046] Figure 1 This is a flowchart of an embodiment of the external parameter calibration method provided in this application;

[0047] Figure 2 This is a comparative diagram of single-frame point cloud data and dense point cloud provided in an embodiment of this application;

[0048] Figure 3 This application provides a point cloud data clustering result and a rod-shaped object point cloud extraction result according to an embodiment of the present application;

[0049] Figure 4 This is a flowchart of edge feature extraction from image data and point cloud data provided in a specific embodiment of this application;

[0050] Figure 5 A flowchart of the bisection method iterative optimization of sensor extrinsic parameters is provided in a specific embodiment of this application;

[0051] Figure 6 This is a structural diagram of the external parameter calibration device provided in the embodiments of this application;

[0052] Figure 7 This is a structural diagram of the electronic device provided in the embodiments of this application. Detailed Implementation

[0053] To make the technical problems, technical solutions, and beneficial effects solved by this application clearer, the following detailed description is provided in conjunction with embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0054] Autonomous driving systems need to fuse data from multiple sensors to perceive the surrounding environment. However, the installation positions and angles of each sensor are different. Therefore, it is necessary to unify the coordinate systems of these sensors through external parameter calibration to ensure that the multi-sensor data can be accurately fused and provide a reliable basis for decision-making.

[0055] However, current sensor extrinsic parameter calibration methods are often cumbersome and inefficient due to their reliance on inefficient depth discontinuity judgments for point cloud edge extraction. Therefore, this application provides an extrinsic parameter calibration method to address these issues.

[0056] This application provides an embodiment of an external parameter calibration method, including:

[0057] Acquire time-synchronized image data and point cloud data, with the point cloud data collected by sensors;

[0058] Edge extraction is performed on the image data to obtain the image edges, and edge extraction is performed on the point cloud data to obtain the point cloud edges;

[0059] Based on the extrinsic parameters to be adjusted, the perturbation range, and the step size, multiple sets of candidate extrinsic parameters for the sensor are generated, and the matching degree between the image edge and the point cloud edge corresponding to each set of candidate extrinsic parameters is determined respectively.

[0060] The candidate extrinsic parameter with the highest matching degree is taken as the new extrinsic parameter to be adjusted. The perturbation range and step size are optimized iteratively using the binary search method. The process of generating multiple sets of candidate extrinsic parameters for the sensor based on the extrinsic parameter to be adjusted, the perturbation range, and the step size is returned until the matching degree meets the convergence condition. The extrinsic parameter calibration result of the sensor is determined based on the current candidate extrinsic parameters.

[0061] As can be seen from the above, the solution provided in this application acquires time-synchronized image data and point cloud data and extracts edges respectively. Then, based on the extrinsic parameters to be adjusted, the perturbation range, and the step size, multiple sets of candidate extrinsic parameters are generated, and the matching degree between the image edge and the point cloud edge corresponding to each set of candidate extrinsic parameters is determined. The extrinsic parameters to be adjusted are then updated with the candidate extrinsic parameters with the highest matching degree. The perturbation range and step size are then optimized iteratively by combining the binary search method until the matching degree converges. This effectively avoids the problem caused by the inefficient depth discontinuity judgment that the point cloud edge extraction relies on in the prior art, simplifies the calibration process, improves the calibration efficiency, and thus improves the cumbersome and inefficient state of the sensor extrinsic parameter calibration process.

[0062] Example 1

[0063] This application provides an external parameter calibration method applied to a testing machine. Please refer to [the relevant documentation]. Figure 1 This includes the following steps:

[0064] S110: Acquires time-synchronized image data and point cloud data, with the point cloud data collected by the sensor.

[0065] First, time-synchronized image data and point cloud data are acquired. Image data is typically collected by a camera (such as an in-vehicle camera), while point cloud data is acquired using sensors such as LiDAR.

[0066] The time synchronization error between image data and point cloud data must be ≤10ms, which is achieved through timestamp calibration using IMU / RTK (Inertial Measurement Unit / Real-Time Kinematic). This ensures that image data and point cloud data are synchronized in time, so that they can truly reflect the environmental information at the same moment, guaranteeing that the edge of the point cloud and the edge of the image originate from the same physical object in the real world, and providing a reliable data source for accurate calibration of external parameters.

[0067] For example, continuously acquired video data can be segmented into 60-second segments (at least 3 segments). Within each segment, 6 frames of image data are extracted at 10-second intervals from 5 to 55 seconds after the start of the video data. Each frame of image data corresponds to point cloud data within 5 seconds. For example, if the sensor scanning frequency is 10Hz, then there are 50 frames of point cloud data in 5 seconds. Then, the pre-calibrated IMU / RTK and sensor extrinsic parameters can be called to determine the time synchronization relationship between the image data and the point cloud data.

[0068] S120: Extract edges from image data to obtain image edges, and extract edges from point cloud data to obtain point cloud edges.

[0069] After data acquisition, edge extraction operations can be performed on both image data and point cloud data. For image data, edge detection algorithms (such as Canny and Sobel) are used to identify sets of pixels in the image that exhibit abrupt changes in brightness or reflect the contours or structural boundaries of objects, thus forming image edges. For point cloud data, point cloud processing techniques (such as voxel segmentation and neighborhood curvature analysis) are used to filter out sets of points in the point cloud that exhibit sparse and abrupt changes in point distribution and reflect the geometric boundaries of physical objects, thus obtaining point cloud edges.

[0070] In this way, by extracting edge features across sensor modalities, the edge features of point cloud data and image data can be used as a bridge to associate image coordinates and sensor coordinates.

[0071] S130: Based on the extrinsic parameters to be adjusted, the perturbation range, and the step size, generate multiple sets of candidate extrinsic parameters for the sensor, and determine the matching degree between the image edge and the point cloud edge corresponding to each set of candidate extrinsic parameters.

[0072] In this step, based on the initial or iterative extrinsic parameters to be adjusted (such as rotation parameters), and combined with a preset perturbation range and step size, multiple sets of perturbations can be applied to the sensor extrinsic parameters to generate a series of possible combinations of extrinsic parameters, i.e., multiple sets of candidate extrinsic parameters. Then, for each set of candidate extrinsic parameters, a specific algorithm is used to calculate the matching degree between the image edge and the point cloud edge, i.e., the degree of overlap between the two types of edges in space. The higher the matching degree, the more likely the candidate extrinsic parameter is to be an accurate extrinsic parameter.

[0073] The matching degree between point cloud edges and image edges can be quantified using metrics such as JC (Joint Correspondence) value and intersection-over-union ratio (IoU), without any specific limitations. In this way, by quantifying the feature matching effect under different extrinsic parameters, comparable quantitative metrics are provided for extrinsic parameter optimization.

[0074] S140: Take the candidate extrinsic parameter with the highest matching degree as the new extrinsic parameter to be adjusted, and optimize the perturbation range and step size through the binary search method. Return to the steps of generating multiple sets of candidate extrinsic parameters of the sensor based on the extrinsic parameter to be adjusted, the perturbation range and step size, until the matching degree meets the convergence condition, and determine the extrinsic parameter calibration result of the sensor based on the current candidate extrinsic parameters.

[0075] After obtaining the matching degree of multiple sets of candidate extrinsic parameters, the set with the highest matching degree can be selected from the multiple sets of candidate extrinsic parameters as the new extrinsic parameter to be adjusted. The perturbation range and step size are narrowed by the bisection method (i.e., the search space is halved each time). Then, return to step S130, that is, generate multiple sets of candidate extrinsic parameters of the sensor again based on the new extrinsic parameter to be adjusted, the adjusted perturbation range and step size. This process is repeated until the change in matching degree meets the convergence condition. At this time, the corresponding candidate extrinsic parameter is the final sensor extrinsic parameter calibration result.

[0076] In this step, the introduction of the bisection method significantly improves the optimization efficiency. By efficiently narrowing the optimization range of external parameters, it reduces unnecessary computation while ensuring accuracy, effectively solving the problems of cumbersome process and low efficiency of traditional methods.

[0077] In one implementation, the convergence conditions include:

[0078] The difference between the matching degree of the current candidate extrinsic parameter and the matching degree of the previous round candidate extrinsic parameter is less than a preset difference; and / or,

[0079] The matching degree of the current candidate extrinsic parameter is greater than the preset convergence value.

[0080] In this implementation, the convergence criteria specifically include two judgment standards that can be used individually or in combination:

[0081] The first criterion focuses on the magnitude of the change in matching degree: if the difference between the matching degree of the current candidate extrinsic parameter and the matching degree of the candidate extrinsic parameter in the previous iteration is less than the preset difference, it indicates that the change in matching degree is small, and the effect of further adjusting the extrinsic parameter on improving the feature matching effect is no longer significant. The extrinsic parameter is close to the optimal value, and further iteration may lead to a waste of computing resources. Therefore, it can be considered that the extrinsic parameter optimization has entered convergence.

[0082] The second criterion focuses on the absolute value of the matching degree: if the matching degree corresponding to the current candidate extrinsic parameter is greater than the preset convergence value (this value is set according to the ideal matching effect of edge features in the scene, such as being close to the theoretical maximum value), it means that the matching degree has reached a sufficiently high level, which means that the overlap between the point cloud edge and the image edge has met the actual application requirements. Even if there is a small optimization space, there is no need to continue iterating. Therefore, convergence can be directly determined to improve calibration efficiency.

[0083] In practical applications, a single standard or a combination thereof can be selected according to the needs of the scenario: for example, in scenarios with extremely high accuracy requirements, both a difference less than a preset difference and a matching degree greater than a preset convergence value can be satisfied simultaneously; in scenarios that prioritize efficiency, the iteration can be terminated if either condition is met. This flexible convergence condition setting ensures both the accuracy of the calibration results and the efficiency of the algorithm, enabling the extrinsic parameter calibration to be stably effective under different application requirements.

[0084] In one implementation, step S130 involves determining the matching degree between the image edges and point cloud edges corresponding to each group of candidate extrinsic parameters, including:

[0085] Based on each set of candidate extrinsics, a mapping relationship between image edges and point cloud edges is determined;

[0086] Based on each set of mapping relationships, the point cloud edges are mapped to the coordinate system of the image data to determine the corresponding mapped pixels of the point cloud edges in the image data;

[0087] Determine the matching degree between image edges and mapped pixels, which is used as the matching degree between image edges and point cloud edges.

[0088] In this implementation, the spatial relationship between the point cloud edge and the image edge can be established through extrinsic parameters, thereby quantifying the degree of overlap between the two.

[0089] Specifically, for each set of candidate extrinsic parameters, the mapping relationship between the point cloud edge and the image edge is first constructed based on the rotation and translation parameters contained in the candidate extrinsic parameters. In other words, the point cloud edge in the radar coordinate system is associated with the image edge in the image coordinate system through geometric transformation.

[0090] Then, based on each set of determined mapping relationships, the point cloud edges can be mapped to the coordinate system of the image data, thereby accurately determining the mapped pixels of the point cloud edges in the image data. This is equivalent to converting the point cloud data to the same coordinate system as the image data, so that the two can be intuitively compared and analyzed.

[0091] Furthermore, the matching degree between image edges and point cloud edges can be obtained by measuring the matching degree between image edges and these mapped pixels. The matching degree can intuitively reflect the fit between point cloud data and image data in terms of edge features under the current candidate extrinsic parameters, providing an important basis for subsequent judgment on whether the extrinsic parameters are appropriate.

[0092] In one implementation, determining the matching degree between image edges and mapped pixels includes:

[0093] The pixel values ​​of the mapped pixels are normalized and averaged to obtain the matching degree between the image edges and the mapped pixels.

[0094] When calculating the matching degree between image edges and mapped pixels, this can be achieved by normalizing the pixel values ​​of the mapped pixels and averaging them.

[0095] Specifically, when the point cloud edges are projected onto the image coordinate system through a mapping relationship, a series of corresponding mapped pixels on the image are obtained. The pixel values ​​of these pixels directly reflect whether the location belongs to the image edge region. For example, the pixel values ​​of edge regions are usually higher, while those of non-edge regions are lower.

[0096] Then, the pixel value of each mapped pixel can be normalized to the range of 0 to 1, eliminating the influence of the absolute value of the pixel on the result. Furthermore, the average of all normalized pixel values ​​can be calculated. This average represents the matching degree between the image edge and the mapped pixel. The closer the average is to 1, the more mapped pixels fall in the image edge region, and the better the matching effect. The closer the average is to 0, the lower the overlap between the mapped pixel and the image edge, and the worse the matching effect.

[0097] This approach quantifies the overall overlap trend between mapped pixels and image edges, providing a concise and intuitive numerical reflection of matching quality. It offers a unified and easily calculable standard for comparing different candidate extrinsic parameters, while also balancing computational efficiency and matching accuracy.

[0098] For example, the known edge of the point cloud The camera's intrinsic parameters K and extrinsic parameters (rotation extrinsic parameter R, translation extrinsic parameter t, where t is fixed) are projected through the function (u, v) = F(K, R, t, E). lThis allows us to map each point cloud edge point to the image pixel coordinate system, obtaining the coordinates u of the mapped pixel. n v n .

[0099] For each mapped pixel u n v n Read the pixel value I(u) of the image at that location. n v n The pixel values ​​in the edge regions are close to 255, and those in the non-edge regions are close to 0. The pixel values ​​of all mapped pixels are normalized (divided by 255 to map the pixel values ​​to the 0-1 range), and then summed to obtain the JC value of a single frame of data, expressed as:

[0100]

[0101] Where m is the total number of edge points in the point cloud, and JC(R) is the matching degree between the image edge and the mapped pixel.

[0102] In one implementation, step S140 involves multiple sets of image data and point cloud data, and determining the sensor's extrinsic parameter calibration result based on the current candidate extrinsic parameters, including:

[0103] The current candidate extrinsic parameters are determined as the reference extrinsic parameters for the sensor;

[0104] Determine the mean and variance of the reference extrinsic parameters corresponding to multiple sets of image data and point cloud data;

[0105] If the variance is greater than the preset error value, return to the steps of acquiring time-synchronized image data and point cloud data;

[0106] If the variance is less than or equal to the preset error value, determine whether the difference between the average value and the current external parameter of the sensor is greater than the preset fluctuation value. If it is greater, the average value is used as the calibration result of the sensor's external parameter.

[0107] When image data and point cloud data contain multiple sets, the process of determining the sensor extrinsic parameter calibration results needs to undergo multiple rounds of verification and screening to ensure the reliability of the results.

[0108] Specifically, the candidate extrinsic parameters after iterative convergence can be temporarily set as the reference extrinsic parameters of the sensor, serving as the preliminary calibration results under a single set of data.

[0109] Subsequently, for multiple sets of image and point cloud data (e.g., data collected at different times and in different scenes), reference extrinsic parameters were calculated for each set of data, and then the mean and variance of these reference extrinsic parameters were statistically analyzed. The variance reflects the consistency of the calibration results across multiple sets of data. A smaller variance indicates more stable calibration results under different scenarios, while a larger variance indicates that the results are significantly affected by scene interference and may contain anomalies.

[0110] If the variance is greater than the preset error value, it means that the dispersion of multiple calibration results is too high and the reliability is insufficient. In this case, the entire process from data acquisition to iterative optimization needs to be re-executed to eliminate the influence of abnormal data or scene interference. If the variance is less than or equal to the preset error value, it indicates that multiple results are consistent, and the difference between the average value and the external parameters currently used by the sensor can be further determined.

[0111] If the difference is greater than the preset fluctuation value, it indicates that there is a significant deviation in the current external parameter. The calculated average value should be used to update the external parameter as the final calibration result. If the difference is less than or equal to the preset fluctuation value, it means that the current external parameter can still meet the accuracy requirements. No adjustment is needed, and the original external parameter can be retained.

[0112] In this way, through statistical analysis and anomaly verification of multiple sets of data, the random errors of single sets of data are avoided, and the dual judgment of variance and difference ensures that the calibration results achieve a balance between stability and accuracy. Finally, the output extrinsic parameters can adapt to the sensor fusion requirements of different scenarios.

[0113] In one implementation, step S120 involves edge extraction of the image data to obtain image edges, including:

[0114] Edge extraction is performed on the image data to obtain reference edges;

[0115] For each target pixel in the reference edge, edge diffusion processing is performed on the target pixel based on the pixel values ​​of the neighboring pixels to obtain the fused pixel value of the target pixel, thus obtaining the image edge.

[0116] In this implementation, more robust edge feature extraction can be achieved through edge diffusion processing. Specifically, before edge extraction from the image data, preprocessing can be performed: first, the image data is converted to grayscale; then, histogram equalization is used to enhance contrast; and finally, Gaussian filtering is applied for noise reduction, thereby further improving the accuracy of subsequent processing.

[0117] Specifically, the image data is first processed using an edge detection algorithm to filter out areas of brightness or color abrupt changes from the complex image texture. These areas constitute reference edges that reflect the contours of objects and can capture the most significant boundary features in the image, providing a basic contour for subsequent processing.

[0118] Based on this, for each target pixel in the reference edge, further edge diffusion processing is performed. The pixel values ​​of the target pixel are combined with the pixel values ​​of the neighboring pixels (i.e., the edge feature information of the surrounding area) and fused by a preset weight ratio to obtain the fused pixel value of the target pixel.

[0119] For example, if the target pixel itself is a strong edge (high pixel value), and the adjacent pixels also have edge features, the fused value will retain the core edge information and transition smoothly; if the adjacent pixels are non-edge areas, the fused value will be appropriately weakened to avoid isolated noise points being misjudged as edges.

[0120] After edge diffusion processing, the final image edge not only retains the core contour of the reference edge, but also enhances the coherence and anti-interference ability of the edge through information fusion of adjacent pixels. Weak edges that might have been broken are connected, and isolated pseudo edges are suppressed, making the image edge more closely match the physical boundary of the real object, and providing a more reliable feature basis for subsequent matching with point cloud edges.

[0121] In one implementation, step S120 involves extracting the point cloud data, which includes an initial point cloud and at least one reference point cloud. Edge extraction is performed on the point cloud data to obtain point cloud edges, including:

[0122] Determine the relative pose of each reference point cloud with respect to the initial point cloud;

[0123] Based on relative pose, the reference point cloud and the initial point cloud are registered and stitched together to obtain a dense point cloud.

[0124] Edge extraction is performed on dense point clouds to obtain point cloud edges.

[0125] In this implementation, when the point cloud data includes multiple data points,

[0126] When a point cloud contains an initial point cloud and at least one reference point cloud, multiple frames of point cloud data can be fused to generate a dense point cloud, thereby obtaining more complete and denser point cloud edge features. The initial point cloud typically refers to the first point cloud in the chronological order of multiple point cloud datasets. For example... Figure 2 As shown, this is a comparison diagram of single-frame point cloud data and dense point cloud. The left side is single-frame point cloud data, and the right side is dense point cloud. By fusing multiple frames of point cloud data, the dense point cloud obtained has a farther scanning distance and clearer edges compared to the single-frame sparse point cloud, which greatly improves the accuracy of point cloud edge feature extraction.

[0127] First, it is necessary to determine the relative pose of each reference point cloud relative to the initial point cloud. The relative pose describes the position and orientation relationship between the reference point cloud and the initial point cloud in three-dimensional space. It is usually calculated through the motion parameters of the sensor or the initial values ​​of the point cloud registration algorithm. This step provides a spatial correlation benchmark for point cloud data acquired at different times or from different perspectives, ensuring that subsequent stitching can accurately reflect the true geometric structure of the object.

[0128] Based on this, the reference point cloud and the initial point cloud are registered and stitched together using the calculated relative pose. The registration process fine-tunes the relative pose to precisely align overlapping areas between different point clouds. Then, stitching merges multiple point clouds into a single entity, forming a denser point cloud with wider coverage and higher point density. Compared to a single initial or reference point cloud, a denser point cloud can more completely represent the 3D contours of an object, reducing the loss of edge features caused by sparse point clouds.

[0129] Finally, edge extraction is performed on the stitched dense point cloud. By analyzing the neighborhood features of each point in the point cloud, a set of points reflecting the geometric boundaries of the object is selected; these point sets together constitute the edge of the point cloud. The use of dense point clouds allows edge extraction to capture more subtle structural features, providing a more accurate 3D boundary reference for subsequent matching with image edges.

[0130] In one implementation, edge extraction is performed on a dense point cloud to obtain the point cloud edges, including:

[0131] Cluster analysis is performed on dense point clouds to identify point cloud clusters with preset shapes;

[0132] The point cloud cluster is divided into multiple voxels;

[0133] Based on the adjacency of voxels, voxels located at the edges are identified as point cloud edges.

[0134] When extracting edges from dense point clouds, voxel segmentation can be used to accurately locate the point cloud edges that reflect the boundaries of objects.

[0135] Specifically, firstly, cluster analysis is performed on the dense point cloud, aggregating the discrete point cloud data based on the spatial distance between points to obtain different clustering results. Then, point cloud clusters that conform to a preset shape are selected based on the geometric features of the preset shape. In this way, by focusing on objects of specific shapes, interference from irrelevant point clouds (such as ground or cluttered vegetation) on edge extraction can be reduced. For example, the preset shape can be a rod-shaped object, such as... Figure 3 As shown, the results are clustering and point cloud cluster extraction of rod-shaped objects (preset shape) in the scene. The left side shows the clustering results, and the right side shows the point cloud clusters of rod-shaped objects filtered from the clustering results. Even when many dense trees are obscuring the view, point cloud clusters with rod-shaped shapes can still be identified through clustering and geometric features.

[0136] Next, the selected point cloud clusters are divided into multiple regular voxels (such as 0.1m×0.1m×0.1m cubic units). Voxelization transforms the 3D point cloud into a structured spatial mesh, which reduces the computational complexity of the original point cloud and reflects the overall shape of the object through the distribution characteristics of the voxel units.

[0137] Finally, edge voxels are determined based on the adjacency of voxels: each voxel is examined in terms of its adjacent voxels in the up, down, left, and right directions. If there are multiple consecutive empty voxels (i.e., areas without point cloud distribution) in a certain direction, it indicates that the voxel is located at the geometric boundary of the point cloud cluster and is marked as an edge voxel. The collection of these edge voxels together constitutes the point cloud edge, which not only preserves the contour features of the target object, but also makes the edge determination more robust through the structured analysis of voxel units, adapting to the subsequent requirements for accurate matching with image edges.

[0138] Example 2

[0139] To facilitate understanding, the following specific implementation method illustrates the process for edge feature extraction from image data and point cloud data provided in this application. Figure 4 The diagram shown is a flowchart of this embodiment, which specifically includes the following steps:

[0140] On the one hand, this includes edge feature extraction from image data:

[0141] The camera can capture images at different times, resulting in an image sequence that includes multiple frames of image data;

[0142] Perform image frame extraction to extract single frame image data Pi (i = 0, ..., 5) from the image sequence, and obtain the time ti (in ms) of the image data;

[0143] Each image data is preprocessed separately. For example, the image data can be converted into a grayscale image and then histogram equalization and Gaussian filtering can be performed.

[0144] Then, edge features can be extracted for each image data based on the Canny algorithm to obtain the reference edge corresponding to each image data.

[0145] Edge diffusion is performed on each target pixel in the extracted reference edge. Specifically, it can be regarded as having four 1x1 convolution kernels: one from top to bottom, one from bottom to top, one from left to right, and one from right to left. Each convolution kernel combines the pixel value of the current target pixel (most significant bit 255, least significant bit 0) with the edge-diffused pixel value of the adjacent pixel to generate a new pixel value of the current target pixel, which is the fused pixel value of the target pixel. In this way, after traversing each target pixel, the image edge of the image data is obtained, which provides an optimization direction for the subsequent calibration of sensor extrinsic parameters.

[0146] On the other hand, edge features can be extracted from point cloud data:

[0147] The lidar scans at its own frequency (e.g., 10Hz) to acquire point cloud data at different times;

[0148] Based on the time ti corresponding to the extracted image data, extract the point cloud data obtained by the LiDAR scanning between ti and ti+5000ms to ensure that the point cloud data is synchronized with the image data in time.

[0149] Motion distortion correction is performed on each frame of point cloud data using IMU / RTK;

[0150] Then, multiple frames of point cloud data can be stitched together at the same time t to generate a dense point cloud in the Lidar coordinate system at that time. Specifically, the relative pose of each frame of point cloud data (i.e., the reference point cloud) scanned after IMU / RTK is determined in the Lidar coordinate system relative to the initial point cloud (i.e., the point cloud data obtained from the first scan). The relative pose is then used as the initial value for NDT (Normal Distribution Transform) registration. Through the NDT registration algorithm, multiple frames of point cloud data are stitched together in the Lidar coordinate system to obtain the dense point cloud corresponding to the image data.

[0151] Point cloud edge feature extraction is based on voxels. Specifically, the dense point cloud is first segmented into ground and non-ground points. Then, cluster analysis is performed on the non-ground points. Based on the clustering results, clusters that do not meet the conditions are filtered out according to geometric features, and point cloud clusters with rod-shaped structures are retained. Then, the rod-shaped point cloud clusters are voxelized. Based on the adjacency between voxels, it is determined whether each voxel is an edge voxel. If it is, the outermost point within the voxel is extracted until the outermost point of all edge voxels is obtained, which is used as the point cloud edge corresponding to the point cloud data.

[0152] After processing the edges of the LiDAR point cloud and the camera image, they can be aligned to provide a matching basis for subsequent extrinsic parameter calibration.

[0153] Example 3

[0154] To facilitate understanding, the process of iteratively optimizing sensor extrinsic parameters using the bisection method provided in this application will be described below through a specific implementation method, such as... Figure 5 The diagram shown is a flowchart of this embodiment, which specifically includes the following steps:

[0155] First, obtain a set of image edges and point cloud edges that are time-aligned;

[0156] The perturbation range and step size in each round of optimization are determined by the bisection method. Perturbations are added to the extrinsic parameters of the sensor to be adjusted to obtain multiple sets of candidate extrinsic parameters. Among them, the extrinsic parameters of the sensor to be adjusted can be rotational extrinsic parameters.

[0157] The JC value (matching degree) is calculated. Specifically, the JC value corresponding to each group of candidate extrinsic parameters can be calculated. If the JC value increases after adding a perturbation and the change in JC is greater than the preset difference, the candidate extrinsic parameter with the largest JC value is taken as the new extrinsic parameter to be adjusted. Based on the binary search method, the perturbation range and step size are reduced by half, and the step of adding perturbation to the extrinsic parameter to be adjusted of the sensor is returned. Otherwise, it is determined that the JC value meets the convergence condition, and the optimization process can be exited. The current candidate extrinsic parameter is taken as the reference extrinsic parameter, and the subsequent extrinsic parameter anomaly detection and correction process continues to determine the extrinsic parameter calibration result.

[0158] The external parameter anomaly detection and correction includes the following steps:

[0159] For multiple sets of time-aligned image edges and point cloud edges, the rotation extrinsic parameters are optimized using the binary search method. The corresponding reference extrinsic parameters for each set are recorded. For example, if there are three sets of data, the reference extrinsic parameters can be represented as: {R1, R2, R3}.

[0160] Then, the average value of the external reference parameters is statistically analyzed. variance Where n represents the number of reference extrinsic parameters;

[0161] Assuming the preset error value τ is 0.005, then, if If calibration fails, re-optimize the rotational extrinsic parameters using the binary search method; otherwise, compare... If the error between the sensor's current extrinsic parameters and the error value is greater than the preset fluctuation value, the calibration is successful, and the sensor's extrinsic parameters are replaced abnormally, updating the rotational extrinsic parameters between the Lidar and the camera to this average value; otherwise, the sensor's current extrinsic parameters are retained.

[0162] Example 4

[0163] This application embodiment also provides an external parameter calibration device 40, such as Figure 6 As shown, it includes:

[0164] The acquisition module 401 is used to acquire time-synchronized image data and point cloud data, wherein the point cloud data is collected by a sensor;

[0165] The edge extraction module 402 is used to extract edges from the image data to obtain image edges, and to extract edges from the point cloud data to obtain point cloud edges;

[0166] The initialization module 403 is used to generate multiple sets of candidate extrinsic parameters of the sensor based on the extrinsic parameters to be adjusted, the perturbation range and the step size, and to determine the matching degree between the image edge and the point cloud edge corresponding to each set of candidate extrinsic parameters.

[0167] The iterative adjustment module 404 is used to take the candidate extrinsic parameter with the largest matching degree as the new extrinsic parameter to be adjusted, and to optimize and iteratively adjust the perturbation range and the step size through the binary search method. It returns to the step of generating multiple sets of candidate extrinsic parameters of the sensor based on the extrinsic parameter to be adjusted, the perturbation range and the step size, until the matching degree meets the convergence condition, and determines the extrinsic parameter calibration result of the sensor based on the current candidate extrinsic parameters.

[0168] This application also provides an electronic device 90, please refer to... Figure 7 It includes a processor 910 and a memory 920, wherein the memory 910 is used to store computer programs; the processor 920 is used to execute the programs stored in the memory 910 to implement the external parameter calibration method described in any embodiment of this application.

[0169] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the external parameter calibration method described in any embodiment of this application.

[0170] In this application, "multiple" refers to two or more.

[0171] In this application, unless otherwise expressly defined, the terms "installation," "connection," and "linking" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal connection between two components. Those skilled in the art can understand the specific meaning of the above terms in this application based on the specific circumstances.

[0172] The terms “first,” “second,” “third,” “fourth,” etc., in this application (if present) are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence.

[0173] In this application, the term "and / or" is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, "external parameter calibration and / or B" can represent: the existence of external parameter calibration alone, the existence of both external parameter calibration and B, or the existence of B alone. Additionally, in this application, the character " / " generally indicates that the preceding and following related objects have an "or" relationship.

[0174] Unless otherwise specified, all steps in this application may be performed sequentially or randomly. For example, the method includes steps external parameter calibration and B, indicating that the method may include steps external parameter calibration and B performed sequentially, or it may include steps B and external parameter calibration performed sequentially. For example, the method may also include step C, indicating that step C may be added to the method in any order. For example, the method may include steps external parameter calibration, B, and C, or it may include steps external parameter calibration, C, and B, or it may include steps C, external parameter calibration, and B, etc.

[0175] The above description is merely a preferred embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A method for calibrating external parameters, characterized in that, include: Acquire time-synchronized image data and point cloud data, wherein the point cloud data is collected by a sensor; Edge extraction is performed on the image data to obtain image edges, and edge extraction is performed on the point cloud data to obtain point cloud edges; Based on the extrinsic parameters to be adjusted, the perturbation range, and the step size, multiple sets of candidate extrinsic parameters for the sensor are generated, and the matching degree between the image edge and the point cloud edge corresponding to each set of candidate extrinsic parameters is determined respectively. The candidate extrinsic parameter with the highest matching degree is taken as the new extrinsic parameter to be adjusted, and the perturbation range and the step size are optimized and iterated by the binary search method. The process of generating multiple sets of candidate extrinsic parameters of the sensor based on the extrinsic parameter to be adjusted, the perturbation range, and the step size is returned until the matching degree meets the convergence condition. The extrinsic parameter calibration result of the sensor is determined according to the current candidate extrinsic parameters.

2. The method according to claim 1, characterized in that, The convergence conditions include: The difference between the matching degree of the current candidate extrinsic parameter and the matching degree of the candidate extrinsic parameter in the previous round is less than a preset difference; and / or, The matching degree corresponding to the current candidate extrinsic parameter is greater than the preset convergence value.

3. The method according to claim 1, characterized in that, The step of determining the matching degree between the image edge and the point cloud edge corresponding to each group of candidate extrinsic parameters includes: Based on each set of candidate extrinsic parameters, a mapping relationship is determined between the image edge and the point cloud edge; Based on each mapping relationship, the point cloud edges are mapped to the coordinate system of the image data to determine the corresponding mapped pixels of the point cloud edges in the image data; The matching degree between the image edge and the mapped pixel is determined as the matching degree between the image edge and the point cloud edge.

4. The method according to claim 3, characterized in that, Determining the matching degree between the image edge and the mapped pixel includes: normalizing the pixel value of the mapped pixel and calculating the average value to obtain the matching degree between the image edge and the mapped pixel.

5. The method according to claim 1, characterized in that, The image data and the point cloud data include multiple sets, and the step of determining the extrinsic parameter calibration result of the sensor based on the current candidate extrinsic parameters includes: The current candidate extrinsic parameters are determined as the reference extrinsic parameters of the sensor; Determine the mean and variance of the reference extrinsic parameters corresponding to multiple sets of the image data and the point cloud data; If the variance is greater than a preset error value, return to the step of acquiring time-synchronized image data and point cloud data; If the variance is less than or equal to the preset error value, determine whether the difference between the average value and the current external parameter of the sensor is greater than the preset fluctuation value. If it is greater, then the average value is used as the external parameter calibration result of the sensor.

6. The method according to claim 1, characterized in that, The step of extracting edges from the image data to obtain image edges includes: Edge extraction is performed on the image data to obtain reference edges; For each target pixel in the reference edge, edge diffusion processing is performed on the target pixel based on the pixel values ​​of the neighboring pixels to obtain the fused pixel value of the target pixel, thus obtaining the image edge.

7. The method according to claim 1, characterized in that, The point cloud data includes an initial point cloud and at least one reference point cloud. The step of edge extraction from the point cloud data to obtain point cloud edges includes: Determine the relative pose of each reference point cloud relative to the initial point cloud; Based on the relative pose, the reference point cloud and the initial point cloud are registered and stitched together to obtain a dense point cloud; Edge extraction is performed on the dense point cloud to obtain the point cloud edges.

8. The method according to claim 7, characterized in that, The step of extracting the edges of the dense point cloud to obtain the point cloud edges includes: Cluster analysis is performed on the dense point cloud to identify point cloud clusters with a preset shape; The point cloud cluster is divided into multiple voxels; Based on the adjacency of the voxels, voxels located at the edges are determined as point cloud edges.

9. An external parameter calibration device, characterized in that, include: The acquisition module is used to acquire time-synchronized image data and point cloud data, wherein the point cloud data is collected by a sensor; An edge extraction module is used to extract edges from the image data to obtain image edges, and to extract edges from the point cloud data to obtain point cloud edges; An initialization module is used to generate multiple sets of candidate extrinsic parameters for the sensor based on the extrinsic parameters to be adjusted, the perturbation range, and the step size, and to determine the matching degree between the image edge and the point cloud edge corresponding to each set of candidate extrinsic parameters. The iterative adjustment module is used to take the candidate extrinsic parameter with the largest matching degree as the new extrinsic parameter to be adjusted, and to optimize and iteratively adjust the perturbation range and the step size through a binary search method. It returns to the step of generating multiple sets of candidate extrinsic parameters of the sensor based on the extrinsic parameter to be adjusted, the perturbation range, and the step size, until the matching degree meets the convergence condition. The extrinsic parameter calibration result of the sensor is determined based on the current candidate extrinsic parameters.

10. An electronic device, characterized in that, Includes processor and memory, of which: Memory, used to store computer programs; A processor for executing a program stored in memory to implement the method described in any one of claims 1-8.

Images