Camera and lidar registration quality detection method, device, equipment and medium

Abstract

The application discloses a camera and laser radar registration quality detection method, device, equipment and medium, and relates to the technical field of intelligent driving. The method comprises the following steps: obtaining a dense depth map based on a color image obtained by a camera, and calculating a depth jump gradient; calculating a depth jump gradient based on a sparse depth map obtained by a laser radar; obtaining an image effective jump region and a laser radar effective jump region respectively; calculating the distance loss of each laser radar point based on each laser radar point in the laser radar effective jump region; obtaining a single-frame average loss based on each laser radar point with a distance loss less than a distance loss threshold; and selecting a sparse depth map with the largest single-frame average loss and marking it as a key review frame. The application extracts the geometric depth jump region of the image and the laser radar depth map, quantifies the matching consistency of the two, calculates the registration error, identifies the frame with the worst registration quality in the whole data segment, and realizes registration quality detection.

Description

Technical Field

[0001] This application relates to the field of intelligent driving technology, specifically to a method, apparatus, equipment, and medium for detecting the registration quality of a camera and a lidar. Background Technology

[0002] In the field of intelligent driving research and development, multimodal data collected by data acquisition vehicles is the core foundation for training intelligent driving models. High-quality labeled data directly determines the perception accuracy and robustness of intelligent driving models. Spatial registration between LiDAR and cameras is a key link in cross-modal data fusion, and its registration accuracy is a core indicator of data quality inspection, directly affecting the quality of image-point cloud cross-domain training data.

[0003] In actual data acquisition, the registration accuracy between lidar and camera is easily affected by multiple factors such as the inherent error of static calibration of the vehicle at the factory, the relative displacement of hardware during dynamic acquisition, the structural deformation of damping components after long-term use, and the amplification of errors due to vehicle bumps and vibrations. Therefore, the detection of registration quality is a necessary step in the quality inspection of intelligent driving data.

[0004] Current technologies employ manual quality inspection to check the registration quality between LiDAR and cameras. Quality inspectors visually check the registration accuracy of only the first frame of a single data clip acquired by the data acquisition vehicle. To control the quality of the entire clip, they must manually examine the registration of every frame. However, this method of checking only the first frame of a single data clip cannot cover the registration quality of all frames in the entire clip. It easily misses frames with registration errors caused by dynamic vehicle movement, leading to potential quality issues in the intelligent driving training data. This stems from the failure to consider the real-time changes in registration accuracy during dynamic driving scenarios. Furthermore, manually inspecting every frame of the entire clip consumes significant manpower and time, resulting in extremely low efficiency. This approach is unsuitable for the quality inspection needs of large-scale intelligent driving data in engineering applications, as manual operation is far less efficient than automated algorithms and cannot handle large-scale data processing quickly.

[0005] Therefore, in order to address the above problems and meet practical needs, a camera-LiDAR registration quality detection technology is proposed. Summary of the Invention

[0006] To address the shortcomings of existing technologies, this application aims to provide a method, apparatus, device, and medium for detecting the registration quality of a camera and a LiDAR system. Based on the geometric consistency of multimodal depth data, it achieves automated evaluation of the registration quality of the camera and LiDAR system. By extracting the geometric depth transition regions between the image and the LiDAR depth map, it quantifies the matching consistency between the two to calculate the registration error, identifies the frame with the worst registration quality in the entire data segment, and realizes automated, high-precision, and engineered registration quality detection throughout the entire process.

[0007] To achieve the above objectives, the technical solution adopted in this application is as follows: In a first aspect, this application provides a method for detecting the registration quality of a camera and a lidar system, the method comprising the following steps: Based on the color images acquired by the camera, and combined with the depth estimation model, a dense depth map aligned with pixels is obtained. The density depth map is calculated to obtain the depth transition gradient corresponding to the color image; Based on the sparse depth map obtained by lidar, the depth transition gradient corresponding to the sparse depth map is calculated. Based on the depth transition gradient corresponding to the color image and the depth transition gradient corresponding to the sparse depth map, the effective transition region of the image and the effective transition region of the lidar are obtained respectively. Based on each lidar point in the effective transition region of the lidar, the pixel point with the closest Euclidean distance is searched in the effective transition region of the image, and the distance loss of each lidar point is calculated. The average loss per frame is obtained based on each of the lidar points whose distance loss is less than a preset distance loss threshold. Based on the average loss of a single frame corresponding to multiple sparse depth maps obtained within the data acquisition period, the sparse depth map with the largest average loss of a single frame is selected and marked as a key review frame.

[0008] Based on the above technical solution, the step of calculating the depth transition gradient corresponding to the color image by the dense depth map includes the following steps: The horizontal gradient and vertical gradient are calculated on the dense depth map respectively, and then the depth transition gradient corresponding to the color image is obtained.

[0009] Based on the above technical solution, the step of calculating the depth transition gradient corresponding to the sparse depth map obtained by lidar includes the following steps: Based on the sparse depth map obtained by LiDAR, parallel computation using CUDA operators is performed. For each pixel in the sparse depth map, the nearest valid LiDAR depth value is searched in the four-directional neighborhood, and the depth transition gradient corresponding to the sparse depth map is calculated.

[0010] Based on the above technical solution, the step of selecting the sparse depth map with the largest single-frame average loss based on the average loss of multiple sparse depth maps obtained within the data acquisition period and marking it as a key review frame includes the following steps: Construct a sliding window and slide it frame by frame according to a preset step size for the average loss of a single frame corresponding to multiple sparse depth maps obtained within the data acquisition period. The sparse depth map with the largest average loss in a single frame is identified and marked as a key frame for review.

[0011] Secondly, this application provides a camera and lidar registration quality detection device, the device comprising: The dense depth map acquisition module is used to obtain a pixel-aligned dense depth map based on the color image acquired by the camera and combined with the depth estimation model. A depth jump gradient acquisition module is used to calculate the dense depth map and obtain the depth jump gradient corresponding to the color image. The depth jump gradient acquisition module is also used to calculate the depth jump gradient corresponding to the sparse depth map based on the sparse depth map obtained by the lidar. The effective transition region acquisition module is used to obtain the effective transition region of the image and the effective transition region of the lidar based on the depth transition gradient corresponding to the color image and the depth transition gradient corresponding to the sparse depth map, respectively. The distance loss acquisition module is used to search for the pixel with the closest Euclidean distance in the effective jump area of ​​the image based on each lidar point in the effective jump area of ​​the lidar, and calculate the distance loss of each lidar point. A single-frame average loss module is used to obtain the single-frame average loss based on each of the lidar points whose distance loss is less than a preset distance loss threshold. The key review frame marking module is used to select the sparse depth map with the largest single-frame average loss based on the single-frame average loss of multiple sparse depth maps obtained within the data acquisition period, and mark it as a key review frame.

[0012] Based on the above technical solution, the depth jump gradient acquisition module is also used to perform horizontal gradient calculation and vertical gradient calculation on the dense depth map respectively, and then calculate the depth jump gradient corresponding to the color image.

[0013] Based on the above technical solution, the depth jump gradient acquisition module is also used to perform parallel computation of CUDA operators based on the sparse depth map obtained by LiDAR, search for the nearest valid LiDAR depth value in the four-directional neighborhood of each pixel in the sparse depth map, and calculate the depth jump gradient corresponding to the sparse depth map.

[0014] Based on the above technical solution, the key review frame marking module is also used to construct a sliding window, and slide the average loss of a single frame corresponding to multiple sparse depth maps obtained within the data acquisition period frame by frame according to a preset step size. The key review frame marking module is also used to identify the sparse depth map with the largest average loss in a single frame and mark it as a key review frame.

[0015] Thirdly, this application provides an apparatus including a processor and a memory, the memory being coupled to the processor, the memory storing instructions that, when executed by the processor, cause the apparatus to perform the steps of the method mentioned in the first aspect.

[0016] Fourthly, this application provides a storage medium having a computer program stored thereon, characterized in that the program, when executed by a processor, implements the steps of the method mentioned in the first aspect.

[0017] Compared with the prior art, the advantages of this application are: This application achieves automated evaluation of camera and LiDAR registration quality based on the geometric consistency of multimodal depth data. By extracting the geometric depth transition regions of the image and the LiDAR depth map, the matching consistency between the two is quantified to calculate the registration error, and the frame with the worst registration quality in the entire data segment is identified. The entire process realizes automated, high-precision, and engineered registration quality detection. Attached Figure Description

[0018] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 This is a flowchart illustrating the steps of the camera and lidar registration quality detection method according to an embodiment of this application. Figure 2 This is a structural block diagram of the camera and lidar registration quality detection device according to an embodiment of this application. Detailed Implementation

[0020] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0021] The embodiments of this application will be further described in detail below with reference to the accompanying drawings.

[0022] This application provides a method, apparatus, device, and medium for detecting the registration quality of a camera and a lidar system. Based on the geometric consistency of multimodal depth data, it achieves automated evaluation of the registration quality of the camera and lidar system. By extracting the geometric depth transition regions of the image and the lidar depth map, it quantifies the matching consistency between the two to calculate the registration error and identifies the frame with the worst registration quality in the entire data segment. The entire process achieves automated, high-precision, and engineered registration quality detection.

[0023] To achieve the aforementioned technical effects, the overall concept of this application is as follows: A method for detecting the registration quality of a camera and a lidar system, comprising the following steps: S1. Based on the color image acquired by the camera, and combined with the depth estimation model, obtain a dense depth map aligned with pixels. S2. Calculate the depth gradient corresponding to the color image by calculating the dense depth map; S3. Based on the sparse depth map obtained by LiDAR, calculate the depth transition gradient corresponding to the sparse depth map. S4. Based on the depth transition gradient corresponding to the color image and the depth transition gradient corresponding to the sparse depth map, the effective transition region of the image and the effective transition region of the lidar are obtained respectively. S5. Based on each lidar point in the effective transition region of the lidar, search for the pixel point with the closest Euclidean distance in the effective transition region of the image, and calculate the distance loss of each lidar point. S6. Based on each lidar point where the distance loss is less than the preset distance loss threshold, obtain the average loss per frame. S7. Based on the average loss of a single frame corresponding to multiple sparse depth maps obtained within the data acquisition period, select the sparse depth map with the largest average loss per frame and mark it as a key review frame.

[0024] The embodiments of this application will be further described in detail below with reference to the accompanying drawings.

[0025] Firstly, see [the following] Figure 1 As shown in the figure, this application provides a method for detecting the registration quality of a camera and a lidar system. The method includes the following steps: S1. Based on the color image acquired by the camera, and combined with the depth estimation model, obtain a dense depth map aligned with pixels. S2. Calculate the depth gradient corresponding to the color image by calculating the dense depth map; S3. Based on the sparse depth map obtained by LiDAR, calculate the depth transition gradient corresponding to the sparse depth map. S4. Based on the depth transition gradient corresponding to the color image and the depth transition gradient corresponding to the sparse depth map, the effective transition region of the image and the effective transition region of the lidar are obtained respectively. S5. Based on each lidar point in the effective transition region of the lidar, search for the pixel point with the closest Euclidean distance in the effective transition region of the image, and calculate the distance loss of each lidar point. S6. Based on each lidar point where the distance loss is less than the preset distance loss threshold, obtain the average loss per frame. S7. Based on the average loss of a single frame corresponding to multiple sparse depth maps obtained within the data acquisition period, select the sparse depth map with the largest average loss per frame and mark it as a key review frame.

[0026] It should be noted that the technical details of the existing technology are as follows: The core logic of the IMU+GPS-assisted LiDAR-Image registration and detection scheme is as follows: Using coarse positioning information provided by the IMU (Inertial Measurement Unit) and GPS, the LiDAR pose is calculated using the LiDAR-ICP algorithm, and the image pose is calculated through image feature point matching. Finally, the consistency between the two poses is compared to achieve registration quality detection. Essentially, it is an extension of calibration-free automated detection, relying on IMU+GPS assisted positioning and eliminating the need for a physical calibration board.

[0027] Assisted positioning: IMU collects angular velocity and acceleration, GPS collects absolute position, and the data are fused to obtain coarse positioning pose (as a reference, not the true value); LiDAR pose (lidar to global) solution: Based on LiDAR point clouds, high-precision LiDAR poses are obtained through iterative optimization using the ICP algorithm (or its variants); Image pose (camera to global) solution: The pose of the image is solved by matching image feature points (such as SIFT, ORB) and combining them with the PnP algorithm; Registration detection: Compare the deviation between lidar to global and camera to global (the difference between the rotation matrix R and the translation matrix T). The smaller the deviation, the higher the registration quality.

[0028] The technical shortcomings of existing technologies are: The positioning accuracy relying on IMU+GPS is susceptible to environmental interference: GPS signals are weak in scenarios such as tunnels, tall buildings, and underground parking garages, resulting in larger positioning errors; long-term use of IMU will generate cumulative errors, and the coarse positioning deviation after the fusion of the two will indirectly affect the pose calculation accuracy of LiDAR and images, thereby affecting the registration and detection results.

[0029] The limitations of the ICP algorithm are as follows: The ICP algorithm is sensitive to the initial pose. If the coarse positioning error provided by IMU+GPS is too large, it will cause ICP to fail to converge or converge to a local optimum, resulting in incorrect LiDAR pose calculation and affecting the accuracy of subsequent pose comparisons.

[0030] Image feature point matching is limited by the scene: when the scene has sparse texture (such as an empty road surface), drastic lighting changes (such as strong light, backlight) or motion blur, it is difficult to extract image feature points and the mismatch rate is high, which will lead to deviation in image pose solution and affect the registration and detection accuracy.

[0031] The computational cost is high: ICP algorithms (especially when the point cloud is large) and image feature point matching (such as SIFT) require a lot of computational resources; if real-time detection is required, additional optimizations such as CUDA acceleration are needed, which increases the engineering implementation cost.

[0032] Unable to detect static calibration error: This solution is mainly for registration quality detection during dynamic acquisition and cannot replace the traditional method based on calibration board. It is difficult to detect static calibration error (such as factory calibration deviation) after sensor installation.

[0033] The threshold for good or bad registration is difficult to set: there is no unified standard for the deviation threshold of pose comparison (i.e. the critical value for judging whether the registration is qualified or unqualified). It needs to be repeatedly adjusted according to the specific scenario (such as urban roads, highways) and sensor model. If the threshold is set too strictly, a large number of normal frames will be misjudged as poor frames. If it is set too loosely, frames with large registration errors will be missed, increasing the extra workload of quality inspectors.

[0034] Therefore, in view of the deficiencies of the prior art, the technical solution of the embodiments of this application aims to solve the following technical problems: To address the limitations and inefficiencies of manual quality inspection methods, which are unsuitable for large-scale engineering-scale intelligent driving data quality inspection, this method enables automated detection of camera and LiDAR registration quality. This addresses the issues of simple pixel matching methods being susceptible to noise interference and inaccurate detection results, enabling precise quantification of registration errors. This addresses the problem that existing detection technologies, which do not incorporate temporal characteristics, cannot identify the worst-registration frames in dynamic driving scenarios. It enables accurate quality inspection of all data segments, providing a reliable basis for quality control of intelligent driving training data.

[0035] In this embodiment, the camera and LiDAR registration quality is automatically evaluated based on the geometric consistency of multimodal depth data. By extracting the geometric depth transition regions of the image and the LiDAR depth map, the matching consistency between the two is quantified to calculate the registration error, and the frame with the worst registration quality in the entire data segment is identified. The entire process realizes automated, high-precision, and engineered registration quality detection.

[0036] Furthermore, the step of calculating the depth transition gradient corresponding to the color image from the dense depth map includes the following steps: The horizontal gradient and vertical gradient are calculated on the dense depth map respectively, and then the depth transition gradient corresponding to the color image is obtained.

[0037] Furthermore, the calculation of the depth transition gradient corresponding to the sparse depth map obtained based on the lidar includes the following steps: Based on the sparse depth map obtained by LiDAR, parallel computation using CUDA operators is performed. For each pixel in the sparse depth map, the nearest valid LiDAR depth value is searched in the four-directional neighborhood, and the depth transition gradient corresponding to the sparse depth map is calculated.

[0038] Furthermore, the step of selecting the sparse depth map with the largest single-frame average loss based on the average loss of multiple sparse depth maps obtained within the data acquisition period and marking it as a key review frame includes the following steps: Construct a sliding window and slide it frame by frame according to a preset step size for the average loss of a single frame corresponding to multiple sparse depth maps obtained within the data acquisition period. The sparse depth map with the largest average loss in a single frame is identified and marked as a key frame for review.

[0039] In summary, the automated detection method for camera and LiDAR registration quality in this application embodiment can be integrated into the on-board data quality inspection terminal of an intelligent driving data acquisition vehicle, enabling real-time detection of registration quality during data acquisition. The manufacturing of the relevant on-board terminal can incorporate the following improvements and innovations: At the hardware level: a high-performance GPU module is configured in the vehicle terminal to ensure the efficient operation of CUDA parallel computing. At the same time, the hardware interface of the terminal is optimized to realize the synchronous acquisition and transmission of camera color images and LiDAR point cloud data, reducing data transmission latency. At the software level: the detection algorithm of this invention is encapsulated as a lightweight software plugin and integrated into the vehicle data processing system to achieve seamless connection between the algorithm and the data acquisition system. At the same time, a visual interface is developed to display the registration loss value and the location of poor quality frames in real time, which is convenient for on-site staff to view. At the process level: In the assembly process of the vehicle terminal, the connection between the GPU module and the motherboard is optimized to improve the vibration resistance of the hardware, adapt to the complex environment of vehicle driving, and avoid the impact of hardware vibration on the algorithm's running efficiency and detection accuracy.

[0040] In addition, this method can also be deployed to a cloud-based intelligent driving data quality inspection platform to perform batch testing on the massive amounts of data collected. The server cluster of the cloud platform can adopt a distributed computing approach to further improve testing efficiency and adapt to the quality inspection needs of large-scale intelligent driving data.

[0041] The key technology of this application's embodiments lies in: A registration error quantification method based on geometric depth transition regions: Focusing on core feature regions of depth transition such as object edges and scene boundaries, the method calculates the Euclidean distance loss between the LiDAR and the effective transition points in the image to achieve accurate quantification of registration error, avoiding interference from noise and meaningless pixels; High-efficiency processing technology for multimodal depth data: The Sobel operator is used to extract depth transition regions in the image, and CUDA parallel computing is used to realize the fast processing of sparse depth maps of LiDAR, which ensures the efficiency of the detection algorithm and is suitable for engineering applications. Sliding window temporal voting mechanism: Combining the scenario characteristics of dynamic driving in intelligent driving, a sliding window is constructed to perform temporal voting on the single-frame registration loss of the entire data segment, accurately identify key frames with consistently poor registration quality, and achieve efficient quality inspection of the entire segment.

[0042] In specific implementation, the technical solution based on the embodiments of this application is as follows: Step 1: Image depth map extraction: Color images acquired by the input data acquisition vehicle ( Image height, (where 3 represents the image width and 3 represents the number of RGB channels). Based on the Depth Anything V3 depth estimation model, it outputs a dense depth map aligned with the image pixels. ,in Represents image coordinates The depth value at that location.

[0043] Step 2: Detection of abrupt changes in image depth map (Sobel operator): For dense depth maps Applying the horizontal Sobel operator respectively and vertical Sobel operator The depth gradient is calculated to extract depth transition regions. The specific calculation is as follows: Horizontal gradient calculation: ; Vertical gradient calculation: ; Depth gradient magnitude (depth jump index): ; Among them, the horizontal Sobel operator Vertical Sobel operator , Characterizes the intensity of depth jumps in an image.

[0044] Step 3: Detection of abrupt changes in the sparse depth map using LiDAR (CUDA acceleration): Sparse depth map after input lidar projection (Pixels without LiDAR have a value of 0), based on parallel computation using CUDA operators, for each pixel Search for the nearest effective lidar depth value in the four-directional neighborhood and calculate the lidar depth jump gradient. Specific steps: Four-directional neighborhood search: for pixels Define the neighborhood search range in four directions: up, down, left, and right, and obtain the effective LiDAR depth set within the neighborhood. ; LiDAR gradient jump calculation: ; Isolated point removal rule: If there is no valid LiDAR signal in the four-directional neighborhood, the pixel is determined to be an isolated point, let And remove them.

[0045] Step 4, Gradient threshold filtering: Set a threshold to filter valid depth transition regions and filter out low-noise pixels. Specific judgment rules are as follows: Effective transition regions in the image: ; Effective transition area of ​​lidar: ; in, This indicates that the pixel is a valid depth transition point in the image. This indicates that the pixel is the effective depth transition point for the LiDAR.

[0046] Step 5: Calculate the distance loss for a single lidar point: For all valid lidar transition points (satisfy In the effective image depth transition region (satisfying) Search for the pixel with the closest Euclidean distance in the search. Calculate the distance loss at this lidar point. : .

[0047] Step 6: Calculate the single-frame registration loss: Outlier Removal: Distance Loss The lidar points are identified as outliers and are not included in the loss calculation; Average loss per frame (registration quality quantification metric): ; in, This represents the total number of valid lidar transition points after removing outliers. For the first Distance loss of each valid point; The larger the value, the greater the registration error and the worse the quality of the frame.

[0048] Step 7: Sliding window timing vote: Based on the data acquisition sequence, a sliding window with a size of 20 frames and a step size of 1 frame (sliding frame by frame) is constructed to achieve voting filtering of frames with poor registration quality: Define a sliding window: ( Index of the window's starting frame. , (Total number of frames). In-window voting rules: For each window Select single-frame loss Largest frame The number of votes for this frame ; Physical meaning of vote count: The larger the value, the worse the registration quality of that frame is across multiple sliding windows.

[0049] Step 8: Extracting frames with poor quality: Based on the number of votes for all frames Sort in descending order and extract the top votes. frame( The number of frames to be reviewed by quality inspectors is set as the key review frames to complete automated testing.

[0050] When necessary, the above-mentioned technical means can be replaced by the following solutions: Depth estimation model replacement: The DepthAnything V3 depth estimation model used in step 1 can be replaced by other monocular image depth estimation models such as DPT and MiDaS, all of which can output dense depth maps aligned with image pixels, thus enabling the extraction of image depth information; Edge detection operator replacement: The Sobel operator used in step 2 to extract abrupt changes in image depth can be replaced with other edge detection operators such as the Canny operator and the Laplacian operator, all of which can effectively extract abrupt changes in depth map. Sliding window parameter replacement: The size of the sliding window in step 7 can be adjusted according to the frame rate of data acquisition and scene characteristics (such as 10 frames, 30 frames), and the step size can also be adjusted according to the detection accuracy requirements (such as 2 frames, 5 frames), which can still achieve the purpose of time-series voting to filter frames with poor registration quality. Acceleration method replacement: The CUDA acceleration used for LiDAR depth map processing in step 3 can be replaced by other parallel computing acceleration methods such as OpenCL, both of which can improve the detection efficiency of LiDAR depth jump regions.

[0051] It should be noted that the technical solution of the embodiments of this application can achieve the following beneficial effects: This application embodiment focuses on the core feature region of registration quality detection by extracting depth jump regions and calculating geometric distance loss, avoiding interference from meaningless pixels and noise, and achieving accurate quantification of registration error. Compared with simple pixel matching methods, the accuracy of detection results is greatly improved. This application introduces a sliding window timing voting mechanism, which, combined with the dynamic driving scenario characteristics of intelligent driving, can accurately identify key frames with consistently poor registration quality in the entire data segment. This solves the problem that the existing technology cannot cover the quality inspection of the entire segment, and provides a reliable basis for the quality control of intelligent driving training data. The embodiments of this application realize the full automation of registration quality inspection. Based on the CUDA operator, the processing of LiDAR depth map is accelerated in parallel, which ensures the detection efficiency. Compared with the manual quality inspection method, it greatly reduces the manpower and time costs and can be directly applied to the quality inspection of large-scale intelligent driving data. The registration loss quantification index and voting screening rules in this application embodiment can be flexibly adjusted (such as gradient threshold, sliding window size, number of frames to be reviewed). It can adapt to different intelligent driving data collection scenarios and has good versatility and scalability.

[0052] Secondly, see Figure 2 As shown, this application embodiment provides a camera and lidar registration quality detection device, the device comprising: The dense depth map acquisition module is used to obtain a pixel-aligned dense depth map based on the color image acquired by the camera and combined with the depth estimation model. A depth jump gradient acquisition module is used to calculate the dense depth map and obtain the depth jump gradient corresponding to the color image. The depth jump gradient acquisition module is also used to calculate the depth jump gradient corresponding to the sparse depth map based on the sparse depth map obtained by the lidar. The effective transition region acquisition module is used to obtain the effective transition region of the image and the effective transition region of the lidar based on the depth transition gradient corresponding to the color image and the depth transition gradient corresponding to the sparse depth map, respectively. The distance loss acquisition module is used to search for the pixel with the closest Euclidean distance in the effective jump area of ​​the image based on each lidar point in the effective jump area of ​​the lidar, and calculate the distance loss of each lidar point. A single-frame average loss module is used to obtain the single-frame average loss based on each of the lidar points whose distance loss is less than a preset distance loss threshold. The key review frame marking module is used to select the sparse depth map with the largest single-frame average loss based on the single-frame average loss of multiple sparse depth maps obtained within the data acquisition period, and mark it as a key review frame.

[0053] In this embodiment, the camera and LiDAR registration quality is automatically evaluated based on the geometric consistency of multimodal depth data. By extracting the geometric depth transition regions of the image and the LiDAR depth map, the matching consistency between the two is quantified to calculate the registration error, and the frame with the worst registration quality in the entire data segment is identified. The entire process realizes automated, high-precision, and engineered registration quality detection.

[0054] Furthermore, the depth jump gradient acquisition module is also used to perform horizontal gradient calculation and vertical gradient calculation on the dense depth map respectively, and then calculate the depth jump gradient corresponding to the color image.

[0055] Furthermore, the depth jump gradient acquisition module is also used to perform parallel computation of CUDA operators based on the sparse depth map obtained by the LiDAR, search for the nearest valid LiDAR depth value in the four-directional neighborhood of each pixel in the sparse depth map, and calculate the depth jump gradient corresponding to the sparse depth map.

[0056] Furthermore, the key review frame marking module is also used to construct a sliding window, and slide the average loss of a single frame corresponding to multiple sparse depth maps obtained within the data acquisition period frame by frame according to a preset step size. The key review frame marking module is also used to identify the sparse depth map with the largest average loss in a single frame and mark it as a key review frame.

[0057] In summary, the camera and lidar registration quality detection device provided in this application embodiment has the same technical principle as the camera and lidar registration quality detection method provided in the first aspect in terms of technical problems, technical solutions and technical effects, so it will not be described in detail here.

[0058] Thirdly, this application provides an apparatus including a processor and a memory, the memory being coupled to the processor, the memory storing instructions that, when executed by the processor, cause the apparatus to perform the steps of the method mentioned in the first aspect.

[0059] Fourthly, this application provides a storage medium having a computer program stored thereon, characterized in that the program, when executed by a processor, implements the steps of the method mentioned in the first aspect.

[0060] In the description of this application, it should be noted that the terms "upper," "lower," etc., indicating the orientation or positional relationship are based on the orientation or positional relationship shown in the accompanying drawings, and are only for the convenience of describing this application and simplifying the description, and do not indicate or imply that the system or component referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of this application. Unless otherwise expressly specified and limited, the terms "installed," "connected," and "linked" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; they can refer to the internal communication between two components. For those skilled in the art, the specific meaning of the above terms in this application can be understood according to the specific circumstances.

[0061] It should be noted that in this application, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0062] The above description is merely a specific embodiment of this application, enabling those skilled in the art to understand or implement this application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this application. Therefore, this application is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features claimed herein.

Claims

1. A method for detecting the registration quality of a camera and a lidar system, characterized in that, The method includes the following steps: Based on the color images acquired by the camera, and combined with the depth estimation model, a dense depth map aligned with pixels is obtained. The density depth map is calculated to obtain the depth transition gradient corresponding to the color image; Based on the sparse depth map obtained by lidar, the depth transition gradient corresponding to the sparse depth map is calculated. Based on the depth transition gradient corresponding to the color image and the depth transition gradient corresponding to the sparse depth map, the effective transition region of the image and the effective transition region of the lidar are obtained respectively. Based on each lidar point in the effective transition region of the lidar, the pixel point with the closest Euclidean distance is searched in the effective transition region of the image, and the distance loss of each lidar point is calculated. The average loss per frame is obtained based on each of the lidar points whose distance loss is less than a preset distance loss threshold. Based on the average loss of a single frame corresponding to multiple sparse depth maps obtained within the data acquisition period, the sparse depth map with the largest average loss of a single frame is selected and marked as a key review frame.

2. The camera and lidar registration quality detection method as described in claim 1, characterized in that, The step of calculating the depth transition gradient corresponding to the color image from the dense depth map includes the following steps: The horizontal gradient and vertical gradient are calculated on the dense depth map respectively, and then the depth transition gradient corresponding to the color image is obtained.

3. The camera and lidar registration quality detection method as described in claim 1, characterized in that, The calculation of the depth transition gradient corresponding to the sparse depth map obtained based on the lidar includes the following steps: Based on the sparse depth map obtained by LiDAR, parallel computation using CUDA operators is performed. For each pixel in the sparse depth map, the nearest valid LiDAR depth value is searched in the four-directional neighborhood, and the depth transition gradient corresponding to the sparse depth map is calculated.

4. The camera and lidar registration quality detection method as described in claim 1, characterized in that, The step of selecting the sparse depth map with the largest single-frame average loss based on the multiple sparse depth maps obtained within the data acquisition period and marking it as a key review frame includes the following steps: A sliding window is constructed, and the average loss of a single frame corresponding to multiple sparse depth maps obtained within the data acquisition period is slid frame by frame according to a preset step size. The sparse depth map with the largest average loss in a single frame is identified and marked as a key frame for review.

5. A camera and lidar registration quality detection device, characterized in that, The device includes: The dense depth map acquisition module is used to obtain a pixel-aligned dense depth map based on the color image acquired by the camera and combined with the depth estimation model. A depth jump gradient acquisition module is used to calculate the dense depth map and obtain the depth jump gradient corresponding to the color image. The depth jump gradient acquisition module is also used to calculate the depth jump gradient corresponding to the sparse depth map based on the sparse depth map obtained by the lidar. The effective transition region acquisition module is used to obtain the effective transition region of the image and the effective transition region of the lidar based on the depth transition gradient corresponding to the color image and the depth transition gradient corresponding to the sparse depth map, respectively. The distance loss acquisition module is used to search for the pixel with the closest Euclidean distance in the effective jump area of ​​the image based on each lidar point in the effective jump area of ​​the lidar, and calculate the distance loss of each lidar point. A single-frame average loss module is used to obtain the single-frame average loss based on each of the lidar points whose distance loss is less than a preset distance loss threshold. The key review frame marking module is used to select the sparse depth map with the largest single-frame average loss based on the single-frame average loss of multiple sparse depth maps obtained within the data acquisition period, and mark it as a key review frame.

6. The camera and lidar registration quality detection device as described in claim 5, characterized in that: The depth jump gradient acquisition module is also used to perform horizontal gradient calculation and vertical gradient calculation on the dense depth map respectively, and then calculate the depth jump gradient corresponding to the color image.

7. The camera and lidar registration quality detection device as described in claim 5, characterized in that: The depth jump gradient acquisition module is also used to perform parallel computation of CUDA operators based on the sparse depth map obtained by the lidar, search for the nearest valid lidar depth value in the four-directional neighborhood of each pixel in the sparse depth map, and calculate the depth jump gradient corresponding to the sparse depth map.

8. The camera and lidar registration quality detection device as described in claim 5, characterized in that: The key review frame marking module is also used to construct a sliding window, and slide the average loss of a single frame corresponding to multiple sparse depth maps obtained within the data acquisition period frame by frame according to a preset step size. The key review frame marking module is also used to identify the sparse depth map with the largest average loss in a single frame and mark it as a key review frame.

9. A device, characterized in that, The device includes a processor and a memory, the memory being coupled to the processor, the memory storing instructions that, when executed by the processor, cause the device to perform the steps of the method according to any one of claims 1 to 4.

10. A medium having a computer program stored thereon, characterized in that, When executed by a processor, the program implements the steps of the method described in any one of claims 1 to 4.

Images