Sensor external parameter calibration method and device

Abstract

The invention discloses a sensor external parameter calibration method and device, and the method comprises the following steps: obtaining point clouds and camera images which are synchronously collected in the movement process of a mobile device, and generating a global map and a pose matrix of each frame of point cloud; and automatically selecting the key frame based on the displacement distance of the mobile device. And constructing a local point cloud set of the key frame from the global map by using the pose matrix of the key frame, projecting the local point cloud set to a pixel plane of a camera, and generating an index map and a radar intensity map. And inputting the corresponding camera image and the radar intensity map into a preset model for feature extraction and matching, and outputting a matching point pair. Constructing a 2D-3D corresponding relation of the key frames, and calculating an external parameter, a minimized re-projection error and an internal point set of each key frame; and screening effective key frames from the plurality of key frames. And performing fusion calibration by using the external parameters of the plurality of effective key frames, and outputting external parameter calibration parameters of the laser radar and the camera.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and in particular to a method and apparatus for calibrating sensor extrinsic parameters. Background Technology

[0002] With the rapid development of technologies such as autonomous driving, mobile robots, and intelligent equipment, cameras and LiDAR have become key sensors for environmental perception and localization mapping. Cameras provide high-resolution texture information, while LiDAR provides accurate 3D geometric ranging data. To achieve high-quality sensor data fusion, one of the key steps is to accurately calculate the extrinsic parameters (rotation and translation) between the camera and LiDAR, i.e., the calibration of camera-LiDAR extrinsic parameters.

[0003] Currently, existing camera and LiDAR extrinsic parameter calibration methods are mainly divided into two categories: target-based calibration methods and target-free calibration methods. Target-based methods rely on dedicated calibration objects, such as checkerboard patterns or reflectors, and can achieve high accuracy, but they are costly to implement and require complex on-site setup and operation, making them difficult to adapt to dynamic environments and large-scale deployments. Target-free calibration, on the other hand, utilizes structural and texture information in the environment for calibration, offering greater flexibility and adaptability, and is particularly suitable for online maintenance and complex scenarios.

[0004] However, existing targetless calibration methods still face many technical challenges. For example, single-frame scan data from lidar often exhibits sparse point clouds, and the point clouds may be distorted due to the movement of vehicles or sensors. This makes it difficult for data based on single-frame point clouds to provide sufficiently stable geometric and textural features, thereby affecting the accuracy of extrinsic parameters.

[0005] Therefore, in practical applications, the targetless calibration method suffers from insufficient accuracy and robustness, and there is an urgent need to improve the accuracy and robustness of the targetless calibration method. Summary of the Invention

[0006] To address the technical problems of insufficient accuracy and robustness in targetless calibration methods mentioned above, the present invention aims to provide a sensor extrinsic parameter calibration method and apparatus.

[0007] The objective of this invention is achieved through the following technical solution:

[0008] In a first aspect, the present invention provides a method for calibrating sensor extrinsic parameters, comprising the following steps:

[0009] Acquire point cloud and camera images synchronously collected during the movement of a mobile device, wherein the mobile device has a lidar and camera to be calibrated by external parameters;

[0010] A continuous-time point cloud registration algorithm is used to fuse point clouds from multiple frames to generate a global map and the pose matrix of each frame's point cloud.

[0011] When the displacement distance of the mobile device reaches a preset distance threshold, the corresponding point cloud is selected as a keyframe;

[0012] Using the pose matrix of the keyframes, a local point cloud of the keyframes is constructed from the global map;

[0013] The local point cloud is projected onto the pixel plane of the camera to generate an index map and a radar intensity map;

[0014] The corresponding camera image and radar intensity map are input into a preset model for feature extraction and matching, and the matching point pair is output. The matching point pair includes the camera image pixel and the radar intensity map pixel.

[0015] Based on the matching point pairs and the index map, a 2D-3D correspondence relationship of key frames is constructed;

[0016] Based on the 2D-3D correspondence of keyframes, calculate the extrinsic parameters, minimize the reprojection error, and the set of interior points for each keyframe;

[0017] Based on minimizing the reprojection error and the set of interior points, select valid keyframes from several keyframes;

[0018] The extrinsic parameters of multiple valid keyframes are fused and calibrated to output the extrinsic calibration parameters of the LiDAR and camera.

[0019] In one possible implementation, constructing a local point cloud set of the keyframes from the global map using the pose matrix of the keyframes specifically includes:

[0020] The point cloud of the global map is transformed using the pose matrix of the keyframe to output the first point cloud;

[0021] The first point cloud is projected onto the pixel plane of the camera using the initial extrinsic parameters of the lidar and the camera, and the projection result is output.

[0022] Based on the projection results, three-dimensional points in the first point cloud that are located in front of the camera and projected onto the pixel plane within the effective pixel area are selected.

[0023] For multiple 3D points at the same pixel location projected onto the pixel plane in the first point cloud, a depth filtering strategy is used to retain the 3D point with the smallest depth.

[0024] The set of multiple filtered 3D points is output as the local point cloud of the keyframe.

[0025] In one possible implementation, the local point cloud is projected onto the camera's pixel plane to generate an index map and a radar intensity map, specifically including:

[0026] The local point cloud is projected onto the camera's pixel plane using the initial extrinsic parameters of the lidar and camera, and a radar intensity map is generated on the pixel plane.

[0027] An index map is generated in the pixel plane, where each pixel of the index map records the index information of the 3D point in the local point cloud set;

[0028] Generate a coordinate system corresponding to the keyframe and the 3D points pointed to by the index map, using the LiDAR coordinate system of the index map. Figure 1 A corresponding local point cloud file.

[0029] In one possible implementation, the corresponding camera image and radar intensity map are input into a preset model for feature extraction and matching, and the output is a matching point pair, which includes camera image pixels and radar intensity map pixels, specifically including:

[0030] The corresponding camera images and radar intensity maps are scaled;

[0031] Input the scaled camera image and radar intensity map into the preset model, and obtain the matching point pair (kpt0, kpt1) output by the preset model, where kpt0 is the pixel of the camera image and kpt1 is the pixel of the radar intensity map.

[0032] Based on the pixel coordinates of the matching point pair (kpt0, kpt1), the corresponding camera image resolution and radar intensity map resolution are calculated.

[0033] Output the inverse-calculated matching point pair (kpt0, kpt1).

[0034] In one possible implementation, constructing the 2D-3D correspondence of keyframes based on the matching point pairs and the index map specifically includes:

[0035] Use the radar intensity map pixels of the matching point pairs to query the index information of the corresponding 3D points in the index map.

[0036] Based on the index information of the three-dimensional points, obtain the corresponding three-dimensional points in the local point cloud file;

[0037] Construct a 2D-3D observation pair (kpt0, P3D) for keyframes, where kpt0 is the camera image pixel and P3D is the 3D point of the local point cloud.

[0038] In one possible implementation, based on the 2D-3D correspondence of keyframes, the extrinsic parameters of each keyframe, the minimization of the reprojection error, and the set of interior points are calculated, specifically including:

[0039] For the 2D-3D observation pair (kpt0, P3D) of the keyframe, the preliminary extrinsic parameters are solved using a random sampling consensus algorithm;

[0040] Based on the initial extrinsic parameters, the reprojection error of the 3D point projected onto the pixel plane of the camera is minimized, and the single-frame extrinsic parameters of the keyframe are solved.

[0041] Based on whether the reprojection error is less than a preset threshold, the 2D–3D correspondence is divided into an interior point set and an exterior point set.

[0042] In one possible implementation, extrinsic parameters from multiple valid keyframes are used for fusion calibration to output extrinsic calibration parameters for the LiDAR and camera, specifically including:

[0043] The 3D points of the interior point set of each valid keyframe are projected onto the camera's pixel plane according to the corresponding single-frame extrinsic parameters, and the reprojection error of the interior point set is calculated.

[0044] Using global extrinsic parameters as the parameters to be optimized, we construct an optimization problem to minimize the sum of squared reprojection errors of all keyframe intra-point sets.

[0045] Solving this optimization problem yields global extrinsic parameters for the LiDAR and camera;

[0046] The global extrinsic parameters are output as extrinsic calibration parameters for the lidar and camera.

[0047] Secondly, the present invention also provides a sensor extrinsic parameter calibration device, comprising:

[0048] The acquisition module is used to acquire point cloud and camera images synchronously collected during the movement of the mobile device, which has a lidar and camera to be calibrated by external parameters.

[0049] The first generation module is used to fuse multiple frames of point clouds using a continuous time point cloud registration algorithm to generate a global map and the pose matrix of each frame of point cloud.

[0050] The selection module is used to select the corresponding point cloud as a keyframe when the displacement distance of the mobile device reaches a preset distance threshold.

[0051] The first construction module is used to construct a local point cloud set of the keyframes from the global map using the pose matrix of the keyframes.

[0052] The second generation module is used to project the local point cloud onto the pixel plane of the camera to generate an index map and a radar intensity map.

[0053] The processing module inputs the corresponding camera image and radar intensity map into a preset model for feature extraction and matching, and outputs matching point pairs, which include camera image pixels and radar intensity map pixels.

[0054] The second construction module is used to construct the 2D-3D correspondence of keyframes based on the matching point pairs and the index map;

[0055] The calculation module is used to calculate the extrinsic parameters, minimize the reprojection error, and the set of interior points for each keyframe based on the 2D-3D correspondence of the keyframes.

[0056] The filtering module is used to filter valid keyframes from several keyframes based on minimizing the reprojection error and the set of interior points.

[0057] The fusion module is used to perform fusion calibration using the extrinsic parameters of multiple valid keyframes, and outputs the extrinsic calibration parameters of the LiDAR and camera.

[0058] Thirdly, the present invention also provides an electronic device, comprising:

[0059] At least one processor and a memory, the at least one processor being coupled to the memory for reading and executing instructions in the memory to perform a sensor extrinsic calibration method as in any possible implementation of the first aspect.

[0060] Fourthly, the present invention also provides a computer-readable medium storing program code that, when executed on a computer, causes the computer to perform a sensor extrinsic parameter calibration method as described in any possible implementation of the first aspect.

[0061] Compared with the prior art, the present invention has at least the following beneficial effects:

[0062] This invention provides a sensor extrinsic parameter calibration method, comprising the following steps: acquiring point cloud and camera images synchronously collected during the movement of a mobile device, wherein the mobile device has a lidar and a camera to be calibrated. A continuous-time point cloud registration algorithm is used to fuse multiple frame point clouds to generate a global map and a pose matrix for each frame of point cloud. When the displacement distance of the mobile device reaches a preset distance threshold, the corresponding point cloud is selected as a keyframe. Using the pose matrix of the keyframe, a local point cloud set of the keyframe is constructed from the global map. The local point cloud set is projected onto the pixel plane of the camera to generate an index map and a radar intensity map. The corresponding camera image and radar intensity map are input into a preset model for feature extraction and matching, outputting matching point pairs, wherein the matching point pairs include pixels from the camera image and pixels from the radar intensity map. Based on the matching point pairs and the index map, a 2D-3D correspondence relationship of the keyframes is constructed. Based on the 2D-3D correspondence relationship of the keyframes, the extrinsic parameters, the minimum reprojection error, and the set of interior points for each keyframe are calculated. Based on the minimum reprojection error and the set of interior points, valid keyframes are selected from several keyframes. The extrinsic parameters of multiple valid keyframes are fused and calibrated to output the extrinsic calibration parameters of the LiDAR and camera.

[0063] This invention provides a novel scheme for sensor extrinsic parameter calibration, effectively overcoming the problems of sparse point clouds, motion distortion, and low-quality frame interference in existing targetless calibration methods. Furthermore, it improves calibration accuracy through multi-frame fusion and keyframe strategies, while enhancing the stability and robustness of calibration results through interior point selection and multi-keyframe extrinsic parameter fusion. The new scheme achieves automated targetless calibration, eliminating the need for manual calibration board deployment and utilizing natural scenes to solve extrinsic parameters. By using an "intensity map + index map + local point cloud" approach, it achieves stable mapping of matched pixels to real 3D points, avoiding uncertain correspondences. Keyframe selection and multi-frame optimization suppress single-frame noise and occasional errors, ensuring stable calibration results that can be directly applied in engineering projects. Simultaneously, it significantly reduces calibration costs, making it suitable for batch delivery, recalibration, and online self-testing scenarios. Attached Figure Description

[0064] Figure 1 This is a schematic diagram of the self-testing device of the present invention;

[0065] Figure 2 This is a schematic flowchart of a sensor extrinsic parameter calibration method according to the present invention;

[0066] Figure 3 This is a schematic diagram of a process for constructing a local point cloud set of keyframes according to the present invention;

[0067] Figure 4 This is a schematic diagram of a process for constructing the 2D-3D correspondence of keyframes according to the present invention;

[0068] Figure 5 This is a schematic diagram illustrating the process of calculating the extrinsic parameters of each keyframe, minimizing the reprojection error, and the set of interior points according to the present invention.

[0069] Figure 6 This is a schematic diagram of the electronic device of the present invention.

[0070] Figure 7 This is a point cloud image based on the initial extrinsic parameters projected onto the front pinhole camera according to the present invention;

[0071] Figure 8 This is a point cloud image of the present invention based on the projection of calibrated extrinsic parameters onto the front pinhole camera;

[0072] Figure 9 This is a point cloud image based on the initial extrinsic parameters projected onto the front fisheye camera according to the present invention;

[0073] Figure 10 This is a point cloud image based on the projection of calibrated extrinsic parameters onto a front fisheye camera, according to the present invention. Detailed Implementation

[0074] To facilitate understanding of the present invention, the technical solutions and advantages of the invention will be further described in detail below with reference to the accompanying drawings and embodiments. Any mechanisms or methods not elaborated in this invention can be referred to in the prior art. The specific structures and features of the present invention are illustrated below by way of example and should not be construed as limiting the present invention in any way. Furthermore, any of the technical features mentioned below (including implicit or disclosed features), as well as any technical features directly shown or implied in the figures, can be arbitrarily combined or deleted among these technical features to form more other embodiments that may not be directly or indirectly mentioned in this invention. The accompanying drawings show preferred embodiments of the present invention. However, the present invention can be implemented in many different forms and is not limited to the embodiments described herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of the disclosure of the present invention.

[0075] In this application embodiment, the terms "module" or "unit" refer to a computer program or part of a computer program that has a predetermined function and works with other related parts to achieve a predetermined goal, and can be implemented wholly or partially using software, hardware (such as processing circuitry or memory), or a combination thereof. Similarly, a processor (or multiple processors or memory) can be used to implement one or more modules or units. Furthermore, each module or unit can be part of an overall module or unit that includes the functionality of that module or unit.

[0076] It is understood that in the specific implementation of this application, point cloud data, images and other related data are involved. When the above embodiments of this application are applied to specific products or technologies, user permission or consent is required, and the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0077] To facilitate understanding, some terms and related concepts involved in the embodiments of this application will be introduced below.

[0078] Calibration is a process of obtaining equipment parameters. The purpose of calibration is to determine the values ​​of certain parameters of the equipment. Calibration includes internal parameter calibration and external parameter calibration.

[0079] Extrinsic parameter calibration refers to the process of determining the coordinate system transformation relationship between device A and device B. By using the extrinsic parameters between device A and device B, points in the coordinate system of device A can be mapped to the coordinate system of device B after rotation and translation, or vice versa. For example, calibrating the extrinsic parameters between a LiDAR (Light Detection and Ranging) system and a camera device involves determining the pose transformation relationship between the coordinate system of the LiDAR and the coordinate system of the camera device. The extrinsic parameters between the two devices are typically represented using rotation and translation matrices.

[0080] Optionally, the extrinsic parameters between the two devices can be adopted. Indicated. Among them, R LI This is an example of a rotation matrix, T LI This is an example of a translation matrix.

[0081] A rotation matrix represents the rotational transformation relationship between the coordinate systems of device A and device B. Optionally, the rotation matrix is ​​a 3x3 matrix, comprising 3 degrees of freedom corresponding to the x-axis, y-axis, and z-axis, respectively. The rotation matrix represents the rotational transformation relationship around these three axes. Optionally, the rotation matrix is ​​represented by the following matrix R. LI express.

[0082]

[0083] Alternatively, to simplify the expression, the rotation matrix can be represented by a quaternion, for example, denoted as q. LI Quaternion q LI For example, (w, x, y, z). R LI and q LI They are equivalent, both representing rotational transformations.

[0084] A translation matrix represents the translation transformation relationship between the coordinate systems of device A and device B. Optionally, the translation matrix is ​​3. A translation matrix of size 1 has three degrees of freedom, corresponding to the x, y, and z axes. The translation matrix represents the translation along these three axes. For example, a translation matrix is ​​represented by the following matrix T. LI express.

[0085] T LI =[t x t y t z ] T

[0086] The sensor extrinsic parameter calibration method provided in this application can be applied to the extrinsic parameter calibration scenarios of lidar. A brief introduction to the extrinsic parameter calibration scenarios of lidar is given below.

[0087] LiDAR is a new type of sensor. It consists of a laser transmitter and a receiver. The laser transmitter generates and emits a light pulse, which strikes an object and reflects back, eventually being received by the receiver. The receiver accurately measures the propagation time of the light pulse from emission to reflection. Based on the height of the laser transmitter and the laser scanning angle, the coordinates of each light spot relative to the center of the LiDAR are accurately calculated.

[0088] Because lidar ranging accuracy can reach the centimeter level, its point cloud data has high spatial precision, making it widely used in surveying, autonomous driving, intelligent robotics, and other intelligent equipment fields. In practical applications, it is often necessary to transform the point cloud information acquired by lidar from its own coordinate system to the camera coordinate system to achieve functions such as multi-sensor fusion, 3D mapping, or environmental perception. The method provided in this application can automatically calibrate the extrinsic parameters between the lidar and camera sensors, achieving accurate point cloud transformation between different coordinate systems, thereby supporting precise multi-sensor fusion applications.

[0089] See Figure 1 The diagram illustrates an application scenario provided in this application. This application scenario includes a device to be calibrated 110 and a calibration processing device 120. The number of devices to be calibrated 110 can be one or more. The number of calibration processing devices 120 can also be one or more. This application does not specifically limit the number of devices to be calibrated 110 and calibration processing devices 120. The devices to be calibrated 110 and calibration processing devices 120 can be directly or indirectly connected via wired or wireless communication, which is not limited herein.

[0090] In this embodiment, the device 110 to be calibrated is equipped with a camera and a LiDAR. The camera is used to acquire images of the environmental scene, and the LiDAR is used to acquire point cloud data of the environmental scene. The device 110 to be calibrated can be a mobile device, such as a vehicle, but is not limited thereto. For example, the vehicle can be any type of electric vehicle, gasoline vehicle, or hybrid vehicle. In a specific application, the mobile device can be an autonomous cleaning vehicle, etc. In short, this application does not limit the specific type of mobile device.

[0091] The calibration processing device 120 is used for calibrating extrinsic parameters between sensors. In this embodiment, the calibration processing device 120 also has human-computer interaction functionality.

[0092] As one possible implementation, the calibration processing device 120 can be, but is not limited to, a smartphone, tablet, laptop, desktop computer, smart speaker, smartwatch, IoT device, smart home appliance, in-vehicle terminal, or other terminal device. The terminal device can have a point cloud rendering-related client installed; this client can be software (such as a browser, map software, etc.), or a webpage, mini-program, etc.

[0093] As another possible implementation, the calibration processing device 120 can also be a server corresponding to the client. The server can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDN), and big data and artificial intelligence platforms.

[0094] The terminal device and the server can be connected directly or indirectly via wired or wireless communication, and this application does not impose any restrictions. The extrinsic parameter calibration method between sensors involved in the embodiments of this application can be implemented by the terminal device or the server alone, or it can be implemented by both the terminal device and the server.

[0095] The embodiments of this application can be applied to various scenarios, including but not limited to intelligent transportation, assisted driving, and autonomous driving. Taking the autonomous driving scenario as an example, the device to be calibrated is a vehicle. The vehicle collects data while driving in the environment and performs extrinsic parameter calibration of the LiDAR and camera.

[0096] The above describes the extrinsic parameter calibration scenarios for lidar. The following methods 200 to 400 will exemplify the process of targetless extrinsic parameter calibration based on the mobile device provided above.

[0097] See Figure 2The diagram shown is a flowchart illustrating a sensor extrinsic parameter calibration method provided in this embodiment. This sensor extrinsic parameter calibration method is a targetless extrinsic parameter calibration method. This process can be applied to calibration processing equipment, and the specific process is as follows:

[0098] S201. Acquire point cloud and camera images synchronously collected during the movement of the mobile device, wherein the mobile device has a lidar and camera to be calibrated by external parameters.

[0099] In this embodiment, the mobile device is equipped with both a LiDAR and a camera during movement, and both perform synchronous data acquisition according to a unified time base. The LiDAR continuously acquires three-dimensional point cloud data of the environment, while the camera synchronously acquires two-dimensional image data at the corresponding moment.

[0100] Understandably, the time synchronization mechanism ensures that each frame of point cloud data corresponds temporally to at least one frame of camera image, thus providing a foundation for subsequent cross-modal data association and extrinsic parameter calibration. This step avoids spatial correspondence errors caused by time drift and improves the reliability of subsequent geometric constraints.

[0101] S202. Use a continuous time point cloud registration algorithm to fuse multiple frame point clouds to generate a global map and the pose matrix of each frame point cloud.

[0102] In this embodiment of the application, a continuous time point cloud registration algorithm is used to spatially align and accumulate multiple frames of LiDAR point clouds to generate a global point cloud map, and the pose matrix of each frame of point cloud in a unified coordinate system is obtained simultaneously.

[0103] In one specific embodiment, the system performs frame-by-frame or sliding-window registration processing on multiple frames of point cloud data acquired by the lidar over continuous time. The point cloud registration algorithm can be implemented based on point cloud features, scan matching, or odometry constraints, and is used to estimate the relative pose relationship between adjacent point cloud frames, and gradually transform and accumulate the point clouds of each frame into a unified reference coordinate system.

[0104] Through the continuous time point cloud registration process, a global point cloud map covering a larger spatial range is constructed on the one hand; on the other hand, the pose matrix corresponding to each frame of point cloud is determined for subsequent key frame selection and multi-sensor data association.

[0105] It should be noted that in this embodiment, the global point cloud map densification process does not have fixed upper and lower limits for the number of fusion frames. Unlike traditional methods based on fixed window length or fixed frame accumulation, this application adopts a data-driven densification determination strategy, namely: whether the availability requirements for subsequent camera-LiDAR feature matching and extrinsic parameter calculation are met, as the basis for judging whether the point cloud densification is sufficient.

[0106] For example, when a 32-line rotating LiDAR is used, the sampling density of a single-frame point cloud in the vertical direction is limited, and the spatial coverage exhibits significant dispersion. Under these conditions, when a single-frame point cloud is directly used for intensity map rendering, it is often difficult to form a continuous and stable intensity texture structure, which is not conducive to extracting and matching a sufficient number of feature points between the camera image and the LiDAR intensity map.

[0107] Therefore, this embodiment introduces the continuous movement of the mobile device to accumulate point clouds collected at different times in a unified coordinate system, enabling the acquisition of a more spatially complete and continuously distributed "visible subset" of the point cloud from the keyframe perspective. This visible subset is further used to generate a LiDAR intensity map, thereby significantly improving the pixel coverage, texture continuity, and the number of extractable feature points of the intensity map.

[0108] In some implementations, the point cloud densification can be indirectly determined by one or more of the following indicators: spatial coverage of effective pixels in the intensity map; number of detectable feature points in the intensity map; number and stability of feature matching pairs based on the intensity map and camera image; and convergence of inlier ratio or reprojection error during subsequent extrinsic parameter calculation. When the above indicators meet the preset usability requirements, the current point cloud accumulation result is considered to meet the subsequent processing requirements.

[0109] S203. When the displacement distance of the mobile device reaches a preset distance threshold, the corresponding point cloud is selected as a key frame.

[0110] In this embodiment, the keyframes are not selected at fixed time intervals, but rather adaptively based on the actual displacement distance of the mobile device in space. Specifically, when the displacement distance of the mobile device reaches a preset distance threshold, the point cloud data collected at the corresponding time is selected as the keyframe.

[0111] In one specific embodiment, the mobile device can be a vehicle equipped with a LiDAR and a camera. While continuously acquiring point cloud data, the calibration processing device obtains spatial pose change information of the mobile device between adjacent time points through a pose estimation module. For example, the calibration processing device calculates the cumulative displacement distance of the mobile device in three-dimensional space based on the translation amount between the poses of two or more adjacent frames. When the cumulative displacement distance reaches or exceeds a preset distance threshold, it determines that the point cloud data corresponding to the current time point meets the keyframe selection criteria and marks the point cloud data as a keyframe.

[0112] By using displacement distance as the basis for keyframe selection, sufficient geometric variation is achieved between adjacent keyframes, ensuring significant differences in viewpoint and spatial distribution between point cloud data from different keyframes. This difference facilitates the establishment of stable and reliable 2D–3D correspondences in subsequent steps and improves the observability of extrinsic parameter estimation.

[0113] In some preferred embodiments, to eliminate motion distortion generated by the mobile device during single-frame laser scanning, motion distortion correction processing is performed on the selected keyframe point cloud. Specifically, based on the pose information corresponding to the start and end times of the laser scanning of the keyframe, time interpolation compensation is performed on each point in the point cloud to make the point cloud represent a unified reference time, thereby improving the consistency of the point cloud's geometric structure.

[0114] In practice, the scan start and end poses are estimated internally by CT-ICP / continuous time scan matching during the processing of the current scan, and are used for point cloud motion compensation in the deskew; the entire process does not rely on the IMU. On the other hand, camera-LiDAR keyframe synchronization: the keyframe LiDAR timestamp is used to select the image frame with the closest time from the camera buffer, and a maximum time deviation threshold is set (an alarm will be triggered if the threshold is exceeded, and the keyframe will be skipped if it is missing).

[0115] Furthermore, after completing the keyframe point cloud correction, a linear iVox (incremental Voxel) structure is used to maintain the global or historical point cloud set. The iVox structure spatially indexes the point cloud through voxelization to support efficient point cloud querying, neighborhood retrieval, and subgraph construction. This structure enables rapid access to large-scale point cloud data while maintaining low storage and computational overhead, providing an efficient data foundation for subsequent construction of local point clouds based on keyframes.

[0116] S204. Using the pose matrix of the keyframes, construct a local point cloud set of the keyframes from the global map.

[0117] In this embodiment, for each keyframe, the point cloud data in the global map is transformed into the keyframe's LiDAR coordinate system using the pose matrix corresponding to that keyframe, thereby obtaining a local point cloud representation with reference to that keyframe. This local point cloud is not obtained by truncating with a fixed spatial radius, but rather serves as a candidate set for subsequent visibility cropping, used to further filter effective point cloud subsets related to the camera's viewpoint.

[0118] In this embodiment, for each selected keyframe, a local point cloud set corresponding to the observation range of that keyframe is extracted from the global point cloud map using its pose matrix in the global map. This local point cloud set represents the three-dimensional spatial structure that the camera can theoretically observe at that keyframe moment. In this way, the originally large-scale global point cloud is transformed into a local point cloud that is highly consistent with the spatial range of a single-frame camera image, creating conditions for the accurate correlation between subsequent two-dimensional images and three-dimensional point clouds.

[0119] Understandably, based on the pose matrix corresponding to each keyframe, point cloud data within the camera's field of view for that keyframe is cropped from the global point cloud map to construct a local point cloud set corresponding to that keyframe. This local point cloud set represents the theoretically observable 3D environment information at that keyframe moment.

[0120] S205. Project the local point cloud onto the pixel plane of the camera to generate an index map and a radar intensity map.

[0121] In this embodiment, the local point cloud is projected onto the camera's pixel plane based on initial extrinsic parameters. During the projection process, the corresponding 3D point index is recorded for each pixel location, thereby generating an index map.

[0122] Simultaneously, based on the laser reflection intensity information of the three-dimensional points in the local point cloud, a radar intensity map aligned with the index map space is constructed. This radar intensity map is used to represent the intensity characteristics of the lidar in a two-dimensional pixel space. Through the construction of the index map, each pixel in the radar intensity map can be traced back to its corresponding point in three-dimensional space, achieving a deterministic mapping relationship between two-dimensional pixels and three-dimensional points.

[0123] In some implementations, the 3D points in the local point cloud are transformed to the camera coordinate system based on the initial extrinsic parameter relationship between the LiDAR and the camera, and then projected onto the camera pixel plane. For multiple 3D points projected to the same pixel location, the point closest to the camera is retained using a depth-first principle. An index map is generated based on the projection results to record the index information of the 3D point corresponding to each pixel location in the local point cloud; simultaneously, a radar intensity map spatially aligned with the index map is generated, with its pixel values ​​representing the LiDAR reflection intensity of the corresponding 3D point.

[0124] For example, the aforementioned constructed local point cloud is projected onto the camera pixel plane to generate a radar intensity map and an index map for cross-modal feature association. The 3D points are projected based on the camera's actual imaging model, which can be a pinhole model, fisheye model, or other equivalent wide-angle imaging model. During the projection process, using the initial extrinsic parameters between the camera and the lidar, the 3D points in the local point cloud are transformed from the lidar coordinate system to the camera coordinate system and further mapped onto the camera's pixel plane, thereby obtaining the pixel coordinate position corresponding to each 3D point.

[0125] By introducing a real camera model, the projection results are made consistent with the actual camera imaging process, thereby ensuring the accuracy of subsequent feature matching and geometric constraints.

[0126] S206. Input the corresponding camera image and radar intensity map into the preset model for feature extraction and matching, and output the matching point pair, which includes the camera image pixel and the radar intensity map pixel.

[0127] In this embodiment, the camera image and radar intensity map are used as cross-modal inputs and fed into a preset feature extraction and matching model. The model extracts features from the camera image and radar intensity map respectively, and establishes a correspondence between them in the feature space.

[0128] The model outputs matching point pairs, which include pixels from both the camera image and the radar intensity map. This step aligns the visual modality and the LiDAR modality at the two-dimensional pixel level, laying the foundation for subsequent geometric constraint construction.

[0129] S207. Based on the matching point pairs and the index map, construct the 2D-3D correspondence of key frames.

[0130] In this embodiment, for each radar intensity map pixel in the matched point pair, the corresponding 3D point coordinates in the local point cloud are queried using an index map, thereby establishing a one-to-one correspondence between 2D pixels in the camera image and 3D spatial points. This method constructs a keyframe-level 2D–3D correspondence set. This correspondence set directly links 2D visual observation with 3D spatial structure and serves as the core source of geometric constraints in subsequent extrinsic parameter solving.

[0131] S208. Based on the 2D-3D correspondence of key frames, calculate the extrinsic parameters of each key frame, minimize the reprojection error, and the set of interior points.

[0132] In this embodiment, for each keyframe, a 2D–3D correspondence set is constructed, and the extrinsic parameters between the LiDAR and the camera are calculated. Specifically, 3D points are projected onto the camera pixel plane according to the extrinsic parameters, and the reprojection error between the projected points and the corresponding 2D pixels is calculated. By minimizing the reprojection error, the extrinsic parameters are optimized, and simultaneously, the set of interior points that satisfy the error constraints is determined, thereby suppressing the extrinsic parameter estimation bias caused by mismatches or noise.

[0133] S209. Based on minimizing the reprojection error and the set of interior points, select valid keyframes from several keyframes.

[0134] In this embodiment, after obtaining the extrinsic parameter estimation results of multiple keyframes, the effectiveness of the keyframes is evaluated based on indicators such as the magnitude of the reprojection error, the number of inliers, and the proportion of inliers corresponding to each keyframe. Valid keyframes with stable extrinsic parameter estimation results and small reprojection errors are selected to avoid the adverse effects of low-quality observations on the final extrinsic parameter fusion result.

[0135] In some examples, the number of points in the set of inner points divided in step S208 is counted. If the number of points in the keyframe is less than a preset threshold, the frame is considered to have insufficient matching information and is discarded.

[0136] In some examples, the average reprojection error or root mean square error of points within the keyframe is calculated. If the average reprojection error is greater than a threshold, the extrinsic parameters of that frame are considered unstable and it is discarded.

[0137] For example, a comprehensive judgment can be made by combining inlier quality and reprojection error to improve the robustness of the elimination strategy.

[0138] S210: Use the extrinsic parameters of multiple valid keyframes to perform fusion calibration and output the extrinsic calibration parameters of the lidar and camera.

[0139] In this embodiment, the extrinsic parameters between the LiDAR and the camera are fused and optimized based on the extrinsic parameter estimation results of multiple valid keyframes. Multi-frame consistency constraints further reduce the impact of single-frame noise on the calibration results. The final output is the extrinsic parameter calibration parameters between the LiDAR and the camera, used to achieve accurate conversion between point cloud data and image data in different coordinate systems, thereby supporting applications such as multi-sensor fusion, environmental perception, and 3D mapping.

[0140] See Figure 3 As shown, this is a flowchart illustrating a process for constructing a local point cloud of keyframes from a global map using the pose matrix of the keyframes, as provided in an embodiment of this application. Specifically, it includes the following steps:

[0141] S301. Transform the point cloud of the global map using the pose matrix of the keyframe to output the first point cloud.

[0142] In practical implementation, all point cloud points on the global map will be used. Through the pose matrix of keyframes Transform to the keyframe lidar coordinate system .in, For a 3D point in the global coordinate system, To transform to a 3D point in the keyframe LiDAR coordinate system.

[0143] Specifically, the global point cloud is converted into a keyframe local reference frame using the inverse transformation of the pose matrix. This ensures that the spatial representation of the point cloud is continuous and consistent under the keyframe reference, facilitating the establishment of an accurate geometric correspondence with the camera projection later.

[0144] S302. Project the first point cloud onto the pixel plane of the camera using the initial extrinsic parameters of the lidar and the camera, and output the projection result.

[0145] In this embodiment, the point obtained in step S301 is... Using the initial extrinsic parameters of lidar and camera Transform to the camera coordinate system, and perform pixel-plane projection based on camera intrinsic parameters and imaging models (such as pinhole model, fisheye model, etc.) to obtain two-dimensional projected coordinates. :

[0146] ;

[0147] in, This represents the camera projection function, which maps 3D points onto the pixel plane.

[0148] Specifically, extrinsic parameters are used to transform the point cloud from the LiDAR coordinate system to the camera coordinate system, achieving cross-modal space alignment. Projection onto the pixel plane is used to determine which 3D points can be observed in the camera image.

[0149] S303. Based on the projection results, select the three-dimensional points in the first point cloud that are located in front of the camera and projected onto the pixel plane within the effective pixel area.

[0150] In this embodiment, the projected points are filtered for visibility, retaining only 3D points that meet the following conditions: located in front of the camera (for example, Z > 0); and the projected pixels fall within the camera's effective imaging area (for example, within the image boundary). This step generates the first batch of valid 3D point sets. .

[0151] Specifically, this step avoids processing occluded points or points outside the imaging area. It ensures geometric consistency between the projected point cloud and the camera image space. It reduces interference from invalid points on subsequent extrinsic parameter calculations and index mapping. This improves computational efficiency and reduces the complexity of subsequent matching algorithms.

[0152] S304. For multiple 3D points in the first point cloud that are projected onto the pixel plane at the same pixel position, a depth filtering strategy is used to retain the 3D point with the smallest depth.

[0153] In this embodiment, when multiple 3D points are projected onto the same pixel location, a depth-first principle (Z-buffer) is adopted, retaining only the 3D point with the smallest depth as the visible point corresponding to that pixel:

[0154] ;

[0155] in, This represents the depth of a 3D point in the camera coordinate system.

[0156] Specifically, it ensures that points corresponding to the same pixel are actually visible, avoiding incorrect projections caused by foreground occlusion. It achieves unique 3D point mapping for each pixel, providing reliable data for the generation of index and intensity maps.

[0157] S305. Output the set of multiple filtered 3D points as the local point cloud of the keyframe.

[0158] In this embodiment, the 3D point set after projection filtering and depth clipping is output as the local point cloud of the keyframe. The local point cloud maintains consistency with the camera viewpoint and pixel mapping, and can be directly used to generate index maps, radar intensity maps, and subsequent cross-modal matching calculations. The local point cloud simultaneously satisfies spatial continuity, visibility constraints, and 2D pixel mapping consistency. It ensures that each 3D point has a corresponding pixel position and can directly participate in 2D–3D extrinsic parameter solving.

[0159] In this embodiment, S205 is to achieve precise spatial alignment between the local point cloud and the camera image, based on the local point cloud set constructed from keyframes. Projection processing is performed to generate radar intensity maps and index maps, providing input for subsequent cross-modal feature matching and extrinsic parameter calculation. Specifically, the steps include:

[0160] S2051. The local point cloud is projected onto the pixel plane of the camera using the initial extrinsic parameters of the lidar and the camera, and a radar intensity map is generated on the pixel plane.

[0161] In this embodiment, the local point cloud is set. The 3D points in the image are obtained through the initial extrinsic parameter matrix of the LiDAR and the camera. Transform to camera coordinate system: ;

[0162] Subsequently, based on the camera's intrinsic parameters and camera model (such as pinhole model, fisheye model, etc.), the 3D points are projected onto the pixel plane to obtain the 2D pixel coordinates. .

[0163] For the visible 3D points obtained from the projection, read their lidar reflection intensity values. The algorithm then uses a depth-based cropping strategy (Z-buffer) to retain the point with the minimum depth at each pixel location as the valid point. The intensity values ​​of all valid points are then subjected to min-max normalization to form a radar intensity image that corresponds one-to-one with the pixel plane.

[0164] By utilizing extrinsic parameters to perform cross-coordinate system transformations, 3D points can be mapped onto the camera image plane. The Z-buffer strategy ensures that each pixel corresponds only to a truly visible foreground point, avoiding errors caused by occlusion. Intensity normalization improves the contrast of different points and enhances the stability of cross-modal feature matching.

[0165] S2052. Generate an index map in the pixel plane, wherein each pixel of the index map records the index information of the three-dimensional point in the local point cloud.

[0166] In this embodiment, an index map is generated on the pixel plane, where each pixel records the local point cloud corresponding to the 3D point. The index information within. Specifically: for each valid pixel The index map stores the sequence number or unique identifier of the 3D point corresponding to the pixel in the local point cloud set. For 3D points that are not projected onto valid pixels or are occluded, their corresponding pixel positions can be marked as null values ​​or invalid identifiers in the index map.

[0167] The index map enables a one-way mapping between 2D pixels and 3D points, allowing 2D features to be traced back to 3D points in space. It ensures that each pixel uniquely corresponds to a 3D point, resolving issues of projection overlap or occlusion.

[0168] S2053. Generate a coordinate system corresponding to the keyframe and the three-dimensional points pointed to by the index map, using the LiDAR coordinate system of the keyframe. Figure 1 A corresponding local point cloud file.

[0169] In this embodiment, based on all valid 3D points pointed to by the index map, they are represented by keyframe LiDAR coordinates. For reference, generate a local point cloud file. Local point cloud file With index Figure 1A one-to-one correspondence ensures that each indexed pixel can find its corresponding 3D point.

[0170] Understandably, by binding the index map and local point cloud files, 2D pixel observations can be accurately traced back to 3D spatial points. This preserves coordinate consistency, ensuring that further extrinsic parameter calculations or rendering are performed within the keyframe LiDAR reference frame.

[0171] In this embodiment, S206, the corresponding camera image and radar intensity map are input into a preset model for feature extraction and matching, and the matching point pairs are output, specifically including:

[0172] S2061. Scale the corresponding camera images and radar intensity maps.

[0173] In this embodiment, the original resolution camera image and radar intensity map are scaled to the input size required by the preset model, so that the model can efficiently process image inputs of different resolutions and ensure spatial consistency of cross-modal feature extraction. The scaling operation unifies the image size, ensuring that the feature point distribution output by the feature extraction model is consistent under different input images. Scale normalization reduces cross-modal feature matching errors caused by differences in input resolution.

[0174] S2062. Input the scaled camera image and radar intensity map into the preset model, and obtain the matching point pair (kpt0, kpt1) output by the preset model, where kpt0 is the pixel of the camera image and kpt1 is the pixel of the radar intensity map.

[0175] In this embodiment, the scaled camera image and radar intensity map are input into a preset feature matching model. The model extracts key feature points from the camera image and radar intensity map respectively through learning or algorithmic mechanisms, and performs matching in the feature space, outputting matching point pairs (kpt0, kpt1), where: kpt0 represents the pixel position of the camera image; kpt1 represents the pixel position of the radar intensity map.

[0176] Cross-modal features are extracted using a pre-defined model, and pixel-level matching is established between two modalities based on the similarity of feature descriptors. The model can be a convolutional neural network, a feature point detection + matching algorithm, or other cross-modal feature extraction framework.

[0177] S2063. Based on the pixel coordinates of the matching point pair (kpt0, kpt1), calculate the corresponding camera image resolution and radar intensity map resolution.

[0178] In this embodiment, the pixel coordinates of the matching point pair (kpt0, kpt1) output by the model are inversely calculated and mapped back to the original camera image resolution and the original radar intensity map resolution. The model output point pair is based on the scaled image coordinates and is inversely calculated back to the original resolution coordinates through a linear scaling ratio, realizing the precise position of the two-dimensional pixel in the real image.

[0179] S2064. Output the inversely calculated matching point pair (kpt0, kpt1).

[0180] In this embodiment, the output of the inverse-calculated matching point pair (kpt0, kpt1) is used as the input for the subsequent construction of the 2D–3D correspondence. The inverse-calculated point pair can be directly associated with 3D points in the local point cloud in the coordinate system of the original image and radar intensity map.

[0181] Please refer to Figure 4 This application embodiment provides a flowchart illustrating step S207, which involves constructing a 2D-3D correspondence between keyframes based on the matching point pairs and the index map. Specifically, it includes:

[0182] S401. Using the radar intensity map pixels of the matching point pair, query the index information of the corresponding 3D point in the index map.

[0183] In this embodiment, firstly, based on the matching point pair (kpt0, kpt1) obtained in step S206, the radar intensity map pixel kpt1 in the radar intensity map is queried. The index information of the 3D point corresponding to the radar intensity map pixel kpt1 is searched in the index map to obtain the unique identifier of the 3D point corresponding to this pixel in the local point cloud set.

[0184] Specifically, for each matching point pair (kpt0, kpt1), the 3D point index corresponding to pixel kpt1 in the radar intensity map is queried through the index map. The index identifies the local point cloud. The position of a three-dimensional point in the image.

[0185] The index map establishes a one-to-one correspondence between each pixel in the radar intensity map and a 3D point in the local point cloud, ensuring that the corresponding 3D point can be accurately located during reverse lookup. This lookup process is a precise retrieval based on spatial mapping relationships and is a crucial step in cross-modal data fusion.

[0186] S402. Based on the index information of the three-dimensional points, obtain the corresponding three-dimensional points in the local point cloud file.

[0187] In this embodiment, based on the 3D point index information queried in step S401, the local point cloud file is used... The corresponding 3D point P3D is obtained from the local point cloud. Specifically, each index information queried corresponds to a 3D point P3D, which is the 3D point obtained by reverse mapping from the radar intensity map pixel point kpt1.

[0188] Local point cloud file It stores all valid 3D points, and each point has a one-to-one mapping with a pixel in the index map. 3D points are extracted directly from the local point cloud file based on the index information, eliminating the need for redundant calculations and improving processing efficiency.

[0189] S403. Construct a 2D-3D observation pair (kpt0, P3D) for keyframes, where kpt0 is the camera image pixel and P3D is the 3D point of the local point cloud.

[0190] In this embodiment, based on the three-dimensional point P3D obtained from step S402 and the camera image pixel kpt0 obtained in step S206, a 2D–3D observation pair (kpt0, Pi3D) of the key frame is constructed.

[0191] Specifically, for each matching point pair (kpt0, kpt1), the 2D–3D correspondence of (kpt0, P3D) is constructed by finding the 3D point P3D in the index graph and establishing a correspondence between it and the pixel point kpt0 in the camera image.

[0192] Please refer to Figure 5 This application embodiment provides S208, which calculates the extrinsic parameters, minimizes the reprojection error, and sets of interior points for each keyframe based on the 2D-3D correspondence of keyframes. Specifically, it includes:

[0193] S501. For the 2D-3D observation pair (kpt0, P3D) of the keyframe, the preliminary extrinsic parameters are solved using a random sampling consensus algorithm.

[0194] In this embodiment, the 2D–3D correspondence of the keyframes is processed using the Random Sample Consensus Algorithm (RANSAC). RANSAC estimates extrinsic parameters (rotation matrix R and translation vector t) from randomly selected interior points. Next, the model quality is evaluated by calculating the reprojection error (the difference between the projected result and the actual observed points), and exterior points are removed from the calculation to enhance the robustness of the extrinsic parameter estimation.

[0195] Specifically, RANSAC works by randomly selecting a minimal set of points (e.g., 15 points) and using it to calculate rotations and displacements. This model is then used to validate other points, and outliers are removed based on matching quality. Initial extrinsic parameters are obtained by minimizing the reprojection error on the inliers, effectively eliminating outliers.

[0196] For example, the minimum sample set of random sampling may contain, for example, 15 2D–3D corresponding points; when the number of interior points obtained based on the model is greater than the threshold, the keyframe is determined to have sufficient geometric constraints and can be used for subsequent extrinsic parameter fusion calculation; otherwise, if the number of interior points is insufficient, the 2D–3D observation quality of the keyframe is considered to be low and it will not participate in the subsequent extrinsic parameter fusion calibration process.

[0197] The RANSAC algorithm can eliminate outliers introduced by noise or mismatches, thus providing a more accurate and stable initial extrinsic parameter estimate. This helps maintain the robustness of the extrinsic parameter estimate in complex scenes, especially when features are non-uniform or illumination varies greatly.

[0198] S502. Based on the preliminary extrinsic parameters, minimize the reprojection error of the three-dimensional point projected onto the pixel plane of the camera, and solve for the single-frame extrinsic parameters of the key frame.

[0199] In this embodiment, the initial extrinsic parameters are further optimized by minimizing the reprojection error of 3D points projected onto the camera pixel plane. The specific steps are as follows: each 3D point P3D is transformed to the camera coordinate system using the initial extrinsic parameters; the transformed 3D points are projected onto the pixel plane to obtain the element coordinates; the reprojection error of each point is calculated, i.e., the difference between the actual pixel coordinates and the predicted pixel coordinates; the rotation matrix R and the translation vector t are optimized by minimizing the sum of squares of all reprojection errors, thereby refining the extrinsic parameters.

[0200] Specifically, this process employs a nonlinear least squares optimization method, adjusting the rotation and translation matrices to minimize the reprojection error at all points. Commonly used optimization algorithms include Levenberg-Marquardt and Ceres Solver. Through this optimization process, a more accurate estimate of the extrinsic parameters is ultimately obtained.

[0201] S503. Based on whether the reprojection error is less than a preset threshold, the 2D–3D correspondence is divided into an interior point set and an exterior point set.

[0202] In this embodiment, the 2D–3D correspondence (kpt0, P3D) is divided into an interior point set and an exterior point set by calculating the reprojection error of each matching point. For each 2D–3D matching point pair (kpt0, P3D), its reprojection error is calculated; if the reprojection error is less than a preset threshold, the point pair is classified as an interior point; otherwise, the point pair is classified as an exterior point.

[0203] Reprojection error is measured by comparing the difference between the projected result and the actual observed point. The smaller the error, the better the geometric consistency of the matched point pair. Setting a reasonable error threshold can effectively filter out reliable interior points while eliminating mismatched or noise points (exterior points).

[0204] In another preferred embodiment, step S208 is based on a combination of RANSAC (Random Sample Consensus Algorithm) and PnP (Camera Localization and Attitude Solution) to handle 2D–3D correspondences. Its core idea is to accurately estimate camera extrinsic parameters by minimizing the reprojection error on the inlier set. The following is a detailed explanation of how to refine the use of RANSAC for extrinsic parameter estimation:

[0205] (1) Initial external parameter solution using RANSAC

[0206] In RANSAC, random sampling and model estimation involve randomly selecting the minimum number of point pairs (usually 3 or 4) from all corresponding 2D–3D point pairs. These point pairs will be used to estimate the camera's extrinsic parameters (including the rotation matrix R and the translation vector t).

[0207] For each randomly selected point set, the camera's rotation and translation are calculated. The extrinsic parameter model is then solved using the Perspective-n-Point (PnP) method, based on the correspondence between these 2D and 3D points. In this process, RANSAC not only calculates extrinsic parameters by randomly selecting point pairs but also utilizes an error evaluation mechanism to assess the quality of each extrinsic parameter model.

[0208] (2) Selecting the optimal model from random sampling

[0209] Interior point statistics and selection: Each time an extrinsic model is generated using RANSAC, the corresponding reprojection error is calculated, i.e., the distance between the projected points and the actual observed points is calculated using this model. Then, the error values ​​of all corresponding 2D–3D point pairs are statistically analyzed, and those point pairs with errors less than a preset threshold are selected; these are called interior points. The set of interior points contains the point pairs that best fit the extrinsic model, while exterior points are those point pairs with large errors due to noise or mismatches.

[0210] This step allows RANSAC to effectively eliminate outliers, ensuring that subsequent optimization is performed only on the set of inliers, thus improving the accuracy and robustness of extrinsic parameter estimation.

[0211] (3) Further refine the external parameter estimation

[0212] Minimizing reprojection error: After selecting the optimal extrinsic model using RANSAC, further optimization can be performed by minimizing the reprojection error. The rotation matrix R and displacement vector t are refined by optimizing the reprojection error of all interior points. By minimizing the squared reprojection error of all interior points, we can obtain a more accurate estimate of the extrinsic parameters.

[0213] Further refine the extrinsic parameters to provide a more accurate geometric relationship between the camera and LiDAR.

[0214] (4) Elimination of outliers and inliers: After minimizing the reprojection error, 2D–3D corresponding point pairs are divided into inliers and outliers based on the error threshold. Inliers are those points with errors less than the threshold, which are considered well-matched. Outliers are those points with errors greater than the threshold, which are considered mismatched or noise points and need to be removed from the data.

[0215] By combining RANSAC and PnP algorithms, robust extrinsic parameter estimation can be achieved, enabling accurate estimation of camera-LiDAR extrinsic parameters in complex and dynamic environments. This not only improves the accuracy of extrinsic parameter estimation but also effectively reduces the impact of mismatches and noise.

[0216] In this embodiment, step S210 involves fusing and calibrating the extrinsic parameters of multiple valid keyframes to output extrinsic calibration parameters for the LiDAR and camera, specifically including:

[0217] S2101. Project the 3D points of the interior point set of each valid keyframe onto the camera's pixel plane according to the corresponding single-frame extrinsic parameters, and calculate the reprojection error of the interior point set.

[0218] In this embodiment, for each valid keyframe, the 3D points in its interior point set are projected onto the camera pixel plane according to the corresponding single-frame extrinsic parameters to obtain the pixel positions. and compared with the actual observed pixels Calculate the reprojection error. Using the single-frame extrinsic parameters of each frame, map the 3D points to the pixel plane, and calculate the reprojection error as a deviation index of the image observation of each frame's extrinsic parameters.

[0219] S2102. Using global extrinsic parameters as parameters to be optimized, construct an optimization problem to minimize the sum of squared reprojection errors of all keyframe intra-point sets.

[0220] In this application, global extrinsic parameters are used as parameters to be optimized. Construct an optimization objective function that minimizes the sum of squared reprojection errors of all valid intra-frame keypoint sets:

[0221] ;

[0222] In the formula, For the effective number of keyframes, For the first The set of interior points of k keyframes This is the camera projection function.

[0223] Specifically, the extrinsic information of each keyframe is fused into a global coordinate system, and the global rotation matrix and translation vector are optimized using a nonlinear least squares method. Minimizing the sum of squared reprojection errors across all frames means that the global extrinsic parameters can achieve optimal consistency across all keyframes.

[0224] S2103. Solve the optimization problem to obtain the global extrinsic parameters for the lidar and camera.

[0225] In this embodiment, a nonlinear optimization algorithm (such as Levenberg-Marquardt or CeresSolver) is used to solve the objective function constructed in step S2102 to obtain the global rotation matrix. and global translation vector , which serves as the global extrinsic calibration parameter for the lidar and camera.

[0226] By using an iterative optimization algorithm, the global reprojection error is converged to a minimum, yielding the optimal global extrinsic parameters. Robust kernel functions (such as the Huber kernel or Tukey kernel) can be added during the optimization process to further reduce the impact of outliers on the global extrinsic parameters.

[0227] S2104. Output the global extrinsic parameters as extrinsic calibration parameters for the lidar and camera.

[0228] The global rotation matrix obtained from step S2103 and global translation vector The output serves as the external parameter calibration parameters for LiDAR and camera, providing a unified reference for subsequent point cloud projection, feature fusion, and environmental perception.

[0229] Please refer to Figures 7-10 According to embodiments of the present invention, keyframe selection, local point cloud construction, radar intensity map and indexing are utilized. Figure 3 Meta-output, PnP+RANSAC extrinsic parameter solving, and multi-frame fusion optimization enable fully automatic camera-LiDAR extrinsic parameter calibration without the need for manual calibration board setup. It can stably and robustly establish the correspondence between 2D pixels and real 3D points in natural scenes. By suppressing single-frame noise and mismatches through key frame screening and global optimization, it significantly improves calibration accuracy and repeatability. At the same time, it reduces the time cost of traditional manual calibration to the minute level, and has the advantages of high precision, low cost, and strong engineering feasibility.

[0230] Based on the same inventive concept, embodiments of this application provide a sensor extrinsic parameter calibration device, comprising:

[0231] The acquisition module is used to acquire multiple frames of point cloud and multiple camera images synchronously collected during the movement of the mobile device, wherein the mobile device has a lidar and a camera to be calibrated by external parameters.

[0232] The first generation module is used to fuse multiple frames of point clouds using a continuous time point cloud registration algorithm to generate a global map and the pose matrix of each frame of point cloud.

[0233] The selection module selects the corresponding point cloud as a keyframe when the displacement distance of the mobile device reaches a preset distance threshold.

[0234] The first construction module is used to construct a local point cloud set of the keyframes from the global map using the pose matrix of the keyframes.

[0235] The second generation module is used to project the local point cloud onto the pixel plane of the camera to generate an index map and a radar intensity map.

[0236] The processing module inputs the corresponding camera image and radar intensity map into a preset model for feature extraction and matching, and outputs matching point pairs, which include camera image pixels and radar intensity map pixels.

[0237] The second construction module is used to construct the 2D-3D correspondence of keyframes based on the matching point pairs and the index map;

[0238] The calculation module is used to calculate the extrinsic parameters, minimize the reprojection error, and the set of interior points for each keyframe based on the 2D-3D correspondence of the keyframes.

[0239] The filtering module is used to filter valid keyframes from several keyframes based on minimizing the reprojection error and the set of interior points.

[0240] The fusion module is used to perform fusion calibration using the extrinsic parameters of multiple valid keyframes, and outputs the extrinsic calibration parameters of the LiDAR and camera.

[0241] Specifically, the sensor extrinsic calibration device can execute and implement the above-mentioned sensor extrinsic calibration method.

[0242] This application also provides an electronic device, including at least one processor, a memory, and a communication interface, wherein the processor is used to execute a sensor extrinsic parameter calibration method described above.

[0243] Figure 6 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application.

[0244] like Figure 6 As shown, the electronic device 600 includes at least one processor 601, a memory 603, and a communication interface 602. The processor 601, memory 603, and communication interface 602 are communicatively connected, or they can communicate via wireless transmission or other means. The communication interface 602 is used to receive 3D point cloud data sent by a sensing module (e.g., LiDAR); the memory 603 stores computer instructions, and the processor 601 executes these computer instructions to perform the self-test method for the autonomous vehicle camera hardware in the aforementioned method embodiment.

[0245] It should be understood that in the embodiments of this application, the processor 601 may be a central processing unit (CPU), or it may be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor.

[0246] The memory 603 may include read-only memory and random access memory, and provides instructions and data to the processor 601. The memory 603 may also include non-volatile random access memory.

[0247] The memory 603 can be volatile memory or non-volatile memory, or it can include both. The non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous linked dynamic random access memory (SLDRAM), and direct rambus RAM (DR RAM).

[0248] It should be understood that the electronic device 600 according to the embodiments of this application can perform the implementation of the embodiments of this application. Figure 2-5 The method shown is described in detail above, and will not be repeated here for the sake of brevity.

[0249] This application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed in a computer, causes the computer to perform any of the above-mentioned sensor extrinsic parameter calibration methods.

[0250] This application also provides a computer program or computer program product that includes instructions that, when executed, cause a computer to perform any of the above-described sensor extrinsic parameter calibration methods.

[0251] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0252] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein can be implemented in hardware, processor-executed software modules, or a combination of both. The software modules can be located in random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disks, removable disks, CD-ROMs, or any other form of storage medium known in the art.

[0253] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of this application. It should be understood that the above description is only a specific embodiment of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of protection of this application.

Claims

1. A method for calibrating extrinsic parameters of a sensor, characterized in that, The method comprises the following steps: Obtaining point clouds and camera images synchronously collected during the movement of a mobile device, the mobile device having a laser radar and a camera to be extrinsic parameter calibrated; Fusing multiple frames of point clouds by using a continuous-time point cloud registration algorithm to generate a global map and a pose matrix of each frame of point clouds; When the displacement distance of the mobile device reaches a preset distance threshold, selecting a corresponding point cloud as a key frame; Constructing a local point cloud set of the key frame from the global map by using the pose matrix of the key frame; Projecting the local point cloud set onto a pixel plane of the camera to generate an index map and a radar intensity map; Inputting the corresponding camera image and radar intensity map into a preset model for feature extraction and matching to output a matching point pair, the matching point pair comprising a camera image pixel point and a radar intensity map pixel point; Based on the matching point pair and the index map, constructing a 2D-3D correspondence relationship of the key frame; Based on the 2D-3D correspondence relationship of the key frame, calculating the extrinsic parameter of each key frame, minimizing the re-projection error, and obtaining an inlier set; According to the minimized re-projection error and the inlier set, screening effective key frames from the key frames; Fusing and calibrating by using the extrinsic parameters of the multiple effective key frames to output the extrinsic parameter calibration parameters of the laser radar and the camera. 2.The sensor extrinsic calibration method of claim 1, wherein, The method of constructing the local point cloud set of the key frame from the global map by using the pose matrix of the key frame specifically comprises: Transforming the point clouds of the global map by using the pose matrix of the key frame to output a first point cloud; Projecting the first point cloud onto a pixel plane of the camera by using the initial extrinsic parameters of the laser radar and the camera to output a projection result; Based on the projection result, screening three-dimensional points in the first point cloud that are located in front of the camera and projected into the effective pixel area of the pixel plane; For multiple three-dimensional points in the first point cloud that are projected into the same pixel position of the pixel plane, a depth screening strategy is used to retain the three-dimensional point with the smallest depth; Outputting the set of the screened multiple three-dimensional points as the local point cloud set of the key frame.

3. The sensor extrinsic parameter calibration method of claim 1, wherein, The method of projecting the local point cloud set onto the pixel plane of the camera to generate the index map and the radar intensity map specifically comprises: Projecting the local point cloud set onto the pixel plane of the camera by using the initial extrinsic parameters of the laser radar and the camera to generate the radar intensity map on the pixel plane; Generating the index map on the pixel plane, wherein each pixel of the index map records the index information of a three-dimensional point in the local point cloud set; Generating a local point cloud file corresponding to the index map in a one-to-one manner by using the laser radar coordinate system corresponding to the key frame.

4. The sensor extrinsic parameter calibration method of claim 3, wherein, The method of inputting the corresponding camera image and radar intensity map into a preset model for feature extraction and matching to output a matching point pair, the matching point pair comprising a camera image pixel point and a radar intensity map pixel point, specifically comprises: Scaling the corresponding camera image and radar intensity map; Inputting the scaled camera image and radar intensity map into the preset model to obtain the matching point pair (kpt0, kpt1) output by the preset model, wherein kpt0 is a camera image pixel point and kpt1 is a radar intensity map pixel point; Based on the pixel point coordinates of the matching point pair (kpt0, kpt1), back calculation is performed to the corresponding camera image resolution and radar intensity map resolution; The back calculated matching point pair (kpt0, kpt1) is output.

5. The sensor extrinsic parameter calibration method of claim 4, wherein, The 2D-3D correspondence relationship of the key frame is constructed based on the matching point pair and the index map, specifically including: Using the radar intensity image pixel point of the matching point pair, the index information of the corresponding three-dimensional point in the index map is queried; According to the index information of the three-dimensional point, the corresponding three-dimensional point is obtained in the local point cloud file; The 2D-3D observation pair (kpt0, P3D) of the key frame is constructed, wherein kpt0 is a camera image pixel point and P3D is a three-dimensional point of a local point cloud set.

6. The sensor extrinsic parameter calibration method of claim 5, wherein, Based on the 2D-3D correspondence relationship of the key frame, the extrinsic parameter of each key frame, the minimization of the reprojection error and the inlier set are calculated, specifically including: The initial extrinsic parameter is solved based on the 2D-3D observation pair (kpt0, P3D) of the key frame by using the random sample consensus algorithm; Based on the initial extrinsic parameter, the single-frame extrinsic parameter of the key frame is solved by minimizing the reprojection error of the three-dimensional point projected onto the pixel plane of the camera; According to whether the reprojection error is less than a preset threshold, the 2D-3D correspondence relationship is divided into an inlier set and an outlier set.

7. The sensor extrinsic parameter calibration method of claim 1, wherein, The extrinsic parameters of a plurality of valid key frames are used for fusion calibration, and the extrinsic calibration parameters of the lidar and the camera are output, specifically including: The three-dimensional points of the inlier set of each valid key frame are projected onto the pixel plane of the camera according to the corresponding single-frame extrinsic parameter, and the reprojection error of the inlier set is calculated; A global extrinsic parameter is used as a to-be-optimized parameter, and an optimization problem is constructed to minimize the sum of squares of the reprojection errors of all key frame inlier sets; The optimization problem is solved to obtain the global extrinsic parameter for the lidar and the camera; The global extrinsic parameter is output as the extrinsic calibration parameter of the lidar and the camera.

8. A sensor extrinsic parameter calibration apparatus, characterized in that, It includes: An acquisition module is configured to acquire point clouds and camera images synchronously collected during movement of a mobile device, the mobile device having a lidar and a camera to be extrinsically calibrated; A first generation module is configured to fuse a plurality of frames of point clouds by using a continuous-time point cloud registration algorithm to generate a global map and a pose matrix of each frame of point cloud; A selection module is configured to select a corresponding point cloud as a key frame when a displacement distance of the mobile device reaches a preset distance threshold; A first construction module is configured to construct a local point cloud set of the key frame from the global map by using a pose matrix of the key frame; A second generation module is configured to project the local point cloud set to a pixel plane of the camera to generate an index map and a radar intensity map; A processing module is configured to input corresponding camera images and radar intensity maps into a preset model for feature extraction and matching, and output a matching point pair, the matching point pair including a camera image pixel point and a radar intensity map pixel point; A second construction module is configured to construct a 2D-3D correspondence relationship of the key frame based on the matching point pair and the index map; A calculation module is configured to calculate an extrinsic parameter of each key frame, minimize a reprojection error, and calculate an inlier set based on a 2D-3D correspondence relationship of the key frame. a screening module configured to screen valid key frames from the key frames according to a minimum reprojection error and an inlier set; a fusion module configured to fuse calibration by using the extrinsic parameters of the valid key frames, and output extrinsic calibration parameters of the lidar and the camera.

9. An electronic device, comprising: comprise: at least one processor and a memory, the at least one processor coupled with the memory and configured to read and execute instructions in the memory to perform a sensor extrinsic calibration method as claimed in any one of claims 1 to 7.

10. A computer readable medium characterized by The computer readable medium stores program code which, when executed on a computer, causes the computer to perform a sensor extrinsic calibration method as claimed in any one of claims 1 to 7.

Images