Laser radar and camera matching method and device, electronic equipment and storage medium

Abstract

The embodiment of the invention provides a laser radar and camera matching method and device, electronic equipment and a storage medium, and relates to the technical field of computer vision. The method comprises the following steps: acquiring an original image shot by a camera, and converting the original image into a dedistorted image based on a preset internal reference matrix and a preset dedistorted matrix; acquiring a point cloud image which is acquired by the laser radar and corresponds to the original image; fitting the point cloud image with the dedistorted image to obtain a fitted image; selecting a plurality of target points in the fitting image; obtaining two-dimensional coordinates of the plurality of target points in the fitting image and three-dimensional coordinates of the plurality of target points in the point cloud image; determining a fusion matrix based on the two-dimensional coordinates and the three-dimensional coordinates; by designing automatic calibration of the internal reference of the camera and automatic calibration of the external reference from the laser radar to the camera, the accurate and reliable internal reference of the camera and the external reference from the laser radar to the camera can be quickly obtained only through a small amount of manual intervention in the whole calibration process, and the robustness of a calibration algorithm and the accuracy of a calibration result are improved.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and more particularly to a method, apparatus, electronic device, and storage medium for matching a lidar and a camera. Background Art

[0002] With the continuous development of computer vision, in order to enable devices to better learn and perceive the surrounding environment, a multi-sensor fusion method is usually adopted. For example, a method of fusing a radar and a camera is adopted.

[0003] In the prior art, external parameter calibration is usually achieved based on feature point matching. Such methods extract feature points from images and lidar data, and then perform feature point matching to solve the external parameter matrix from the lidar to the camera. The feature points extracted by such methods are usually points such as corner points and edge points that are significantly different from the points in the surrounding area. In actual operation, the requirements for point selection are relatively high, and due to the uncertainty of point selection, the calibration accuracy and robustness cannot be well guaranteed. Summary of the Invention

[0004] In view of this, an object of the present invention is to provide a method, apparatus, electronic device, and storage medium for matching a lidar and a camera. By designing the automatic calibration of the internal parameters of the camera and the automatic calibration of the external parameters from the lidar to the camera, only a small amount of manual intervention is required during the entire calibration process, and accurate and reliable camera internal parameters and external parameters from the lidar to the camera can be obtained quickly, improving the robustness of the calibration algorithm and the accuracy of the calibration results.

[0005] In a first aspect, an embodiment of the present invention provides a method for matching a lidar and a camera, including: obtaining an original image captured by a camera, and converting the original image into a de-distorted image based on a preset internal parameter matrix and a preset de-distortion matrix; obtaining a point cloud map collected by the lidar corresponding to the original image; fitting the point cloud map and the de-distorted image to obtain a fitted image; selecting a plurality of target points in the fitted image; obtaining the two-dimensional coordinates of the plurality of target points in the fitted image and the three-dimensional coordinates in the point cloud map; and determining a fusion matrix based on the two-dimensional coordinates and the three-dimensional coordinates.

[0006] In some preferred embodiments of the present invention, the internal parameter matrix is determined by the following steps: obtaining a plurality of calibration images collected by the camera; wherein the plurality of calibration images are images of the same calibration board taken from different angles; obtaining the coordinates of the same feature point in the plurality of calibration images; and determining the internal parameter matrix of the camera based on the coordinates of the plurality of same feature points in the plurality of calibration images.

[0007] In some preferred embodiments of the present invention, the de-distortion matrix is determined by the following steps: determining the de-distortion matrix based on the plurality of calibration images and the internal parameter matrix through a preset function; wherein the de-distortion matrix includes radial distortion coefficients and tangential distortion coefficients.

[0008] In some preferred embodiments of the present invention, after the step of determining the fusion matrix based on two-dimensional coordinates and three-dimensional coordinates, the method further includes: taking the fusion matrix determined based on two-dimensional coordinates and three-dimensional coordinates as the basic fusion matrix; selecting detection points in the fitting image, and determining the first two-dimensional coordinates of the detection points in the fitting image and the first three-dimensional coordinates in the point cloud map; determining the second two-dimensional coordinates based on the first three-dimensional coordinates and the basic fusion matrix; calculating the error between the first two-dimensional coordinates and the second two-dimensional coordinates; if the error is less than a preset error threshold, determining the basic fusion matrix as the fusion matrix.

[0009] In some preferred embodiments of the present invention, after the step of calculating the error between the first two-dimensional coordinates and the second two-dimensional coordinates, the method further includes: if the error is greater than or equal to the preset error threshold, inputting the internal parameter matrix and the de-distortion matrix into a preset parameter adjustment model to output the fusion matrix.

[0010] In some preferred embodiments of the present invention, the method further includes: fusing the data collected by the lidar and the data collected by the camera based on the fusion matrix.

[0011] In a second aspect, an embodiment of the present invention provides a matching device for a lidar and a camera, including: a basic image determination module, configured to obtain an original image captured by the camera, and convert the original image into a de-distorted image based on a preset internal parameter matrix and a preset de-distortion matrix; a point cloud image determination module, configured to obtain a point cloud map corresponding to the original image collected by the lidar; an image fitting module, configured to fit the point cloud map and the de-distorted image to obtain a fitting image; a target point selection module, configured to select a plurality of target points in the fitting image; a coordinate determination module, configured to obtain the two-dimensional coordinates of the plurality of target points in the fitting image and the three-dimensional coordinates in the point cloud map; a fusion matrix determination module, configured to determine the fusion matrix based on the two-dimensional coordinates and the three-dimensional coordinates.

[0012] In some preferred embodiments of the present invention, the device further includes: an internal parameter matrix determination module, configured to obtain a plurality of calibration images collected by the camera; wherein, the plurality of calibration images are images of the same calibration board taken from different angles; obtaining the coordinates of the same feature point in the plurality of calibration images; determining the internal parameter matrix of the camera based on the coordinates of the plurality of same feature points in the plurality of calibration images.

[0013] In a third aspect, an embodiment of the present invention provides an electronic device, including a processor and a memory, where the memory stores computer-executable instructions that can be executed by the processor, and the processor executes the computer-executable instructions to implement the above-mentioned matching method for the lidar and the camera.

[0014] Fourthly, an embodiment of the present invention provides a storage medium storing computer-executable instructions, which, when called and executed by a processor, cause the processor to implement the above-mentioned method for matching a lidar and a camera.

[0015] The present invention brings the following beneficial effects:

[0016] An embodiment of the present invention provides a method, device, electronic device, and storage medium for matching a lidar and a camera. The method includes: obtaining an original image captured by a camera, and converting the original image into a de-distorted image based on a preset internal parameter matrix and a preset de-distortion matrix; obtaining a point cloud map collected by the lidar corresponding to the original image; fitting the point cloud map and the de-distorted image to obtain a fitted image; selecting a plurality of target points in the fitted image; obtaining the two-dimensional coordinates of the plurality of target points in the fitted image and the three-dimensional coordinates in the point cloud map; determining a fusion matrix based on the two-dimensional coordinates and the three-dimensional coordinates; by designing automatic calibration of the internal parameters of the camera and automatic calibration of the external parameters from the lidar to the camera, only a small amount of manual intervention is required during the entire calibration process, and accurate and reliable camera internal parameters and external parameters from the lidar to the camera can be obtained quickly, improving the robustness of the calibration algorithm and the accuracy of the calibration result. Description of the Drawings

[0017] In order to more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the following will briefly introduce the drawings required for use in the description of the specific embodiments or the prior art. Obviously, the drawings in the following description are some embodiments of the present invention. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0018] Figure 1 It is a flowchart of a method for matching a lidar and a camera provided by an embodiment of the present invention;

[0019] Figure 2 It is a flowchart of another method for matching a lidar and a camera provided by an embodiment of the present invention;

[0020] Figure 3 It is a flowchart of yet another method for matching a lidar and a camera provided by an embodiment of the present invention;

[0021] Figure 4 It is a schematic diagram of an automatic internal parameter calibration module provided by an embodiment of the present invention;

[0022] Figure 5 It is a schematic structural diagram of another matching device for a lidar and a camera provided by an embodiment of the present invention;

[0023] Figure 6A schematic structural diagram of an electronic device provided by an embodiment of the present invention.

[0024] Icons: 1000 - Internal parameter automatic calibration module; 310 - Basic image determination module; 320 - Point cloud image determination module; 330 - Image fitting module; 340 - Target point selection module; 350 - Coordinate determination module; 360 - Fusion matrix determination module; 400 - Memory; 401 - Processor; 402 - Bus; 403 - Communication interface. Detailed implementation manners

[0025] To make the objectives, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. Apparently, the described embodiments are some but not all of the embodiments of the present invention. Components of the embodiments of the present invention described and illustrated in the accompanying drawings here can be arranged and designed in various different configurations.

[0026] Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the claimed present invention, but merely represents selected embodiments of the present invention. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts fall within the scope of protection of the present invention.

[0027] It should be noted that like reference numerals and letters denote like items in the following drawings. Therefore, once an item is defined in one drawing, it does not need to be further defined and explained in subsequent drawings.

[0028] In the description of the present invention, it should be noted that the terms "center", "upper", "lower", "left", "right", "vertical", "horizontal", "inner", "outer", etc. indicate the orientation or positional relationship based on the orientation or positional relationship shown in the drawings, or the orientation or positional relationship in which the product of the present invention is usually placed during use. These are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and thus should not be construed as a limitation of the present invention. In addition, the terms "first", "second", "third", etc. are only used for descriptive distinction and cannot be construed as indicating or implying relative importance.

[0029] In addition, terms such as "horizontal", "vertical", "overhanging", etc. do not mean that the components are required to be absolutely horizontal or overhanging, but can be slightly inclined. For example, "horizontal" only means that its direction is more horizontal relative to "vertical", and does not mean that the structure must be completely horizontal, but can be slightly inclined.

[0030] In the description of the present invention, it should also be noted that, unless otherwise clearly specified and defined, the terms "set", "install", "connect", and "couple" should be understood in a broad sense. For example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be directly connected or indirectly connected through an intermediate medium, and it can be the communication inside two components. For those of ordinary skill in the art, the specific meanings of the above terms in the present invention can be understood according to specific situations.

[0031] With the continuous development of computer vision, in order to enable devices to better learn and perceive the surrounding environment, a multi-sensor fusion method is usually adopted. For example, a method of fusing radar and camera is adopted. In the prior art, external parameter calibration is usually achieved based on feature point matching. Such methods extract feature points from images and lidar data, and then perform feature point matching to solve the external parameter matrix from the lidar to the camera. The feature points extracted by such methods are usually points such as corner points and edge points that are significantly different from the points in the surrounding area. In actual operation, the requirements for point selection are relatively high, and due to the uncertainty of point selection, its calibration accuracy and robustness cannot be well guaranteed.

[0032] The calibration technology of lidar and camera is one of the key technologies in the fields of computer vision, robotics, autonomous driving, etc., and is mainly used to match and fuse the data from two different sensors, namely lidar and camera. In the above application fields, lidar and camera are often used simultaneously to obtain more accurate and comprehensive environmental information. In the field of autonomous driving, sensors such as lidar and camera are often used to jointly perceive the surrounding environment to assist the vehicle in tasks such as positioning, obstacle detection, and path planning. In order to achieve high-precision environmental perception, it is necessary to accurately fuse the data of different sensors, which requires the external parameter calibration of lidar and camera. The so-called external parameter calibration is to obtain the transformation matrix from the lidar coordinate system to the camera coordinate system through corresponding algorithms. In robot navigation and SLAM (Simultaneous Localization and Mapping) technology, lidar and camera are often used to construct environmental maps, locate robots, and perceive the surrounding environment. Lidar provides high-precision distance information, while the camera can obtain visual information such as color and texture. Fusing the data of the two can improve the accuracy and robustness of robot navigation and mapping.

[0033] The prior art mainly includes: Extrinsic parameter calibration methods based on feature point matching: Such methods extract feature points from images and lidar data and then perform feature point matching to solve the extrinsic parameter matrix from the lidar to the camera. The feature points extracted by such methods are usually points with obvious differences from the points in the surrounding area, such as corner points and edge points; Extrinsic parameter calibration methods based on special calibration plates: Such methods usually use calibration plates with significant edge features (such as checkerboards), use the camera to capture images of the calibration plate, and then use the lidar to obtain the three-dimensional coordinates of the corresponding points on the calibration plate to solve the transformation matrix from the lidar to the camera. Since there is a relatively accurate correspondence between the pixel points and 3D points selected by such methods, the calibration results are usually relatively accurate; Extrinsic parameter calibration methods based on deep learning: Such methods train corresponding deep learning models by manually annotating a large amount of accurate calibration data, and then use the trained models to infer the extrinsic transformation parameters for aligning the 3D point cloud with the 2D image. This method does not require manual feature design and can directly infer the extrinsic parameters from the lidar to the camera based on the input images and point clouds.

[0034] However, for the extrinsic parameter calibration methods based on feature point matching: Since the calibration results of such methods depend on the quality of feature point extraction and the accuracy of feature point matching, their calibration accuracy and robustness cannot be well guaranteed; For the extrinsic parameter calibration methods based on special calibration plates: Since special calibration plates need to be made before calibration, and the production and use processes of the calibration plates are relatively cumbersome, there are problems of high calibration cost and low calibration efficiency. And such methods need to ensure that the calibration plate appears completely in the fields of view of both the camera and the lidar, with a poor scope of application; For the extrinsic parameter calibration methods based on deep learning: Although such methods are convenient and fast during inference, the previous data annotation and model training and parameter tuning work consume a large amount of time and effort, and have high requirements for computing resources and poor model interpretability. These above problems limit the practical implementation and application of the extrinsic parameter calibration algorithms based on deep learning.

[0035] The technical problem to be solved by the present invention is to design a complete set of automatic calibration systems for lidar and cameras, including an automatic calibration system for the internal parameters of the camera and an automatic calibration system for the external parameters from the lidar to the camera. This automatic calibration system realizes reliable, efficient, and low-cost calibration of the internal parameters of the camera and the external parameters from the lidar to the camera without relying on special calibration plates or expensive computing resources. First, the Zhang Zhengyou calibration method is used to obtain the internal parameter matrix (focal lengths fx, fy, optical center coordinates cx, cy) and distortion coefficients (k1, k2, p1, p2, k3) of the camera. Then, the internal parameter matrix and distortion coefficients are used to correct the distortion of the images collected in any scene. Subsequently, the distortion-corrected images are visualized with the point clouds collected at the same time and in the same scene. By manually selecting the corresponding points in the images and point clouds, the reliability of the correspondence between 2D image pixels and 3D points is ensured. Furthermore, the PnP algorithm and the pixel coordinates and 3D coordinates of these corresponding points are used to solve the external parameter transformation matrix for converting from the lidar coordinate system to the camera coordinate system. If the manually selected pixel points correspond precisely to the 3D points, the external parameters solved by the PnP algorithm can be directly used as the final calibration results. However, if there are deviations between the manually selected pixel points and the 3D points, the external parameters solved by the PnP algorithm can still be used as relatively accurate initial values for the subsequent external parameter automatic calibration algorithm based on the image segmentation large model. Finally, the accurate external parameter matrix is obtained through this automatic calibration algorithm.

[0036] The proposed internal and external parameter automatic calibration system of the present invention only requires a small amount of manual intervention (manually selecting a small number of corresponding points) during the entire calibration process, and can quickly obtain accurate and reliable internal parameters of the camera and external parameters from the lidar to the camera. Moreover, this automatic calibration system has good algorithm robustness, can adapt to the calibration requirements under various different environmental conditions, improves the accuracy and stability of the multi-sensor fusion system, and provides a reliable and efficient internal and external parameter calibration solution for applications in fields such as autonomous driving and robot navigation.

[0037] The following will describe in detail some embodiments of the present invention with reference to the accompanying drawings. Without conflict, the following embodiments and the features in the embodiments can be combined with each other.

[0038] Embodiment 1

[0039] In modern technology, image processing and point cloud processing are two important research fields. They each have a wide range of applications, such as machine vision, autonomous driving, and augmented reality. However, due to the different characteristics of image and point cloud data, directly fusing them together often cannot obtain ideal results. Therefore, a series of preprocessing and conversion steps are required to convert the image and point cloud data into a format that can be fused with each other.

[0040] An embodiment of the present invention provides a method for matching a lidar and a camera. Refer to Figure 1 the flowchart of a method for matching a lidar and a camera provided by an embodiment of the present invention as shown in

[0041] Step S102, obtain the original image captured by the camera, and convert the original image into a de-distorted image based on a preset internal parameter matrix and a preset de-distortion matrix.

[0042] Specifically, we need to obtain the original image captured by the camera. This process is usually completed by a camera. The camera can convert light into an electrical signal, and then the signal processor converts the electrical signal into a digital signal and finally stores it as an image file. During this process, the internal parameters of the camera (such as focal length, principal point coordinates, etc.) will affect the imaging effect of the image, and these parameters are usually referred to as the internal parameter matrix; we need to convert the original image into a de-distorted image based on a preset internal parameter matrix and a preset de-distortion matrix. The de-distortion matrix is used to eliminate the distortion in the image (such as lens distortion, perspective distortion, etc.), and it is usually calculated based on the internal parameter matrix of the camera and the shooting conditions (such as focal length, field of view angle, etc.). By multiplying the original image by the de-distortion matrix, we can eliminate the distortion in the image and obtain a more accurate image.

[0043] Step S104, obtain the point cloud map collected by the lidar corresponding to the original image.

[0044] Specifically, we also need to obtain the point cloud map collected by the lidar corresponding to the original image. A lidar is a device that can measure the distance of an object. It emits laser pulses and receives the reflected laser pulses, and can calculate the round-trip time of the laser pulses to obtain the distance of the object. By rotating the lidar, we can obtain the three-dimensional information of the object, and this information is usually represented in the form of a point cloud.

[0045] Step S106, fit the point cloud map and the de-distorted image to obtain a fitted image.

[0046] Specifically, we need to fit the point cloud map and the de-distorted image to obtain a fitted image. This process is usually completed by the ICP (Iterative Closest Point) algorithm. The ICP algorithm is an iterative closest point matching algorithm that can align a set of points (such as a point cloud map) with another set of points (such as a de-distorted image) so that the distance between the corresponding points in the two point sets is minimized. By the ICP algorithm, we can align the point cloud map and the de-distorted image to obtain a fitted image.

[0047] Step S108, select multiple target points in the fitted image.

[0048] Specifically, in the fitted image, we select multiple target points. These target points can be specific objects in the image (such as vehicles, pedestrians, etc.) or specific regions in the image (such as roads, buildings, etc.). By selecting the target points, we can divide the image into multiple regions, with each region corresponding to a target point.

[0049] Step S110, obtain the two-dimensional coordinates of multiple target points in the fitted image and the three-dimensional coordinates in the point cloud map.

[0050] Specifically, we need to obtain the two-dimensional coordinates of multiple target points in the fitted image and the three-dimensional coordinates in the point cloud map. This process is usually completed through feature extraction and matching. Feature extraction is the process of converting image or point cloud data into feature vectors that can be used for comparison, while feature matching is the process of comparing two feature vectors to find the most similar pair. Through feature extraction and matching, we can obtain the two-dimensional and three-dimensional coordinates of the target points.

[0051] Step S112, determine the fusion matrix based on the two-dimensional coordinates and the three-dimensional coordinates.

[0052] Specifically, we need to determine the fusion matrix based on the two-dimensional coordinates and the three-dimensional coordinates. The fusion matrix is used to fuse the two-dimensional coordinates and the three-dimensional coordinates together. It is usually a 4x4 matrix. By taking the two-dimensional coordinates and the three-dimensional coordinates as the two inputs of the fusion matrix respectively, we can fuse them together to obtain a new coordinate system. This new coordinate system contains both the information of the two-dimensional coordinates and the information of the three-dimensional coordinates, and can be used for subsequent data processing and analysis.

[0053] An embodiment of the present invention provides a matching method for a lidar and a camera. The method includes: obtaining an original image captured by the camera, and converting the original image into a de-distorted image based on a preset internal parameter matrix and a preset de-distortion matrix; obtaining a point cloud map collected by the lidar corresponding to the original image; fitting the point cloud map with the de-distorted image to obtain a fitted image; selecting multiple target points in the fitted image; obtaining the two-dimensional coordinates of the multiple target points in the fitted image and the three-dimensional coordinates in the point cloud map; determining a fusion matrix based on the two-dimensional coordinates and the three-dimensional coordinates; by designing the automatic calibration of the internal parameters of the camera and the automatic calibration of the external parameters from the lidar to the camera, only a small amount of manual intervention is required during the entire calibration process, and accurate and reliable camera internal parameters and external parameters from the lidar to the camera can be obtained quickly, improving the robustness of the calibration algorithm and the accuracy of the calibration result.

[0054] Embodiment Two

[0055] Based on the above embodiment, an embodiment of the present invention provides another matching method for a lidar and a camera. Refer to Figure 2Another flowchart of the matching method for a lidar and a camera provided by the embodiment of the present invention shown in the figure, the method comprising:

[0056] Step S202, obtain the original image captured by the camera, and convert the original image into a de-distorted image based on a preset internal parameter matrix and a preset de-distortion matrix.

[0057] Further, in some preferred embodiments of the present invention, the internal parameter matrix is determined by the following steps: obtain multiple calibration images collected by the camera; wherein, the multiple calibration images are images of the same calibration board taken from different angles; obtain the coordinates of the same feature point in the multiple calibration images; determine the internal parameter matrix of the camera based on the coordinates of multiple same feature points in the multiple calibration images.

[0058] Specifically, determining the internal parameter matrix of the camera is an important task in computer vision, which can help us better understand how the camera captures and represents the world. This process generally involves obtaining multiple calibration images collected by the camera, which are usually images of the same calibration board taken from different angles. Then, we need to obtain the coordinates of the same feature point in the multiple calibration images. Finally, we can determine the internal parameter matrix of the camera based on the coordinates of multiple same feature points in the multiple calibration images.

[0059] First, we need to obtain multiple calibration images collected by the camera. These images are usually images of the same calibration board taken from different angles. The calibration board is an object with known size and shape, which usually contains a series of evenly distributed feature points. By taking pictures of these calibration boards at different angles and distances, we can obtain a set of images containing a lot of information. Next, we need to obtain the coordinates of the same feature point in the multiple calibration images. This can be done by using computer vision algorithms, such as corner detection or feature matching algorithms. These algorithms can help us find specific feature points in the image and calculate their exact positions in the image. Then, we can determine the internal parameter matrix of the camera based on the coordinates of multiple same feature points in the multiple calibration images. The internal parameter matrix is a matrix that describes how the camera captures and represents the world, and it contains information such as the focal length of the camera and the position of the principal point. By comparing the coordinates of the same feature point in different images, we can calculate this internal parameter matrix. This process may require some complex mathematical calculations, but there are many ready-made computer vision libraries that can help us complete these calculations. For example, OpenCV is a widely used computer vision library that provides many tools and functions for processing images and cameras.

[0060] Generally speaking, determining the intrinsic matrix of a camera is a process involving image processing, computer vision, and mathematical calculations. Although it may require some professional knowledge and skills, through learning and practice, we can master this process and utilize it to improve our camera systems and applications.

[0061] Furthermore, in some preferred embodiments of the present invention, the distortion correction matrix is determined through the following steps: the distortion correction matrix is determined based on multiple calibration images and the intrinsic matrix through a preset function; wherein, the distortion correction matrix includes radial distortion coefficients and tangential distortion coefficients.

[0062] Specifically, determining the distortion correction matrix is an important task in computer vision, which can help us better understand how the camera captures and represents the world. This process generally involves obtaining multiple calibration images captured by the camera, which are usually taken of the same calibration board from different angles. Then, we need to obtain the coordinates of the same feature point in multiple calibration images. Finally, we can determine the intrinsic matrix of the camera based on the coordinates of multiple same feature points in multiple calibration images.

[0063] First, we need to obtain multiple calibration images captured by the camera. These images are usually taken of the same calibration board from different angles. The calibration board is an object with known size and shape, which usually contains a series of uniformly distributed feature points. By taking pictures of these calibration boards at different angles and distances, we can obtain a set of images containing a large amount of information. Next, we need to obtain the coordinates of the same feature point in multiple calibration images. This can be accomplished by using computer vision algorithms, such as corner detection or feature matching algorithms. These algorithms can help us find specific feature points in the image and calculate their exact positions in the image. Then, we can determine the intrinsic matrix of the camera based on the coordinates of multiple same feature points in multiple calibration images. The intrinsic matrix is a matrix that describes how the camera captures and represents the world, and it contains information such as the focal length of the camera and the position of the principal point. By comparing the coordinates of the same feature point in different images, we can calculate this intrinsic matrix. After obtaining the intrinsic matrix, we can determine the distortion correction matrix through a preset function. The distortion correction matrix includes radial distortion coefficients and tangential distortion coefficients, which describe the degree of bending of light by the camera lens. By multiplying the intrinsic matrix and the distortion correction matrix, we can eliminate lens distortion and obtain a more accurate image.

[0064] This process may require some complex mathematical calculations, but there are many ready-made computer vision libraries that can help us complete these calculations. For example, OpenCV is a widely used computer vision library that provides many tools and functions for processing images and cameras.

[0065] Step S204: Obtain the point cloud map corresponding to the original image collected by the lidar.

[0066] Step S206: Fit the point cloud map with the undistorted image to obtain a fitted image.

[0067] Step S208: Select multiple target points in the fitted image.

[0068] Step S210: Obtain the two-dimensional coordinates of multiple target points in the fitted image and the three-dimensional coordinates in the point cloud map.

[0069] Step S212: Determine the fusion matrix based on the two-dimensional coordinates and the three-dimensional coordinates.

[0070] The above steps are similar to those in Embodiment 1 and will not be elaborated here again.

[0071] Next, we will introduce in detail the process of determining the fusion matrix based on the two-dimensional coordinates and the three-dimensional coordinates through steps S214 to S224.

[0072] Step S214: Take the fusion matrix determined based on the two-dimensional coordinates and the three-dimensional coordinates as the basic fusion matrix.

[0073] Step S216: Select a detection point in the fitted image and determine the first two-dimensional coordinates of the detection point in the fitted image and the first three-dimensional coordinates in the point cloud map.

[0074] Step S218: Determine the second two-dimensional coordinates based on the first three-dimensional coordinates and the basic fusion matrix.

[0075] Step S220: Calculate the error between the first two-dimensional coordinates and the second two-dimensional coordinates.

[0076] Step S222: If the error is less than the preset error threshold, determine the basic fusion matrix as the fusion matrix.

[0077] Step S224: If the error is greater than or equal to the preset error threshold, input the internal parameter matrix and the undistortion matrix into the preset parameter adjustment model and output the fusion matrix.

[0078] First, we need to obtain the two-dimensional coordinates of multiple target points in the fitted image and the three-dimensional coordinates in the point cloud map. This process is usually completed through feature extraction and matching. Feature extraction is the process of converting image or point cloud data into feature vectors that can be used for comparison, while feature matching is the process of comparing two feature vectors to find the most similar pair. Through feature extraction and matching, we can obtain the two-dimensional and three-dimensional coordinates of the target points. Then, we need to determine the fusion matrix based on the two-dimensional and three-dimensional coordinates. The fusion matrix is used to fuse the two-dimensional and three-dimensional coordinates together, and it is usually a 4x4 matrix. By taking the two-dimensional and three-dimensional coordinates as the two inputs of the fusion matrix respectively, we can fuse them together to obtain a new coordinate system. This new coordinate system contains both the information of the two-dimensional coordinates and the information of the three-dimensional coordinates, and can be used for subsequent data processing and analysis.

[0079] Specifically, we can determine the fusion matrix through the following steps:

[0080] First, we need to take the two-dimensional and three-dimensional coordinates as the two inputs of the fusion matrix respectively. This usually requires performing certain transformations on the two-dimensional and three-dimensional coordinates to make them meet the requirements of the fusion matrix. For example, we may need to perform operations such as translation and rotation on the two-dimensional coordinates to align them with the three-dimensional coordinates; or we may need to perform operations such as projection and scaling on the three-dimensional coordinates to align them with the two-dimensional coordinates.

[0081] Then, we need to calculate each element of the fusion matrix. This usually needs to be completed through a certain optimization algorithm. For example, we can solve each element of the fusion matrix through the least squares method; or we can optimize each element of the fusion matrix through the gradient descent method; or we can search for the optimal fusion matrix through the genetic algorithm.

[0082] Finally, we need to verify the correctness of the fusion matrix. This usually needs to be completed through a certain test method. For example, we can verify the correctness of the fusion matrix by comparing the errors between the target points in the fitted image and the target points in the point cloud map; or we can verify the correctness of the fusion matrix by comparing the differences between the two-dimensional and three-dimensional coordinates of the target points in the fitted image; or we can verify the correctness of the fusion matrix by comparing the change trends of the two-dimensional and three-dimensional coordinates of the target points in the fitted image.

[0083] Generally speaking, through the above steps, we can determine the fusion matrix based on two-dimensional coordinates and three-dimensional coordinates. This fusion matrix can be used for subsequent data processing and analysis. For example, we can use this fusion matrix to fuse a two-dimensional image and a three-dimensional point cloud; or we can use this fusion matrix to register a two-dimensional image and a three-dimensional point cloud; or we can use this fusion matrix to transform a two-dimensional image and a three-dimensional point cloud, etc.

[0084] Furthermore, we need to select detection points in the fitted image and determine the first two-dimensional coordinates of the detection points in the fitted image and the first three-dimensional coordinates in the point cloud map. These detection points can be specific objects in the image (such as vehicles, pedestrians, etc.) or specific regions in the image (such as roads, buildings, etc.). By selecting detection points, we can segment the image into multiple regions, with each region corresponding to a detection point.

[0085] Then, we can determine the second two-dimensional coordinates based on the first three-dimensional coordinates and the basic fusion matrix. This can be done by taking the first three-dimensional coordinates as an input to the basic fusion matrix. By taking the first three-dimensional coordinates as an input to the basic fusion matrix, we can map it to a new two-dimensional space to obtain the new two-dimensional coordinates.

[0086] Next, we need to calculate the error between the first two-dimensional coordinates and the second two-dimensional coordinates. This can be done by calculating the Euclidean distance between the two coordinates. If the error is less than the preset error threshold, we determine the basic fusion matrix as the fusion matrix. If the error is greater than or equal to the preset error threshold, we input the intrinsic matrix and the de-distortion matrix into the preset parameter tuning model to output the fusion matrix.

[0087] The parameter tuning model is a model that can automatically adjust parameters to optimize the model performance. In this example, we can use a common parameter tuning model - Grid Search. Grid Search is an exhaustive search method that tries all possible parameter combinations and then selects the parameter combination that makes the model performance optimal.

[0088] Generally speaking, through the above steps, we can effectively fuse the original image collected by the camera and the point cloud map collected by the lidar together, providing richer and more accurate information for subsequent processing and applications. Although this process involves knowledge and technologies in multiple fields, through learning and practice, we can master this process and use it to improve our camera system and applications.

[0089] In addition, this process can be further extended and optimized. For example, we can use more complex fusion algorithms, such as the ICP (Iterative Closest Point) algorithm or the NDT (Normal Distributions Transform) algorithm, to improve the accuracy and stability of the fusion. We can also use more complex parameter tuning models, such as Bayesian Optimization or Genetic Algorithm, to optimize the parameters of the fusion matrix.

[0090] In addition, we can also consider other factors, such as lighting conditions, the shape and texture of objects, the movement of the camera, etc., to further improve the accuracy and stability of the fusion. For example, we can use a lighting compensation algorithm to eliminate the influence of lighting changes; we can use a shape matching algorithm to handle objects with complex shapes; we can use a motion estimation algorithm to handle the movement of the camera.

[0091] Generally speaking, through continuous learning and practice, we can continuously improve and perfect this process to make it more accurate, stable and efficient. This will provide us with more possibilities and opportunities to explore and utilize the potential of computer vision and robotics technologies.

[0092] Furthermore, in some preferred embodiments of the present invention, the method further includes: fusing the data collected by the lidar with the data collected by the camera based on the fusion matrix.

[0093] Specifically, we need to obtain a fusion matrix determined based on two-dimensional coordinates and three-dimensional coordinates. This fusion matrix is a matrix that describes how to transform two-dimensional coordinates and three-dimensional coordinates, and it is usually a 4x4 matrix. We can determine this fusion matrix through various methods, such as fitting by the least squares method or optimizing a specific objective function.

[0094] Next, we need to register the data collected by the lidar and the data collected by the camera. This can be accomplished by using the ICP (Iterative Closest Point) algorithm or other similar algorithms. The ICP algorithm is an iterative method that can match one point cloud with another point cloud to obtain their optimal fit. Through registration, we can align the data collected by the lidar and the data collected by the camera to a common coordinate system.

[0095] Then, we need to convert the registered LiDAR data and camera data into point cloud format. This can be done by using PCL (Point Cloud Library) or other similar libraries. PCL is a widely used point cloud processing library that provides many algorithms and tools for processing point cloud data.

[0096] Next, we need to project the point cloud data onto the image plane. This can be accomplished by using the camera's intrinsic matrix. The intrinsic matrix is a matrix that describes how the camera maps the three-dimensional world onto the two-dimensional image, and it usually includes information such as focal length and principal point coordinates. By projecting the point cloud data onto the image plane, we can represent the position and shape of the point cloud in the image.

[0097] Finally, we need to fuse the projected point cloud data with the image data. This can be achieved by using a fusion matrix. The fusion matrix is a matrix that describes how to transform two-dimensional coordinates and three-dimensional coordinates, and it can help us map the point cloud data into the image space, thus realizing the fusion of point cloud data and image data.

[0098] Generally speaking, through the above steps, we can effectively fuse the data collected by the LiDAR and the data collected by the camera, providing richer and more accurate information for subsequent processing and applications. Although this process involves knowledge and technologies in multiple fields, through learning and practice, we can master this process and use it to improve our robot system and applications.

[0099] Embodiment III

[0100] See Figure 3 The flowchart of another matching method between a LiDAR and a camera provided by the embodiment of the present invention shown in Figure 4 and the schematic diagram of an intrinsic parameter automatic calibration module 1000 provided by the embodiment of the present invention shown in

[0101] 1. First, collect the calibration board image data. Let the calibration board cover as much of the camera screen as possible, and take images of the calibration board at the four edges (top, bottom, left, and right) and the center position of the camera screen. At each position, tilt or rotate the calibration board by a certain angle in the four directions (up, down, left, and right) and then take pictures.

[0102] 2. Subsequently, use the Zhang Zhengyou calibration method [1] to perform intrinsic parameter calibration on the calibration board images collected in the first step, obtaining the corresponding camera intrinsic matrix (focal lengths fx, fy, optical center coordinates cx, cy) and distortion coefficients k1, k2, p1, p2, k3.

[0103] 3. Then, the distorted image in the original scene is corrected for distortion using the calibrated camera intrinsic matrix and distortion coefficients to obtain a normal undistorted image.

[0104] 4. Subsequently, the undistorted image and the 3D point cloud collected by the lidar are visualized simultaneously (the image and the 3D point cloud are scans of the same scene at the same moment), and then pixel points and 3D points are selected manually (at least 4 pairs of corresponding points need to be selected) to obtain the pixel coordinates and 3D point coordinates at the same position of the image and the 3D point cloud. Since the corresponding points are selected by visual observation during the manual point selection process, compared with the calibration method based on feature point matching, the negative impacts caused by feature point detection errors and mismatches are excluded, and the corresponding points selected manually are more accurate and reliable. And since only a small number of corresponding points (4 pairs or more) need to be selected during the point selection process, the entire point selection process is very fast and efficient.

[0105] 5. After obtaining the image pixel points and the corresponding 3D point coordinates, these two types of coordinates are used as the input of the PnP algorithm, and then the PnP algorithm is used to solve the 3D->2D coordinate transformation (i.e., the extrinsic parameters from the lidar coordinate system to the camera coordinate system).

[0106] 6. Subsequently, the 3D point cloud is projected onto the undistorted 2D image using the extrinsic parameters solved by the PnP algorithm, and the quality of the extrinsic parameter calibration result is judged by observing the projection error. If the current extrinsic parameters meet the calibration accuracy requirements, these extrinsic parameters are directly used as the final result. If the calibration accuracy requirements have not been met yet, the extrinsic parameters solved by the PnP algorithm are used as the initial values to provide to the OpenCalib[2] algorithm for automatic extrinsic parameter calibration.

[0107] The embodiment of the present invention provides another matching method for lidar and camera, which combines the internal parameters of the camera involved in the external parameter calibration from lidar to camera into a set of calibration systems. At the same time, it visualizes 2D images and 3D point clouds, and based on this, an algorithm for point selection correspondence is carried out. It is an external parameter calibration method that uses the PnP algorithm to solve the coordinate transformation relationship between 3D-2D corresponding points. The internal parameter calibration of the camera is connected to the external parameter calibration from lidar to camera through the algorithm, constructing a complete internal and external parameter calibration system, reducing human input, and greatly improving the calibration efficiency and the reliability of the calibration results. By manually observing and selecting the corresponding points between the image and the point cloud, compared with the calibration method of feature point matching, the errors caused by the uncertainty and mis-matching of feature point detection are excluded. The technical solution of the present invention is more excellent in terms of the reliability and accuracy of the calibration results. Compared with the external parameter calibration algorithm that requires a special calibration board, the technical solution of the present invention has a wider applicable scenario and lower cost. There is no need to make a specific calibration board, and only one picture and point cloud need to be collected by the camera and lidar respectively at the same time to start the calibration. And during the calibration process, it is not required that there are objects with obvious edge features such as a calibration board in the image and the point cloud. When selecting points, only the common part of the image and the point cloud needs to be observed for point selection correspondence to obtain a more accurate calibration result. This solution has stronger applicability to different scenarios, and the entire calibration process only requires clicking the mouse, with a very simple operation method, a faster and more efficient calibration process. Even when relying on manual point selection, if the correspondence between the selected pixel points and 3D points is not very accurate, the external parameter results calibrated by this solution can be used as approximate initial values, and then other calibration methods such as OpenCalib can be used to solve the external parameters to obtain a reliable calibration result.

[0108] Embodiment 4

[0109] Based on the above embodiments, the embodiment of the present invention provides another matching device for lidar and camera. Refer to Figure 5 the structural schematic diagram of a matching device for lidar and camera provided by the embodiment of the present invention as shown. The device includes:

[0110] A basic image determination module 310, configured to obtain the original image captured by the camera, and convert the original image into a de-distorted image based on a preset internal parameter matrix and a preset de-distortion matrix;

[0111] A point cloud image determination module 320, configured to obtain the point cloud map corresponding to the original image collected by the lidar;

[0112] An image fitting module 330, configured to fit the point cloud map and the de-distorted image to obtain a fitted image;

[0113] A target point selection module 340, configured to select multiple target points in the fitted image;

[0114] A coordinate determination module 350, configured to obtain the two-dimensional coordinates of multiple target points in the fitting image and the three-dimensional coordinates in the point cloud map;

[0115] A fusion matrix determination module 360, configured to determine a fusion matrix based on the two-dimensional coordinates and the three-dimensional coordinates.

[0116] Furthermore, in some preferred embodiments of the present invention, the device further includes: an internal parameter matrix determination module, configured to obtain multiple calibration images collected by the camera; wherein, the multiple calibration images are images of the same calibration board taken from different angles; obtain the coordinates of the same feature point in the multiple calibration images; and determine the internal parameter matrix of the camera based on the coordinates of the multiple same feature points in the multiple calibration images.

[0117] Furthermore, in some preferred embodiments of the present invention, the device further includes: a distortion removal matrix determination module, configured to determine a distortion removal matrix based on the multiple calibration images and the internal parameter matrix through a preset function; wherein, the distortion removal matrix includes radial distortion coefficients and tangential distortion coefficients.

[0118] Furthermore, in some preferred embodiments of the present invention, the device further includes: a fusion matrix determination module, configured to use the fusion matrix determined based on the two-dimensional coordinates and the three-dimensional coordinates as the basic fusion matrix; select a detection point in the fitting image, and determine the first two-dimensional coordinates of the detection point in the fitting image and the first three-dimensional coordinates in the point cloud map; determine the second two-dimensional coordinates based on the first three-dimensional coordinates and the basic fusion matrix; calculate the error between the first two-dimensional coordinates and the second two-dimensional coordinates; if the error is less than a preset error threshold, determine the basic fusion matrix as the fusion matrix.

[0119] Furthermore, in some preferred embodiments of the present invention, the device further includes: a fusion matrix adjustment module, configured to, if the error is greater than or equal to the preset error threshold, input the internal parameter matrix and the distortion removal matrix into a preset parameter adjustment model to output a fusion matrix.

[0120] Furthermore, in some preferred embodiments of the present invention, the device further includes: a data fusion module, configured to fuse the data collected by the lidar and the data collected by the camera based on the fusion matrix.

[0121] Those skilled in the art can clearly understand that for the convenience and brevity of description, the specific working process of the above-described lidar and camera matching device can refer to the corresponding process in the embodiment of the lidar and camera matching method described above, and will not be elaborated here.

[0122] Embodiment 5

[0123] The embodiment of the present invention also provides an electronic device for running the lidar and camera matching method; see Figure 6Schematic structural diagram of an electronic device provided by an embodiment of the present invention. The electronic device includes a memory 400 and a processor 401. Among them, the memory 400 is used to store one or more computer instructions, and the one or more computer instructions are executed by the processor 401 to implement the above-mentioned matching method between the lidar and the camera.

[0124] Furthermore, Figure 6 The shown electronic device further includes a bus 402 and a communication interface 403. The processor 401, the communication interface 403, and the memory 400 are connected through the bus 402.

[0125] Among them, the memory 400 may include a high-speed random access memory (RAM, Random Access Memory), and may also include a non-volatile memory, such as at least one disk memory. Through at least one communication interface 403 (which can be wired or wireless), a communication connection is realized between this system network element and at least one other network element. The Internet, wide area network, local area network, metropolitan area network, etc. can be used. The bus 402 can be an ISA bus, a PCI bus, an EISA bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. For the sake of representation, Figure 6 only a bidirectional arrow is used in the figure, but it does not mean that there is only one bus or one type of bus.

[0126] The processor 401 may be an integrated circuit chip with the ability to process signals. In the implementation process, each step of the above method can be completed by the integrated logic circuit of the hardware in the processor 401 or the instructions in the form of software. The above-mentioned processor 401 may be a general-purpose processor, including a central processing unit (CPU for short), a network processor (NP for short), etc.; it may also be a digital signal processor (DSP for short), an application specific integrated circuit (ASIC for short), a field-programmable gate array (FPGA for short), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components. It can implement or execute each method, step, and logic block diagram disclosed in the embodiments of the present invention. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc. The steps of the method disclosed in combination with the embodiments of the present invention can be directly embodied as being executed and completed by the hardware decoding processor, or executed and completed by the combination of the hardware and software modules in the decoding processor. The software module may be located in a mature storage medium in the art such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, or an electrically erasable programmable memory, a register, etc. This storage medium is located in the memory 400, and the processor 401 reads the information in the memory 400 and combines its hardware to complete the steps of the method in the foregoing embodiments.

[0127] The embodiments of the present invention also provide a storage medium storing computer-executable instructions, which, when called and executed by a processor, cause the processor to implement the above-mentioned service recommendation method. For the specific implementation, reference can be made to the method embodiments, and details are not described herein again.

[0128] The computer program product of the method, device, and electronic device for matching a lidar and a camera provided by the embodiments of the present invention includes a storage medium storing program code, and the instructions included in the program code can be used to execute the methods in the foregoing method embodiments. For the specific implementation, reference can be made to the method embodiments, and details are not described herein again.

[0129] Those skilled in the art can clearly understand that for the convenience and conciseness of description, the specific working processes of the above-described system and / or device can refer to the corresponding processes in the foregoing method embodiments, and details are not described herein again.

[0130] In addition, in the description of the embodiments of the present invention, unless otherwise clearly defined and limited, the terms "installation", "connection", and "coupling" should be understood in a broad sense. For example, it may be a fixed connection, a detachable connection, or an integral connection; it may be a mechanical connection or an electrical connection; it may be directly connected or indirectly connected through an intermediate medium, and it may be the communication inside two components. For those of ordinary skill in the art, the specific meanings of the above terms in the present invention can be understood according to specific situations.

[0131] If the above functions are realized in the form of software function units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or a part of this technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The foregoing storage medium includes: various media such as USB flash drives, mobile hard disks, read-only memories (ROMs), random access memories (RAMs), magnetic disks, or optical discs that can store program codes.

[0132] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions described in the foregoing embodiments, or perform equivalent replacements on some or all of the technical features; and these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the various embodiments of the present invention.

Claims

1. A matching method for a lidar and a camera, characterized in that Including: Obtain the original image captured by the camera, and convert the original image into a de-distorted image based on a preset internal parameter matrix and a preset de-distortion matrix; Obtain the point cloud map corresponding to the original image collected by the lidar; Fit the point cloud map and the de-distorted image to obtain a fitted image; Select multiple target points in the fitted image; Obtain the two-dimensional coordinates of multiple target points in the fitted image and the three-dimensional coordinates in the point cloud map; Determine the fusion matrix based on the two-dimensional coordinates and the three-dimensional coordinates.

2. The matching method of the lidar and the camera according to claim 1, wherein Determine the internal parameter matrix through the following steps: Obtain multiple calibration images collected by the camera; among them, multiple calibration images are images of the same calibration board taken at different angles; Obtain the coordinates of the same feature point in multiple calibration images; Determine the internal parameter matrix of the camera based on the coordinates of multiple same feature points in multiple calibration images.

3. The matching method of the lidar and the camera according to claim 2, characterized in that Determine the de-distortion matrix through the following steps: Determine the de-distortion matrix based on multiple calibration images and the internal parameter matrix through a preset function; where the de-distortion matrix includes radial distortion coefficients and tangential distortion coefficients.

4. The matching method of the lidar and the camera according to claim 1, wherein After the step of determining the fusion matrix based on the two-dimensional coordinates and the three-dimensional coordinates, the method further includes: Take the fusion matrix determined based on the two-dimensional coordinates and the three-dimensional coordinates as the basic fusion matrix; Select a detection point in the fitted image, and determine the first two-dimensional coordinates of the detection point in the fitted image and the first three-dimensional coordinates in the point cloud map; Determine the second two-dimensional coordinates based on the first three-dimensional coordinates and the basic fusion matrix; Calculate the error between the first two-dimensional coordinates and the second two-dimensional coordinates; If the error is less than a preset error threshold, determine the basic fusion matrix as the fusion matrix.

5. The matching method of the lidar and the camera according to claim 4, wherein After the step of calculating the error between the first two-dimensional coordinates and the second two-dimensional coordinates, the method further includes: If the error is greater than or equal to the preset error threshold, input the internal parameter matrix and the de-distortion matrix into a preset parameter adjustment model, and output the fusion matrix.

6. The matching method of the lidar and the camera according to claim 1, characterized in that, Also including: Fuse the data collected by the lidar and the data collected by the camera based on the fusion matrix.

7. A matching device for a lidar and a camera, characterized in that, Including: A basic image determination module, configured to obtain the original image captured by the camera, and convert the original image into a de-distorted image based on a preset internal parameter matrix and a preset de-distortion matrix; A point cloud image determination module, configured to obtain the point cloud map corresponding to the original image collected by the lidar; An image fitting module, configured to fit the point cloud map and the de-distorted image to obtain a fitted image; A target point selection module, configured to select multiple target points in the fitted image; A coordinate determination module, configured to obtain the two-dimensional coordinates of multiple target points in the fitted image and the three-dimensional coordinates in the point cloud map; A fusion matrix determination module, configured to determine the fusion matrix based on the two-dimensional coordinates and the three-dimensional coordinates.

8. The matching device for a lidar and a camera according to claim 7, characterized in that, Also including: An internal parameter matrix determination module, configured to obtain multiple calibration images collected by the camera; among them, multiple calibration images are images of the same calibration board taken at different angles; Obtain the coordinates of the same feature point in the multiple calibration images; determine the internal parameter matrix of the camera based on the coordinates of multiple such same feature points in the multiple calibration images.

9. An electronic device, characterized in that, It includes a processor and a memory. The memory stores computer-executable instructions that can be executed by the processor. The processor executes the computer-executable instructions to implement the matching method of the lidar and the camera according to any one of claims 1 to 6 above.

10. A storage medium, characterized in that, The storage medium stores computer-executable instructions. When the computer-executable instructions are called and executed by the processor, the computer-executable instructions prompt the processor to implement the matching method of the lidar and the camera according to any one of claims 1 to 6.

Images