A laser radar-camera external parameter self-calibration method and device

Abstract

The application discloses a laser radar-camera external parameter self-calibration method and device, and the method comprises the following steps: obtaining scene pose data and a plurality of fisheye images with different visual angles through a data acquisition device; obtaining a mesh model according to the scene pose data; respectively correcting the plurality of fisheye images to obtain respective corresponding corrected images; performing visibility analysis on each vertex in the mesh model according to the mesh model and the corrected images to obtain a vertex visible image set; wherein the vertex visible image set stores a visible relationship between each vertex and the corrected image, and the vertex is visible in the corrected image which has the visible relationship with the vertex; and optimizing an external parameter matrix corresponding to each corrected image according to the vertex visible image set to obtain an optimization result, and adjusting the camera external parameter corresponding to the corrected image by using the optimization result. The application improves the accuracy of picture splicing in a calibration scene without markers.

Description

Technical Field

[0001] This application relates to the field of camera calibration, specifically to a method and apparatus for self-calibrating the extrinsic parameters of a lidar-camera system. Background Technology

[0002] In image measurement and machine vision applications, to determine the relationship between the three-dimensional geometric position of a point on the surface of a spatial object and its position in a camera image, a geometric model of camera imaging must be established. The parameters of this geometric model are the camera parameters, and the process of solving for these camera parameters is called camera calibration. The purpose of camera calibration is to obtain the camera's intrinsic and extrinsic parameters and use these parameters to correct the camera image.

[0003] In related technologies, camera calibration often requires the use of markers locally. By establishing a correspondence between points with known coordinates on the markers and points in the camera image, the intrinsic and extrinsic parameters of the camera model are obtained, thus achieving local calibration. However, for data acquisition needs in remote scenarios, the data acquisition equipment needs to be locally calibrated using markers, then disassembled into multiple components and transported to the destination. Since the extrinsic parameters of the reassembled data acquisition equipment are not the same as those obtained during local calibration, and marker-based calibration is difficult to implement remotely, the image stitching effect obtained after reassembly at the destination is poor. Summary of the Invention

[0004] In view of this, the first aspect of this application provides a self-calibration method for lidar-camera extrinsic parameters, the method comprising:

[0005] Scene pose data and multiple fisheye images from different perspectives are obtained through data acquisition equipment;

[0006] A mesh model is obtained based on the scene pose data;

[0007] Distortion correction is performed on the multiple fisheye images respectively to obtain the corrected images corresponding to each of the multiple fisheye images;

[0008] Based on the Mesh model and the corrected image, a visibility analysis is performed on each vertex in the Mesh model to obtain a set of vertex visible images; wherein, the set of vertex visible images stores the visibility relationship between each vertex and the corrected image, and in the corrected image with the visibility relationship with the vertex, the vertex is visible;

[0009] Based on the set of vertex visible images, the extrinsic parameter matrix corresponding to each of the corrected images is optimized to obtain the optimization result, and the camera extrinsic parameters corresponding to the corrected image are adjusted using the optimization result.

[0010] Preferably, the step of iteratively optimizing the extrinsic parameter matrix corresponding to each corrected image based on the vertex visible image set to obtain an optimization result, and using the optimization result to adjust the camera extrinsic parameters corresponding to the corrected image, includes:

[0011] For each of the corrected images, the visible vertices in the corrected image are determined according to the set of visible vertex images; the visible vertices are the vertices in the Mesh model that have the visibility relationship with the corrected image.

[0012] For each visible vertex, based on the set of visible images of the vertex, the corrected image that has the visible relationship with the visible vertex is determined as the reference corrected image corresponding to the corrected image;

[0013] Iterative optimization is performed on the reference extrinsic matrix corresponding to each of the reference corrected images, and the loss function corresponding to the reference extrinsic matrix is ​​calculated.

[0014] The reference extrinsic matrix corresponding to the loss function that meets the optimization conditions is used as the optimization result, and the camera extrinsic parameters corresponding to the corrected image are adjusted using the optimization result.

[0015] Preferably, the iterative optimization of the reference extrinsic parameter matrix corresponding to each of the reference corrected images to determine the loss function corresponding to the reference extrinsic parameter matrix includes:

[0016] For each visible vertex, the k-th grayscale error of the visible vertex is determined based on the grayscale value of the visible vertex in the k-th reference corrected image and the standard value of the visible vertex; wherein the k-th grayscale error corresponds to the k-th reference corrected image, and k is an integer greater than or equal to 1;

[0017] Based on the kth grayscale error of all visible vertices in the corrected image, the kth loss function corresponding to the kth reference corrected image is obtained.

[0018] Preferably, the step of performing visibility analysis on each vertex in the mesh model based on the mesh model and the corrected image to obtain a set of vertex visible images includes:

[0019] Based on the mesh model and the corrected image, image depth information is obtained; wherein, the image depth information includes the distance between each vertex in the mesh model and the camera that acquired the fisheye image;

[0020] Based on the image depth information, visibility analysis is performed on each vertex in the Mesh model to obtain a set of vertex visible images.

[0021] Preferably, the step of performing distortion correction on the plurality of fisheye images respectively to obtain the corrected image corresponding to each of the plurality of fisheye images includes:

[0022] Based on the local calibration results, a first transformation matrix is ​​obtained between the inertial measurement unit (IMU) coordinate system and the camera coordinate system; wherein, the camera coordinate system is the coordinate system corresponding to the fisheye image;

[0023] Based on the positions of the vertices in the Mesh model, a second transformation matrix between the world coordinate system and the IMU coordinate system is obtained; wherein, the world coordinate system is the coordinate system corresponding to the Mesh model;

[0024] Based on the first transformation matrix and the second transformation matrix, a third transformation matrix between the world coordinate system and the camera coordinate system is obtained;

[0025] The distortion is corrected for each of the multiple fisheye images according to the third transformation matrix to obtain the corrected image corresponding to each of the multiple fisheye images.

[0026] Preferably, the step of performing visibility analysis on each vertex in the mesh model based on the mesh model and the corrected image to obtain a set of vertex visible images includes:

[0027] Based on the mesh model and the corrected image, image depth information is obtained; wherein, the image depth information includes the distance between each vertex in the mesh model and the camera that acquired the fisheye image;

[0028] Generate a depth boundary map based on the image depth information;

[0029] Based on the third transformation matrix, the vertex coordinates in the Mesh model are transformed into vertex camera coordinates in the camera coordinate system;

[0030] Based on the camera intrinsic parameters and the pinhole model, obtain the vertex pixel coordinates of the vertex camera coordinates in the pixel coordinate system on the image plane;

[0031] Based on the depth boundary map and the vertex pixel coordinates, a visibility analysis is performed on the i-th vertex and the corrected image to obtain the set of visible images corresponding to the i-th vertex; wherein, the set of visible images corresponding to the i-th vertex includes all the corrected images visible to the i-th vertex, and i is an integer greater than or equal to 1;

[0032] A vertex visible image set is generated based on the visible image set corresponding to each vertex in the Mesh model.

[0033] Preferably, generating a depth boundary map based on the image depth information includes:

[0034] The image depth information is plotted into a depth map;

[0035] After applying Gaussian filtering to the depth map, the gradient values ​​of the pixels in the depth map are calculated.

[0036] For each pixel in the depth map, if the gradient value of the pixel is greater than the gradient threshold, then the pixel value of the pixel is set to a first pixel value; if the gradient value of the pixel is not greater than the gradient threshold, then the pixel value of the pixel is set to a second pixel value; the first pixel value is not equal to the second pixel value.

[0037] A depth boundary map is drawn based on the pixel values ​​set for each pixel in the depth map.

[0038] Preferably, the method further includes:

[0039] The corrected image is upsampled using a linear interpolation method to obtain an n-layer image corresponding to the corrected image; wherein, in the n-layer image, the first layer image is the corrected image, the (j+1)th layer image is obtained by upsampling the j-th layer image, n is an integer greater than 1, and j is an integer greater than or equal to 1.

[0040] The step of optimizing the extrinsic parameter matrix corresponding to each corrected image based on the vertex visible image set, obtaining the optimization result, and using the optimization result to adjust the camera extrinsic parameters corresponding to the corrected image includes:

[0041] Based on the vertex visible image set, the n-layer images corresponding to each of the corrected images are sorted in descending order, and the extrinsic parameter matrices corresponding to each of the n-layer images are optimized to obtain the layered optimization results corresponding to each of the n-layer images.

[0042] The hierarchical optimization result that meets the optimization conditions is used as the optimization result of the corrected image, and the camera extrinsic parameters corresponding to the corrected image are adjusted using the optimization result.

[0043] Preferably, after optimizing the camera extrinsic parameters corresponding to the corrected image based on the optimization matrix, the method further includes:

[0044] By reconstructing textures, the texture map is rendered onto the Mesh model, and the optimized image stitching effect is visualized and output.

[0045] A second aspect of this application provides a self-calibration device for the extrinsic parameters of a lidar-camera system, the device comprising:

[0046] The data acquisition unit is used to: acquire scene pose data and multiple fisheye images from different perspectives through a data acquisition device; wherein the scene pose data includes at least: gyroscope data, accelerometer data, and point cloud data;

[0047] The data processing unit is used to: obtain a mesh model based on the scene pose data;

[0048] An image correction unit is used to: perform distortion correction on the plurality of fisheye images respectively, and obtain a corrected image corresponding to each of the plurality of fisheye images;

[0049] An image analysis unit is configured to: perform visibility analysis on each vertex in the Mesh model based on the Mesh model and the corrected image to obtain a set of vertex visible images; wherein the set of vertex visible images stores the visibility relationship between each vertex and the corrected image, and the vertex is visible in the corrected image where the visibility relationship exists with the vertex;

[0050] The extrinsic parameter optimization unit is used to: optimize the extrinsic parameter matrix corresponding to each of the corrected images based on the vertex visible image set, obtain the optimization result, and use the optimization result to adjust the camera extrinsic parameters corresponding to the corrected image.

[0051] As can be seen from the above technical solutions, this application has the following advantages: This application obtains scene pose data and multiple fisheye images from different perspectives through a data acquisition device; wherein, the scene pose data includes at least: gyroscope data, accelerometer data, and point cloud data; a mesh model is obtained based on the scene pose data; camera calibration is performed using the scene pose data and fisheye images obtained through the data acquisition device, reducing the operational difficulty of camera calibration; distortion correction is performed on multiple fisheye images respectively to obtain corrected images corresponding to each of the multiple fisheye images; based on the mesh model and the corrected images, visibility analysis is performed on each vertex in the mesh model to obtain a set of vertex visible images; wherein The vertex visible image set stores the visibility relationship between each vertex and the corrected image. In the corrected image with a visible relationship to a vertex, the vertex is visible. Based on the vertex visible image set, the extrinsic parameter matrix corresponding to each corrected image is optimized to obtain the optimization result, and the optimization result is used to adjust the camera extrinsic parameters corresponding to the corrected image. The two-dimensional image and the three-dimensional space are linked through image depth information, and the visibility of the vertex coordinates in the Mesh model and the corrected image is analyzed to achieve self-calibration of camera extrinsic parameters without markers. This simplifies the camera extrinsic parameter calibration process, improves the convenience of camera extrinsic parameter calibration, and improves the accuracy of image stitching in calibration scenarios without markers. Attached Figure Description

[0052] Figure 1 A flowchart of a LiDAR-camera extrinsic parameter self-calibration method provided in this application embodiment;

[0053] Figure 2 A flowchart of a LiDAR-camera extrinsic parameter self-calibration method provided for an embodiment of the scenario in this application;

[0054] Figure 3 A schematic diagram of the data preprocessing process provided for an embodiment of the scenario in this application;

[0055] Figure 4 A schematic diagram of the extrinsic parameter optimization preprocessing flow provided for the scenario embodiments of this application;

[0056] Figure 5 A schematic diagram of an image pyramid provided for an embodiment of the scenario in this application;

[0057] Figure 6 A schematic diagram illustrating the process of acquiring image depth information provided in this application scenario embodiment;

[0058] Figure 7 A schematic diagram of the depth map boundary segmentation process provided in the scenario embodiment of this application;

[0059] Figure 8 A schematic diagram of the visibility analysis process of vertices and images provided for an embodiment of the scenario in this application;

[0060] Figure 9 Comparison images of image stitching effects provided for embodiments of this application;

[0061] Figure 10 This is a schematic diagram of a lidar-camera extrinsic parameter self-calibration device provided in an embodiment of this application. Detailed Implementation

[0062] Embodiments of this application will now be described in more detail with reference to the accompanying drawings. While some embodiments of this application are shown in the drawings, it should be understood that this application can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this application. It should be understood that the drawings and embodiments of this application are for illustrative purposes only and are not intended to limit the scope of protection of this application.

[0063] A lidar-camera system is a data acquisition device equipped with lidar and a camera. Lidar is a radar system that uses emitted laser beams to detect the position, velocity, and other characteristics of a target. Its working principle involves emitting a detection signal towards the target, then comparing the received signal reflected from the target with the emitted signal. After appropriate processing, information about the target can be obtained, such as distance, azimuth, altitude, velocity, attitude, and even shape parameters, thereby enabling target detection, tracking, and identification. The camera is a panoramic camera used to acquire fisheye images from different perspectives. Lidar, as a sensor, helps the camera achieve positioning during the imaging process, constructing high-precision planar maps, and, when used in conjunction with other optical sensors, generating virtual images.

[0064] See Figure 1 As shown in the embodiments of this application, a self-calibration method for the extrinsic parameters of a lidar-camera system includes the following steps:

[0065] Step 101: Obtain scene pose data and multiple fisheye images from different perspectives using a data acquisition device.

[0066] The scene pose data includes at least: gyroscope data, accelerometer data, and point cloud data. The data acquisition equipment includes an Inertial Measurement Unit (IMU), a LiDAR, and a panoramic camera. The IMU module obtains gyroscope and accelerometer data to measure the angular velocity and acceleration of an object in three-dimensional space, and uses this to calculate the object's pose. The LiDAR scans the spatial scene to obtain point cloud data. The panoramic camera obtains multiple fisheye images from different perspectives. It should be noted that the data acquisition equipment can be a LiDAR-camera combination.

[0067] Step 102: Obtain the mesh model based on the scene pose data.

[0068] In this context, "mesh" refers to the grid of a model. A 3D model is composed of polygons, which in turn are composed of triangles. Therefore, the surface of a 3D model is actually made up of multiple interconnected triangular faces. In 3D space, the set of points and edges that make up these triangles constitutes the mesh. By using scene pose data obtained from data acquisition equipment to run a Simultaneous Localization and Mapping (SLAM) process, 3D point clouds, trajectories, and maps can be obtained. The mesh model can then be obtained by reconstructing the 3D point cloud.

[0069] Step 103: Perform distortion correction on multiple fisheye images to obtain the corrected images corresponding to each fisheye image.

[0070] When correcting distortion in fisheye images, the coordinate transformation between the world coordinate system and the camera coordinate system is used to correct the fisheye image into a planar image. A fisheye image is an image captured using a fisheye lens. The image seen through a fisheye lens differs significantly from the real world. In real life, objects have regular, fixed shapes, but in images obtained through a fisheye lens, only the central object remains unchanged; other objects that should be horizontal or vertical are distorted. In the corrected planar image, the shapes of objects are consistent with the actual image, restoring the true scene as seen from the viewer's perspective.

[0071] In one possible implementation, step 103 can be implemented in the following way, specifically including:

[0072] Step 1031: Based on the local calibration results, obtain the first transformation matrix between the inertial measurement unit (IMU) coordinate system and the camera coordinate system.

[0073] The origin of the IMU coordinate system is at the origin of the gyroscope and accelerometer coordinates, and the x, y, and z axes are parallel to the corresponding axes of the gyroscope and accelerometer, respectively. The coordinates in the IMU coordinate system are used to represent the pose of the data acquisition device in the point cloud data. The camera coordinate system is the π coordinate system corresponding to the fisheye image. The coordinate transformation matrix between the IMU coordinate system and the camera coordinate system can be obtained by the local calibration result obtained by the local calibration of the lidar-camera system, which is the first transformation matrix.

[0074] It should be noted that there are deviations between the camera coordinate systems corresponding to fisheye images from different perspectives. The first transformation matrix corresponding to each fisheye image from different perspectives can be obtained based on the local calibration results.

[0075] Step 1032: Based on the position of the vertices in the Mesh model, obtain the second transformation matrix between the world coordinate system and the IMU coordinate system.

[0076] The world coordinate system is the three-dimensional coordinate system corresponding to the Mesh model; the coordinate transformation matrix between the IMU coordinate system and the world coordinate system of each vertex is obtained by the position of the vertex in the Mesh model, which is the second transformation matrix.

[0077] Step 1033: Based on the first and second transformation matrices, obtain the third transformation matrix between the world coordinate system and the camera coordinate system.

[0078] Based on the coordinate transformation matrix between the camera coordinate system and the IMU coordinate system, and the coordinate transformation matrix between the IMU coordinate system and the world coordinate system, the coordinate transformation matrix between the camera coordinate system and the world coordinate system is obtained, which is the third transformation matrix.

[0079] Step 1034: Perform distortion correction on multiple fisheye images according to the third transformation matrix to obtain the corrected images corresponding to each of the multiple fisheye images.

[0080] The third transformation matrix is ​​used to transform the coordinates in the camera coordinate system to the coordinates in the world coordinate system, transforming the fisheye image into a normal image, thereby correcting the distortion of the fisheye image and obtaining the corrected images corresponding to the fisheye images from different perspectives.

[0081] By transforming between different coordinate systems, distortion correction is performed on fisheye images to obtain corrected images, thereby improving the visual effect of the images.

[0082] Step 104: Based on the Mesh model and the corrected image, perform visibility analysis on each vertex in the Mesh model to obtain a set of vertex visible images.

[0083] The vertex visible image set stores the visibility relationship between each vertex and the corrected image. In the corrected image where a vertex has a visibility relationship with another vertex, the vertex is visible. When performing visibility analysis on each vertex in the mesh model, image depth information can be obtained first from the mesh model and the corrected image. Then, based on this image depth information, visibility analysis is performed on each vertex in the mesh model, resulting in the vertex visible image set. The image depth information includes the distance between each vertex in the mesh model and the camera that captured the fisheye image, which can be used to analyze the correspondence between points in 3D space and 2D images.

[0084] Specifically, step 104 can be implemented in the following ways, including:

[0085] Step 1041: Obtain image depth information based on the Mesh model and the corrected image.

[0086] Projection calculations are performed between each vertex in the Mesh model and the corrected image. Each vertex in the Mesh model is projected onto the corresponding pixel in the corrected image, and the coordinates of the vertex in the Mesh model are mapped to the coordinates in the corrected image. The depth information corresponding to each pixel in the corrected image is obtained based on the coordinate mapping relationship and saved as image depth information.

[0087] Step 1042: Generate a depth boundary map based on the image depth information.

[0088] Among them, the depth boundary map is an image drawn based on the object boundaries extracted from the image depth information, which describes the boundaries of objects in the image.

[0089] Specifically, an embodiment of this application provides a method for generating a depth boundary map, which includes the following steps:

[0090] Step S1: Draw a depth map from the image depth information.

[0091] In the depth map, the grayscale value of each pixel is used to represent the distance between the viewpoint and the scene object; while in this embodiment, the viewpoint is the camera that captures the fisheye image, the scene object is the Mesh model, and the grayscale value is used to represent the distance between each vertex in the Mesh model and the camera that captures the fisheye image.

[0092] Step S2: After applying Gaussian filtering to the depth map, calculate the gradient values ​​of the pixels in the depth map.

[0093] The depth map is filtered, and the gradient value of each pixel in the depth map is calculated. Specifically, a two-dimensional coordinate system xoy can be established in the depth map. First-order gradients of the pixels in the depth map are solved to obtain the first-order depth gradient information in the x and y directions for each pixel. The gradient value of the pixel is then calculated based on the first-order depth gradient information in the x and y directions.

[0094] Step S3: For each pixel in the depth map, if the gradient value of the pixel is greater than the gradient threshold, then set the pixel value of the pixel to the first pixel value; if the gradient value of the pixel is not greater than the gradient threshold, then set the pixel value of the pixel to the second pixel value.

[0095] When the gradient value of a pixel is greater than the gradient threshold, the distance change rate of that pixel is relatively large, and the pixel is considered a boundary point, which is then drawn using the first pixel value. When the gradient value of a pixel is not greater than the gradient threshold, the distance change rate of that pixel is relatively small, and the pixel is considered a non-boundary point, which is then drawn using the second pixel value. The first pixel value is not equal to the second pixel value. For example, the first pixel value can be set to 255, corresponding to white, and the second pixel value can be set to 0, corresponding to black.

[0096] Step S4: Draw the depth boundary map based on the pixel values ​​set for each pixel in the depth map.

[0097] When processing a depth boundary map, an expansion operation can be performed to enlarge the boundary lines in the depth boundary map. After the boundary lines are expanded, the object boundaries become clearer and smoother, thus making the depth boundary map closer to the real scene.

[0098] The generated depth boundary map, combined with the mesh model and the corrected image, is used to analyze the visibility between each vertex in the mesh model and the corrected image, thus improving the analysis efficiency.

[0099] Step 1043: Based on the third transformation matrix, transform the vertex coordinates in the Mesh model into vertex camera coordinates in the camera coordinate system.

[0100] Based on the third transformation matrix that describes the transformation relationship between the world coordinate system and the camera coordinate system, the coordinate values ​​of vertices in the Mesh model in the world coordinate system are converted into the corresponding coordinate values ​​of the vertices in the camera coordinate system, i.e., vertex camera coordinates.

[0101] Step 1044: Based on the camera intrinsic parameters and the pinhole model, obtain the vertex pixel coordinates of the camera vertex in the pixel coordinate system on the image plane.

[0102] Camera intrinsic parameters are those related to the camera's inherent characteristics, such as focal length and pixel size. Based on the vertex camera coordinates (i.e., the coordinates of the camera's optical center in the world coordinate system), and using the pinhole camera model, these vertex camera coordinates can be transformed into their corresponding vertex pixel coordinates in the pixel coordinate system on the image plane during image capture. The pixel coordinate system is a two-dimensional Cartesian coordinate system (uov) on the image plane.

[0103] Step 1045: Based on the depth boundary map and vertex pixel coordinates, perform visibility analysis on the i-th vertex and the corrected image to obtain the set of visible images corresponding to the i-th vertex.

[0104] The set of visible images corresponding to the i-th vertex includes all corrected images visible to the i-th vertex. That is, the i-th vertex is visible in the aforementioned corrected images, and i is an integer greater than or equal to 1.

[0105] Using vertex coordinates, optical center coordinates, and the pinhole model, the distance d from each vertex to the image plane can be calculated. When performing visibility analysis on a vertex and a corrected image, if d is less than 0, or d is greater than the visibility threshold, or the vertex pixel coordinates are located on the boundary of the depth boundary map corresponding to the corrected image, or the vertex exceeds the resolution of the corrected image, the vertex and the corrected image are not visible to each other; otherwise, the vertex and the corrected image are visible to each other.

[0106] Specifically, when d is less than 0, the vertex is considered to be behind the camera capturing the corrected image, and therefore the vertex and the corrected image are not visible to each other; when d is greater than the visibility threshold, d is considered to be beyond the visible distance of the camera capturing the corrected image, and therefore the vertex and the corrected image are not visible to each other; when the vertex pixel coordinates are located at the boundary of the depth boundary map corresponding to the corrected image, the vertex and the corrected image are not visible to each other; when the vertex exceeds the resolution of the corrected image, the vertex is outside the visible range of the camera capturing the corrected image, and therefore the vertex and the corrected image are not visible to each other.

[0107] Step 1046: Generate a vertex visible image set based on the visible image set corresponding to each vertex in the Mesh model.

[0108] By performing visibility analysis on each vertex and the corrected image, we obtained the set of visible images for each vertex in the Mesh model. This set of visible images can then be used to generate a vertex visible image set containing all vertices and their visible images. Within this vertex visible image set, the visibility relationship between each vertex and the corrected image is stored; vertices with a visibility relationship are visible to each other in the corrected image.

[0109] By performing visibility analysis between vertices in the Mesh model and the corrected image, a set of vertex visible images is obtained. The Mesh model is then mapped to the corrected image so that the extrinsic parameter matrix corresponding to the corrected image can be optimized in the future.

[0110] Step 105: Based on the set of vertex visible images, optimize the extrinsic matrix corresponding to each corrected image to obtain the optimization result, and use the optimization result to adjust the camera extrinsic parameters corresponding to the corrected image.

[0111] The camera's extrinsic parameter matrix describes the camera's position in the world coordinate system. Camera extrinsic parameters are the camera's pose in the world coordinate system, such as its position and rotation direction. Unlike the constant intrinsic parameters, extrinsic parameters change with camera movement. In camera calibration, optimizing camera extrinsic parameters can improve the accuracy of the results. For example, when using multiple cameras to acquire images from different perspectives and needing to stitch these images together to obtain a complete image, optimizing camera extrinsic parameters can improve the stitching quality.

[0112] In one possible implementation, step 105 can be achieved through the following steps:

[0113] Step 1051: For each corrected image, determine the visible vertices in the corrected image based on the set of vertex visible images.

[0114] In this context, visible vertices are those vertices in the Mesh model that have a visible relationship with the corrected image. When analyzing the extrinsic parameter matrix corresponding to each corrected image, the visible vertices in the current corrected image are determined based on the set of visible images of the vertices.

[0115] Step 1052: For each visible vertex, based on the set of visible vertex images, determine the corrected image that has a visible relationship with the visible vertex, and use it as the reference corrected image corresponding to the corrected image.

[0116] Based on the set of visible vertex images, the visibility relationship between vertices and the corrected image can be obtained. For each visible vertex in the current corrected image, the visible image of each visible vertex is determined, that is, the corrected image that has a visibility relationship with the visible vertex. The aforementioned corrected image is determined as the reference corrected image used when analyzing the extrinsic parameter matrix corresponding to the current corrected image.

[0117] Step 1053: Iteratively optimize the reference extrinsic matrix corresponding to each reference corrected image and calculate the loss function corresponding to the reference extrinsic matrix.

[0118] Different corrected images correspond to different extrinsic parameter matrices, and different extrinsic parameter matrices correspond to different camera poses. Therefore, the process of analyzing the current corrected image is to select the optimal camera pose from the camera poses corresponding to the reference corrected image to adjust the camera pose corresponding to the current corrected image so that the gray value of the visible vertex in the current corrected image is closest to the true value.

[0119] Specifically, the iterative optimization of the k-th reference corrected image corresponding to the visible vertex includes the following steps:

[0120] Step M1: For each visible vertex, determine the k-th grayscale error of the visible vertex based on the grayscale value of the visible vertex in the k-th reference corrected image and the standard value of the visible vertex.

[0121] The k-th grayscale error corresponds to the k-th reference corrected image, where k is an integer greater than or equal to 1. The standard value is the average grayscale value of the visible vertex in all its corresponding corrected images. In this embodiment, it is used to represent the true grayscale value of the visible vertex. The k-th grayscale error between the visible vertex and the k-th reference corrected image is obtained by calculating the error between the grayscale value of the visible vertex in the k-th reference corrected image and the standard value of the visible vertex.

[0122] Step M2: Based on the kth grayscale error of all visible vertices in the corrected image, obtain the kth loss function corresponding to the kth reference corrected image.

[0123] For the k-th reference corrected image, the loss function corresponding to the k-th reference corrected image is obtained by summing the k-th grayscale error between the image and all vertices visible to it.

[0124] Step 1054: Use the reference extrinsic matrix corresponding to the loss function that meets the optimization conditions as the optimization result, and use the optimization result to adjust the camera extrinsic parameters corresponding to the corrected image.

[0125] The optimization condition is to minimize the loss function, which is the reference extrinsic matrix corresponding to the minimum loss function. After iteratively optimizing the reference extrinsic matrix of the reference corrected image corresponding to all vertices in the current corrected image, the reference corrected image corresponding to the minimum error between the gray value of the visible vertex in the current corrected image and the standard value of the visible vertex is obtained, i.e., the minimum loss function is obtained. The camera pose is then adjusted according to the reference extrinsic matrix corresponding to the aforementioned reference corrected image.

[0126] For each corrected image, the visible vertices in the corrected image are determined based on the set of visible vertex images. For each visible vertex and its corresponding reference extrinsic matrix of each corrected image, iterative optimization is performed. The loss function corresponding to the reference extrinsic matrix is ​​calculated, and the reference extrinsic matrix corresponding to the minimum loss function is taken as the optimization result, which improves the accuracy of the optimization result and thus improves the accuracy of image stitching.

[0127] Furthermore, the corrected image can be upsampled using a linear interpolation method to obtain an n-layer image corresponding to the corrected image.

[0128] In the n-layer image, the first-layer image is the corrected image, and the (j+1)-th-layer image is obtained by upsampling the j-th-layer image. n is an integer greater than 1, and j is an integer greater than or equal to 1.

[0129] Step 105 can then be achieved in the following way:

[0130] Based on the set of vertex visible images, the n-layer images corresponding to each corrected image are sorted in descending order, and the extrinsic parameter matrices corresponding to each of the n-layer images are optimized to obtain the layered optimization results corresponding to each of the n-layer images. The layered optimization results that meet the optimization conditions are used as the optimization results of the corrected image, and the camera extrinsic parameters corresponding to the corrected image are adjusted using the optimization results.

[0131] The n-layer images are sorted in descending order, starting from the nth layer and ending at the 1st layer. The extrinsic parameter matrix corresponding to each layer image is optimized separately. The optimization process for each layer image can be referred to steps 1051 to 1054.

[0132] After obtaining the layer optimization results corresponding to each of the n layers of images, the extrinsic parameter matrix corresponding to the layer optimization result with the smallest loss function is selected as the optimization result of the corrected image corresponding to the n layers of images, and the camera extrinsic parameters corresponding to the corrected image are adjusted using the aforementioned extrinsic parameter matrix.

[0133] By upsampling the corrected image, multiple images with different resolutions corresponding to the corrected image are obtained, which increases the data input during camera extrinsic parameter optimization and makes the obtained optimization results more accurate.

[0134] Furthermore, in Figure 1 In the method shown, step 105 may further include:

[0135] Step 106: Render the texture map to the Mesh model through texture reconstruction, and visualize the optimized image stitching effect.

[0136] Texture reconstruction utilizes texture techniques to map images onto the three-dimensional geometric surface of a target, forming the object's color information. By rendering two-dimensional images onto a three-dimensional model, the image stitching effect after camera extrinsic parameter optimization is visualized, allowing for intuitive evaluation of the optimization results and improving their recognizability.

[0137] This application embodiment obtains scene pose data and multiple fisheye images from different perspectives using a data acquisition device. The scene pose data includes at least gyroscope data, accelerometer data, and point cloud data. A mesh model is obtained based on the scene pose data. Camera calibration is performed using the scene pose data and fisheye images obtained from the data acquisition device, reducing the operational difficulty of camera calibration. Distortion correction is performed on multiple fisheye images to obtain corrected images corresponding to each fisheye image. Based on the mesh model and the corrected images, visibility analysis is performed on each vertex in the mesh model to obtain a set of vertex visible images. The vertex visible image set includes... The system stores the visibility relationship between each vertex and the corrected image. In the corrected image where there is a visibility relationship with a vertex, the vertex is visible. Based on the set of visible vertex images, the extrinsic parameter matrix corresponding to each corrected image is optimized to obtain the optimization result. The optimization result is then used to adjust the camera extrinsic parameters corresponding to the corrected image. The two-dimensional image and the three-dimensional space are linked through image depth information. Visibility analysis is performed on the vertex coordinates in the Mesh model and the corrected image, enabling self-calibration of camera extrinsic parameters without markers. This simplifies the camera extrinsic parameter calibration process, improves the convenience of camera extrinsic parameter calibration, and enhances the accuracy of image stitching in calibration scenarios without markers.

[0138] See Figure 2 As shown, the scenario embodiment provided in this application includes the following steps:

[0139] Step 201: Data preprocessing.

[0140] like Figure 3 As shown, data acquisition is performed by remotely controlling the hardware acquisition device through the terminal control device. The device obtains the camera's intrinsic and extrinsic parameter calibration information during local calibration, obtains gyroscope and accelerometer data through the IMU hardware inertial navigation module, obtains laser scanning data through the LiDAR 3D module, and obtains fisheye image information in four directions through the panoramic camera vision module. The obtained data is then input into the SLAM process, and after running, point cloud data, trajectory, and map are obtained. Poisson reconstruction is performed on the point cloud data to obtain the Mesh model.

[0141] Step 202: External parameter optimization preprocessing:

[0142] like Figure 4 As shown, based on the joint calibration product obtained during local calibration, the first transformation matrix T_imu_pi between the IMU coordinate system and the π coordinate system corresponding to the four fisheye images is obtained.

[0143] Based on the positions of the vertices in the Mesh model, obtain the second transformation matrix T_imu_world between the world coordinate system and the IMU coordinate system.

[0144] Based on T_imu_pi and T_imu_world, the third transformation matrix T_world_pi between the world coordinate system and the camera coordinate system is obtained through matrix operations.

[0145] After applying T_world_pi to the four fisheye images to correct distortion, the image resolution was adjusted to obtain corrected images with a resolution of 1140×1140 for each of the four fisheye images, and the camera intrinsic parameters corresponding to the corrected images were saved.

[0146] Step 203: Construct the image pyramid.

[0147] By using linear interpolation, each image in the corrected image is upsampled with a sampling factor of 1.2 to obtain an 8-layer image pyramid, such as... Figure 5 As shown, the top layer is the original corrected image. Simultaneously, the camera intrinsic parameters corresponding to each layer of the image are saved.

[0148] Step 204: Obtain image depth information.

[0149] like Figure 6 As shown, T_world_pi is used to perform projection calculation on the Mesh model and the corrected image, and finally the depth information corresponding to each pixel in all corrected images is obtained and saved as image depth information.

[0150] Step 205: Depth map boundary segmentation.

[0151] like Figure 7 As shown, a depth map is obtained based on the image depth information. After Gaussian filtering, the gradient information of each pixel in the depth map in the x and y directions is calculated, and then the total gradient of each pixel is obtained. Boundary segmentation is performed based on the total gradient of all pixels. Pixels with a total gradient greater than the gradient threshold are set to white, and pixels with a total gradient not greater than the gradient threshold are set to black. After dilation, a depth boundary map is obtained. The gradient threshold is set to 0.1 meters.

[0152] Step 206: Vertex and image visibility analysis.

[0153] like Figure 8 As shown, using the optical center coordinates from the camera's intrinsic parameters and the vertex coordinates from the mesh model, the vertex coordinates are transformed to the camera coordinate system, and the vertex projection coordinates (u, v) of each vertex in the mesh model in the pixel coordinate system are calculated. Based on the vertex projection coordinates and the depth boundary map, visibility analysis is performed on the vertices and the corrected images, obtaining the set of images where vertices are visible and the set of vertices where images are visible. The set of images where vertices are visible is grouped according to different vertices, and each group includes the corrected images visible to that vertex. The set of vertices where images are visible is grouped according to different corrected images, and each group includes the vertices visible in that corrected image.

[0154] Step 207: Optimize the problem solution.

[0155] The formula used to optimize the extrinsic parameter matrix corresponding to each corrected image is shown below:

[0156]

[0157] r i,p =C(p)-Γ i (p,T i )

[0158] Where, f(T) i ) is the loss function for the i-th corrected image, and C(p) is the standard value of vertex p; P i T is the set of visible vertices of the i-th image; i It is the extrinsic parameter matrix corresponding to the i-th corrected image, which can be represented as (T_world_imu·T_imu_pi). T to indicate; r i,p It is the residual term, used to represent the grayscale error between the standard value and the grayscale value of vertex p; Γ i (p,T i ) is the gray value of vertex p in the i-th corrected image.

[0159] Among them, Γ i (p,T i This is equivalent to a composite function, as shown in the following equation:

[0160]

[0161] g(p,T i Let be the coordinates of vertex p in the three-dimensional coordinate system, calculated using the following formula:

[0162] g(p,T i ) = T i ·p=(g x ,gy ,g z ,g w ) T

[0163] Among them, g x ,g y ,g z Let g be the coordinates of vertex p on the x-axis, y-axis, and z-axis in the three-dimensional coordinate system, respectively. w Let be the angle value of vertex p in the three-dimensional coordinate system.

[0164] The transformation formula from the 3D coordinate system to the pixel coordinate system is shown below:

[0165]

[0166] Among them, f x ,f y Let c be the coordinates of vertex p in the pixel coordinate system. x ,c y These are the coordinates of the origin in the pixel coordinate system.

[0167] The vertex p is transformed into the camera coordinate system using the camera pose, and then transformed into the pixel coordinate system based on the camera intrinsic parameters to obtain the gray value of the projection point of vertex p in the pixel coordinate system. The Gauss-Newton method is then used to calculate f(T). i ) to perform optimization and solution.

[0168] After iteratively optimizing the 8-layer image pyramid from bottom to top, f(T) is obtained. i The minimum value corresponds to a set of IMU-camera extrinsic parameter matrices T_imu_pi. The extrinsic parameter matrix T_master_imu of the lidar-IMU can be considered fixed. Thus, the lidar-camera extrinsic parameter matrix can be obtained as shown in the following formula:

[0169] T_master_pi=T_master_imu·T_imu_pi

[0170] The extrinsic parameter matrix above is used to adjust the extrinsic parameters of the LiDAR-camera system, thereby optimizing the image stitching effect. Figure 9 As shown, Figure 9 (a) and Figure 9 (b) Image stitching effect obtained without using the LiDAR-camera extrinsic parameter self-calibration method provided in this application; Figure 9 (c) and Figure 9 (d) is the image stitching effect obtained after using the LiDAR-camera extrinsic parameter self-calibration method provided in this application, wherein... Figure 9 (c) and Figure 9 (a) Correspondence Figure 9 (d) and Figure 9(b) Correspondingly, it can be seen that the self-calibration method for lidar-camera extrinsic parameters provided in this application improves the accuracy of image stitching and optimizes the image stitching effect.

[0171] Although the operations are described in a specific order, this should not be construed as requiring these operations to be performed in the specific order shown or in a sequential order. In certain environments, multitasking and parallel processing may be advantageous.

[0172] It should be understood that the steps described in the method embodiments of this application may be performed in different orders and / or in parallel. Furthermore, the method embodiments may include additional steps and / or omit the steps shown. The scope of this application is not limited in this respect.

[0173] See Figure 10 As shown, the lidar-camera extrinsic parameter self-calibration device provided in this application includes: a data acquisition unit 1001, a data processing unit 1002, an image correction unit 1003, an image analysis unit 1004, and an extrinsic parameter optimization unit 1005.

[0174] The data acquisition unit 1001 is used to: acquire scene pose data and multiple fisheye images from different perspectives through a data acquisition device; wherein the scene pose data includes at least: gyroscope data, accelerometer data, and point cloud data;

[0175] The data processing unit 1002 is used to obtain a mesh model based on scene pose data.

[0176] The image correction unit 1003 is used to: perform distortion correction on multiple fisheye images respectively, and obtain the corrected images corresponding to each of the multiple fisheye images.

[0177] The image analysis unit 1004 is used to: perform visibility analysis on each vertex in the Mesh model based on the Mesh model and the corrected image to obtain a set of vertex visible images; wherein, the vertex visible image set stores the visibility relationship between each vertex and the corrected image, and the vertex is visible in the corrected image where there is a visibility relationship with the vertex.

[0178] The extrinsic parameter optimization unit 1005 is used to: optimize the extrinsic parameter matrix corresponding to each corrected image based on the vertex visible image set, obtain the optimization result, and use the optimization result to adjust the camera extrinsic parameters corresponding to the corrected image.

[0179] Furthermore, the extrinsic parameter optimization unit 1005 is specifically used for:

[0180] For each corrected image, the visible vertices in the corrected image are determined based on the set of visible vertex images; the visible vertices are the vertices in the Mesh model that have a visible relationship with the corrected image.

[0181] For each visible vertex, based on the set of visible images of the vertex, a corrected image that has a visible relationship with the visible vertex is determined and used as the reference corrected image corresponding to the corrected image;

[0182] For each reference corrected image, the reference extrinsic matrix is ​​iteratively optimized, and the loss function corresponding to the reference extrinsic matrix is ​​calculated.

[0183] The reference extrinsic matrix corresponding to the loss function that meets the optimization conditions is used as the optimization result, and the camera extrinsic parameters corresponding to the corrected image are adjusted using the optimization result.

[0184] Furthermore, the extrinsic parameter optimization unit 1005 is specifically used for:

[0185] For each visible vertex, the k-th grayscale error of the visible vertex is determined based on the grayscale value of the visible vertex in the k-th reference corrected image and the standard value of the visible vertex; where the k-th grayscale error corresponds to the k-th reference corrected image, and k is an integer greater than or equal to 1;

[0186] Based on the kth grayscale error of all visible vertices in the corrected image, the kth loss function corresponding to the kth reference corrected image is obtained.

[0187] Furthermore, the image analysis unit 1004 is specifically used for:

[0188] Based on the mesh model and the corrected image, image depth information is obtained; wherein, the image depth information includes the distance between each vertex in the mesh model and the camera that acquired the fisheye image;

[0189] Based on the image depth information, visibility analysis is performed on each vertex in the Mesh model to obtain a set of vertex visible images.

[0190] Furthermore, the image correction unit 1003 is specifically used for:

[0191] Based on the local calibration results, the first transformation matrix between the inertial measurement unit (IMU) coordinate system and the camera coordinate system is obtained; wherein, the camera coordinate system is the coordinate system corresponding to the fisheye image;

[0192] Based on the positions of the vertices in the Mesh model, obtain the second transformation matrix between the world coordinate system and the IMU coordinate system; where the world coordinate system is the coordinate system corresponding to the Mesh model.

[0193] Based on the first and second transformation matrices, the third transformation matrix between the world coordinate system and the camera coordinate system is obtained;

[0194] The distortion is corrected for each of the multiple fisheye images based on the third transformation matrix, resulting in the corrected images corresponding to each fisheye image.

[0195] Furthermore, the image analysis unit 1004 is specifically used for:

[0196] Based on the mesh model and the corrected image, image depth information is obtained; wherein, the image depth information includes the distance between each vertex in the mesh model and the camera that acquired the fisheye image;

[0197] Generate a depth boundary map based on the image depth information;

[0198] Based on the third transformation matrix, the vertex coordinates in the Mesh model are transformed into vertex camera coordinates in the camera coordinate system;

[0199] Based on the camera intrinsic parameters and the pinhole model, obtain the vertex pixel coordinates in the pixel coordinate system on the image plane, where the vertex camera coordinates are located.

[0200] Based on the depth boundary map and vertex pixel coordinates, a visibility analysis is performed on the i-th vertex and the corrected image to obtain the set of visible images corresponding to the i-th vertex; where the set of visible images corresponding to the i-th vertex includes all corrected images visible to the i-th vertex, and i is an integer greater than or equal to 1;

[0201] Generate a set of visible images for each vertex based on the set of visible images corresponding to each vertex in the Mesh model.

[0202] Furthermore, the image analysis unit 1004 is specifically used for:

[0203] Draw a depth map from the image's depth information;

[0204] After applying Gaussian filtering to the depth map, the gradient values ​​of the pixels in the depth map are calculated.

[0205] For each pixel in the depth map, if the gradient value of the pixel is greater than the gradient threshold, then the pixel value of the pixel is set to the first pixel value; if the gradient value of the pixel is not greater than the gradient threshold, then the pixel value of the pixel is set to the second pixel value; the first pixel value is not equal to the second pixel value.

[0206] Draw a depth boundary map based on the pixel values ​​set for each pixel in the depth map.

[0207] Furthermore, Figure 10 The illustrated device further includes an image sampling unit for:

[0208] The corrected image is upsampled using a linear interpolation method to obtain an n-layer image corresponding to the corrected image. Among the n-layer images, the first layer image is the corrected image, and the (j+1)th layer image is obtained by upsampling the j-th layer image. n is an integer greater than 1, and j is an integer greater than or equal to 1.

[0209] The extrinsic parameter optimization unit 1005 is specifically used for:

[0210] Based on the set of vertex visible images, the n-layer images corresponding to each corrected image are sorted in descending order, and the extrinsic matrix of each of the n-layer images is optimized to obtain the layered optimization results of each of the n-layer images.

[0211] The hierarchical optimization results that meet the optimization conditions are used as the optimization results of the corrected image, and the camera extrinsic parameters corresponding to the corrected image are adjusted using the optimization results.

[0212] Furthermore, Figure 10 The device shown further includes a visualization output unit for: rendering a texture map to a Mesh model through texture reconstruction, and visualizing the optimized image stitching effect.

[0213] The names of the messages or information exchanged between multiple devices in the embodiments of this application are for illustrative purposes only and are not intended to limit the scope of these messages or information.

[0214] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0215] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be an indirect coupling or communication connection between apparatuses or units through some interfaces, and may be electrical, mechanical, or other forms.

[0216] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0217] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0218] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing computer programs, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0219] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. A method for self-calibrating extrinsic parameters of a lidar-camera system, characterized in that, The method includes: Scene pose data and multiple fisheye images from different perspectives are obtained through data acquisition equipment; A mesh model is obtained based on the scene pose data; Distortion correction is performed on the multiple fisheye images respectively to obtain the corrected images corresponding to each of the multiple fisheye images; Based on the Mesh model and the corrected image, a visibility analysis is performed on each vertex in the Mesh model to obtain a set of vertex visible images; wherein, the set of vertex visible images stores the visibility relationship between each vertex and the corrected image, and in the corrected image with the visibility relationship with the vertex, the vertex is visible; Based on the set of vertex visible images, the extrinsic parameter matrix corresponding to each of the corrected images is optimized to obtain the optimization result, and the camera extrinsic parameters corresponding to the corrected image are adjusted using the optimization result.

2. The method according to claim 1, characterized in that, The step of iteratively optimizing the extrinsic parameter matrix corresponding to each corrected image based on the vertex visible image set to obtain the optimization result, and using the optimization result to adjust the camera extrinsic parameters corresponding to the corrected image, includes: For each of the corrected images, the visible vertices in the corrected image are determined according to the set of visible vertex images; the visible vertices are the vertices in the Mesh model that have the visibility relationship with the corrected image. For each visible vertex, based on the set of visible images of the vertex, the corrected image that has the visible relationship with the visible vertex is determined as the reference corrected image corresponding to the corrected image; Iterative optimization is performed on the reference extrinsic matrix corresponding to each of the reference corrected images, and the loss function corresponding to the reference extrinsic matrix is ​​calculated. The reference extrinsic matrix corresponding to the loss function that meets the optimization conditions is used as the optimization result, and the camera extrinsic parameters corresponding to the corrected image are adjusted using the optimization result.

3. The method according to claim 2, characterized in that, The step of iteratively optimizing the reference extrinsic parameter matrix corresponding to each of the reference corrected images to determine the loss function corresponding to the reference extrinsic parameter matrix includes: For each visible vertex, the k-th grayscale error of the visible vertex is determined based on the grayscale value of the visible vertex in the k-th reference corrected image and the standard value of the visible vertex; wherein the k-th grayscale error corresponds to the k-th reference corrected image, and k is an integer greater than or equal to 1; Based on the kth grayscale error of all visible vertices in the corrected image, the kth loss function corresponding to the kth reference corrected image is obtained.

4. The method according to claim 1, characterized in that, The step of performing visibility analysis on each vertex in the Mesh model based on the Mesh model and the corrected image to obtain a set of vertex visible images includes: Based on the mesh model and the corrected image, image depth information is obtained; wherein, the image depth information includes the distance between each vertex in the mesh model and the camera that acquired the fisheye image; Based on the image depth information, visibility analysis is performed on each vertex in the Mesh model to obtain a set of vertex visible images.

5. The method according to claim 1, characterized in that, The step of performing distortion correction on the plurality of fisheye images to obtain the corrected images corresponding to each of the plurality of fisheye images includes: Based on the local calibration results, a first transformation matrix is ​​obtained between the inertial measurement unit (IMU) coordinate system and the camera coordinate system; wherein, the camera coordinate system is the coordinate system corresponding to the fisheye image; Based on the positions of the vertices in the Mesh model, a second transformation matrix between the world coordinate system and the IMU coordinate system is obtained; wherein, the world coordinate system is the coordinate system corresponding to the Mesh model; Based on the first transformation matrix and the second transformation matrix, a third transformation matrix between the world coordinate system and the camera coordinate system is obtained; The distortion is corrected for each of the multiple fisheye images according to the third transformation matrix to obtain the corrected image corresponding to each of the multiple fisheye images.

6. The method according to claim 5, characterized in that, The step of performing visibility analysis on each vertex in the Mesh model based on the Mesh model and the corrected image to obtain a set of vertex visible images includes: Based on the mesh model and the corrected image, image depth information is obtained; wherein, the image depth information includes the distance between each vertex in the mesh model and the camera that acquired the fisheye image; Generate a depth boundary map based on the image depth information; Based on the third transformation matrix, the vertex coordinates in the Mesh model are transformed into vertex camera coordinates in the camera coordinate system; Based on the camera intrinsic parameters and the pinhole model, obtain the vertex pixel coordinates of the vertex camera coordinates in the pixel coordinate system on the image plane; Based on the depth boundary map and the vertex pixel coordinates, a visibility analysis is performed on the i-th vertex and the corrected image to obtain the set of visible images corresponding to the i-th vertex; wherein, the set of visible images corresponding to the i-th vertex includes all the corrected images visible to the i-th vertex, and i is an integer greater than or equal to 1; A vertex visible image set is generated based on the visible image set corresponding to each vertex in the Mesh model.

7. The method according to claim 6, characterized in that, The step of generating a depth boundary map based on the image depth information includes: The image depth information is plotted into a depth map; After applying Gaussian filtering to the depth map, the gradient values ​​of the pixels in the depth map are calculated. For each pixel in the depth map, if the gradient value of the pixel is greater than the gradient threshold, then the pixel value of the pixel is set to a first pixel value; if the gradient value of the pixel is not greater than the gradient threshold, then the pixel value of the pixel is set to a second pixel value; the first pixel value is not equal to the second pixel value. A depth boundary map is drawn based on the pixel values ​​set for each pixel in the depth map.

8. The method according to claim 1, characterized in that, The method further includes: The corrected image is upsampled using a linear interpolation method to obtain an n-layer image corresponding to the corrected image; wherein, in the n-layer image, the first layer image is the corrected image, the (j+1)th layer image is obtained by upsampling the j-th layer image, n is an integer greater than 1, and j is an integer greater than or equal to 1. The step of optimizing the extrinsic parameter matrix corresponding to each corrected image based on the vertex visible image set, obtaining the optimization result, and using the optimization result to adjust the camera extrinsic parameters corresponding to the corrected image includes: Based on the vertex visible image set, the n-layer images corresponding to each of the corrected images are sorted in descending order, and the extrinsic parameter matrices corresponding to each of the n-layer images are optimized to obtain the layered optimization results corresponding to each of the n-layer images. The hierarchical optimization result that meets the optimization conditions is used as the optimization result of the corrected image, and the camera extrinsic parameters corresponding to the corrected image are adjusted using the optimization result.

9. The method according to claim 1, characterized in that, After adjusting the camera extrinsic parameters corresponding to the corrected image using the optimization result, the method further includes: By reconstructing textures, the texture map is rendered onto the Mesh model, and the optimized image stitching effect is visualized and output.

10. A self-calibration device for extrinsic parameters of a lidar-camera system, characterized in that, The device includes: The data acquisition unit is used to: acquire scene pose data and multiple fisheye images from different perspectives through a data acquisition device; wherein the scene pose data includes at least: gyroscope data, accelerometer data, and point cloud data; The data processing unit is used to: obtain a mesh model based on the scene pose data; An image correction unit is used to: perform distortion correction on the plurality of fisheye images respectively, and obtain a corrected image corresponding to each of the plurality of fisheye images; An image analysis unit is configured to: perform visibility analysis on each vertex in the Mesh model based on the Mesh model and the corrected image to obtain a set of vertex visible images; wherein the set of vertex visible images stores the visibility relationship between each vertex and the corrected image, and the vertex is visible in the corrected image where the visibility relationship exists with the vertex; The extrinsic parameter optimization unit is used to: optimize the extrinsic parameter matrix corresponding to each of the corrected images based on the vertex visible image set, obtain the optimization result, and use the optimization result to adjust the camera extrinsic parameters corresponding to the corrected image.

Images