A method and apparatus for joint calibration of a lidar and camera

Abstract

The application relates to the technical field of computers and discloses a joint calibration method and device for a laser radar and a camera, which comprises the following steps: acquiring a first image dataset and a first point cloud dataset of a calibration board, the first image dataset being collected by the camera under at least one posture of the calibration board, and the first point cloud dataset being collected by the laser radar under at least one posture of the calibration board; processing the first image dataset by using a two-dimensional image semantic segmentation model to identify a pixel feature set of the calibration board in the image; processing the first point cloud dataset by using a three-dimensional point cloud voxel segmentation model to identify a voxel feature set of the calibration board in the point cloud; and determining calibration parameters between the camera and the laser radar based on the pixel feature set and the voxel feature set.

Description

Technical Field

[0001] This invention relates to the field of computer technology, and more specifically to a method and apparatus for the joint calibration of a lidar and a camera. Background Technology

[0002] Currently, joint calibration technology for LiDAR and cameras typically requires manual calibration. Specifically, this involves manually selecting feature points on the calibration object and manually matching the correspondences of these feature points in the LiDAR point cloud and the image. These corresponding points are then input into the calibration algorithm to calculate the extrinsic parameters between the LiDAR and the camera. Existing technologies suffer from low calibration efficiency and accuracy that relies heavily on human experience, resulting in high labor costs. Therefore, how to accurately and efficiently jointly calibrate the extrinsic parameters between LiDAR and the camera is a problem that urgently needs to be solved. Summary of the Invention

[0003] In view of this, the present invention provides a method and apparatus for joint calibration of lidar and camera, in order to solve the problem of how to accurately and efficiently jointly calibrate the extrinsic parameters between lidar and camera.

[0004] In a first aspect, the present invention provides a joint calibration method for a lidar and a camera, the method comprising: Obtain the first image dataset and the first point cloud dataset of the calibration board. The first image dataset is acquired by the camera in at least one pose of the calibration board, and the first point cloud dataset is acquired by the lidar in at least one pose of the calibration board. The first image dataset is processed using a two-dimensional image semantic segmentation model to identify the pixel feature set of the calibration board in the image; The first point cloud dataset was processed using a 3D point cloud voxel segmentation model to identify the voxel feature set of the calibration board in the point cloud. Based on pixel feature sets and voxel feature sets, the calibration parameters between the camera and the LiDAR are determined.

[0005] In one alternative implementation, the two-dimensional image semantic segmentation model is trained on a target segmentation network with undetermined parameters using a second image dataset. The second image dataset was acquired by the camera in multiple poses on the calibration board; The training process for the target segmentation network includes: training and validation sub-processes; The training and validation sub-processes include: After one round of training of the target segmentation network is completed, the parameters of the target segmentation network are updated using the validation set; By finding the second image corresponding to the training operation from the second image dataset, the second image is input into the image segmentation network to obtain the prediction result of the image segmentation network corresponding to the training operation, and the loss between the prediction result of the image segmentation network corresponding to the training operation and the annotation result of the second image corresponding to the training operation is calculated. The loss is used to update the parameters of the image segmentation network. By finding the second image corresponding to the test operation from the second image dataset, the target segmentation network is evaluated to obtain the evaluation result corresponding to the test operation; After training the target segmentation network, the following is also included: Based on the evaluation results of each round of training, the optimal target segmentation network is determined as the two-dimensional image semantic segmentation model.

[0006] In one alternative implementation, the 3D point cloud voxel segmentation model is obtained by training a point cloud segmentation network with undetermined parameters using a second point cloud dataset. The second cloud dataset was collected by the LiDAR under multiple poses on the calibration board; The training process for a point cloud segmentation network includes: training and validation sub-processes; The training and validation sub-processes include: After one round of training of the point cloud segmentation network is completed, the parameters of the point cloud segmentation network are updated using the validation set; By finding the second point cloud corresponding to the training operation from the second point cloud dataset, the second point cloud is input into the point cloud segmentation network to obtain the prediction result of the point cloud segmentation network corresponding to the training operation, and the loss between the prediction result of the point cloud segmentation network corresponding to the training operation and the annotation result of the second point cloud corresponding to the training operation is calculated. The loss is used to update the parameters of the point cloud segmentation network. By finding the second point cloud corresponding to the test operation from the second point cloud dataset, the point cloud segmentation network is evaluated to obtain the evaluation result corresponding to the test operation; After training the point cloud segmentation network, the following steps are also included: Based on the evaluation results of each round of training, the optimal point cloud segmentation network is determined as the 3D point cloud voxel segmentation model.

[0007] In one optional implementation, calibration parameters between the camera and the LiDAR are determined based on pixel feature sets and voxel feature sets, including: By using a multi-point perspective pose solving method, preset camera intrinsic parameters, and preset distortion coefficients, pixel feature sets and voxel feature sets are processed to obtain calibration parameters.

[0008] In an optional implementation, after determining the calibration parameters between the camera and the LiDAR based on pixel feature sets and voxel feature sets, the method further includes: Based on the calibration parameters, the point cloud data collected in at least one pose in the first point cloud dataset are divided into different colors according to the distance to obtain the color-coded point cloud data. The colored point cloud data is projected onto image data acquired in at least one pose in the first image dataset to obtain the image data of the projected point cloud data.

[0009] In one alternative implementation, the image data includes image colors; After determining the calibration parameters between the camera and the LiDAR based on pixel feature sets and voxel feature sets, the process also includes: Based on calibration parameters and point cloud data acquired in at least one pose, extract the image color corresponding to the point cloud data acquired in at least one pose from the image data acquired in at least one pose. The image color is projected onto point cloud data acquired in at least one pose to obtain point cloud data with the image color.

[0010] Secondly, the present invention provides a joint calibration device for a lidar and a camera, the device comprising: The acquisition module is used to acquire the first image dataset and the first point cloud dataset of the calibration board. The first image dataset is acquired by the camera in at least one pose of the calibration board, and the first point cloud dataset is acquired by the lidar in at least one pose of the calibration board. The first recognition module is used to process the first image dataset using a two-dimensional image semantic segmentation model to identify the pixel feature set of the calibration board in the image; The second recognition module is used to process the first point cloud dataset using a three-dimensional point cloud voxel segmentation model to identify the voxel feature set of the calibration board in the point cloud. The determination module is used to determine the calibration parameters between the camera and the LiDAR based on pixel feature sets and voxel feature sets.

[0011] Thirdly, the present invention provides a computer device, comprising: a memory and a processor, the memory and the processor being communicatively connected to each other, the memory storing computer instructions, and the processor executing the computer instructions to perform the joint calibration method of lidar and camera described in the first aspect or any corresponding embodiment thereof.

[0012] Fourthly, the present invention provides a computer-readable storage medium storing computer instructions for causing a computer to execute the joint calibration method of lidar and camera according to the first aspect or any corresponding embodiment thereof.

[0013] Fifthly, the present invention provides a computer program product, including computer instructions for causing a computer to execute the joint calibration method of lidar and camera according to the first aspect or any corresponding embodiment thereof.

[0014] The embodiments provided by this invention have the following advantages: By processing image data and point cloud data separately, and using the pixel features of the image and the voxel features of the point cloud for joint analysis, the geometric relationship between the LiDAR and the camera can be determined more accurately, thereby calculating more accurate extrinsic parameters. Traditional joint calibration usually relies on manual selection of feature points, while this method, by introducing two-dimensional image semantic segmentation and three-dimensional point cloud voxel segmentation models, can automatically identify calibration features in images and point clouds, reducing manual intervention and improving efficiency and reliability. By processing image data and point cloud data under various poses, this method enhances its adaptability, making it applicable not only to different poses and positions but also capable of handling complex conditions in different scenarios, thus possessing high versatility in practical applications. Due to the use of automated semantic segmentation and voxel segmentation models, manual intervention in the calibration process can be reduced, lowering the time and cost of manual calibration and mitigating errors that may occur when manually selecting feature points. Attached Figure Description

[0015] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0016] Figure 1 This is a schematic flowchart of a joint calibration method for a lidar and a camera according to an embodiment of the present invention; Figure 2 This is a schematic diagram of a joint calibration device for a lidar and a camera according to an embodiment of the present invention; Figure 3 This is a schematic diagram of the hardware structure of a computer device according to an embodiment of the present invention. Detailed Implementation

[0017] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0018] According to an embodiment of the present invention, a joint calibration method for lidar and camera is provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0019] This embodiment provides a joint calibration method for lidar and camera, which can be used with a computer. Figure 1 This is a flowchart of a joint calibration method for a lidar and a camera according to an embodiment of the present invention, as shown below. Figure 1 As shown. The method includes: Step S101: Obtain the first image dataset and the first point cloud dataset of the calibration board.

[0020] The first image dataset was acquired by the camera in at least one pose of the calibration board, and the first point cloud dataset was acquired by the LiDAR in at least one pose of the calibration board.

[0021] During the calibration process, image data and point cloud data need to be acquired first. Both the camera and the LiDAR need to acquire data from multiple different poses (positions and angles) of the calibration board. This data will be used for subsequent calibration processing. The first image dataset consists of image data captured by the camera, recording the appearance of the calibration board from the camera's viewpoint. The first point cloud dataset consists of point cloud data acquired by the LiDAR, recording the three-dimensional spatial position of the calibration board.

[0022] Step S102: The first image dataset is processed using a two-dimensional image semantic segmentation model to identify the pixel feature set of the calibration board in the image.

[0023] The acquired image data is processed to identify pixel features related to the calibration board. A two-dimensional image semantic segmentation model is used to separate the calibration board portion from other irrelevant parts of the image, thereby identifying the pixel feature set of the calibration board. The task of the semantic segmentation model is to classify each pixel in the image, enabling the model to identify pixel regions belonging to the calibration board.

[0024] Step S103: Process the first point cloud dataset using the three-dimensional point cloud voxel segmentation model to identify the voxel feature set of the calibration plate in the point cloud.

[0025] The acquired point cloud data is processed to identify voxel features in the point cloud that are relevant to the calibration board. The 3D point cloud voxel segmentation model divides the point cloud into voxels (similar to pixels in 3D space) and identifies voxel regions relevant to the calibration board. This method allows for the acquisition of the calibration board's position and shape information in 3D space.

[0026] Step S104: Determine the calibration parameters between the camera and the LiDAR based on the pixel feature set and the voxel feature set.

[0027] In the preceding steps, we obtained the pixel feature set from the image and the voxel feature set from the point cloud, respectively. Now, by combining these two feature sets, we can determine the calibration parameters between the camera and the LiDAR, i.e., their geometric relationship. Calibration parameters include extrinsic parameters (the position and orientation of the camera and LiDAR), which help to combine the two-dimensional image data captured by the camera with the three-dimensional point cloud data measured by the LiDAR, thereby achieving precise spatial positioning and synchronization.

[0028] The joint calibration method for LiDAR and camera provided by this invention processes image data and point cloud data separately, and performs joint analysis using pixel features of the image and voxel features of the point cloud. This enables a more accurate determination of the geometric relationship between the LiDAR and camera, thereby calculating more accurate extrinsic parameters. Traditional joint calibration typically relies on manual selection of feature points, while this method, by introducing two-dimensional image semantic segmentation and three-dimensional point cloud voxel segmentation models, can automatically identify calibration features in images and point clouds, reducing manual intervention and improving efficiency and reliability. This method enhances its adaptability by processing image and point cloud data under various poses, making it applicable not only to different poses and positions but also capable of handling complex conditions in different scenarios, thus possessing high versatility in practical applications. The use of automated semantic segmentation and voxel segmentation models reduces manual intervention in the calibration process, lowering the time and cost of manual calibration and mitigating errors that may occur when manually selecting feature points.

[0029] In one optional implementation, the two-dimensional image semantic segmentation model is trained on a target segmentation network with undetermined parameters using a second image dataset. The second image dataset is acquired by the camera in multiple poses on a calibration board. The training process for the target segmentation network includes training and validation sub-processes. These sub-processes include: after one round of training of the target segmentation network, updating the parameters of the target segmentation network using a validation set; using a second image from the second image dataset corresponding to the training operation, inputting the second image into the image segmentation network to obtain the prediction result corresponding to the training operation output by the image segmentation network, and calculating the loss between the prediction result corresponding to the training operation output by the image segmentation network and the labeled result of the second image corresponding to the training operation, with the loss used to update the parameters of the image segmentation network; using a second image from the second image dataset corresponding to the test operation, evaluating the target segmentation network to obtain the evaluation result corresponding to the test operation. After training the target segmentation network, the process further includes: determining the optimal target segmentation network as the two-dimensional image semantic segmentation model based on the evaluation results of each round of training.

[0030] In this embodiment, the target segmentation network is trained using a second image dataset. The second image dataset consists of images captured by a camera at various poses on a calibration board. Training images extracted from the second image dataset are input into the target segmentation network. The network makes predictions based on the input images and outputs corresponding image segmentation results. The prediction results are compared with the labeled results (i.e., the true segmented targets), and a loss value is calculated. The target segmentation network is validated using a validation set to evaluate its performance. The model parameters are updated and optimized using the validation set. The loss between the target segmentation network's prediction results and the true labels is calculated. The loss value is obtained through a loss function. This loss value reflects the gap between the network's prediction results and the actual labels; the smaller the loss, the better the model's performance. The model is finally evaluated using images from the test set. The evaluation results typically use common metrics to measure the model's performance in practical applications. The evaluation results after each training round are used to select the optimal model. By comparing the evaluation results after each training round, the target segmentation network with the best performance is finally selected.

[0031] The method provided in this embodiment utilizes images acquired under different poses from a second image dataset for training, allowing the model to learn from diverse perspectives and thus improving its segmentation ability in different scenes and angles. This training method helps reduce overfitting caused by uneven data distribution and enhances the model's generalization ability. Parameter updates during training and validation help the network continuously optimize in multiple training rounds, improving image segmentation accuracy. During validation, timely updates to network parameters effectively prevent overfitting, ensuring the model performs well even on new and unseen data. Automatically selecting the optimal model after each training and evaluation round avoids manual intervention, reduces bias from human judgment, and ensures the final selected model has the best performance. This automated process improves training efficiency and ensures the model's stability and efficiency. By training the image segmentation network, this method can automatically label a large area, improving labeling efficiency and reducing the time and cost of manual labeling, especially in applications requiring large-scale image segmentation tasks, bringing significant cost benefits.

[0032] In one optional implementation, the 3D point cloud voxel segmentation model is trained on a point cloud segmentation network with undetermined parameters using a second point cloud dataset. The second point cloud dataset is collected by a LiDAR at multiple poses on a calibration board. The training process for the point cloud segmentation network includes training and validation sub-processes. These sub-processes include: after one round of training of the point cloud segmentation network, updating the network parameters using a validation set; using a second point cloud corresponding to the training operation found in the second point cloud dataset, inputting the second point cloud into the point cloud segmentation network to obtain the network's output prediction result corresponding to the training operation, and calculating the loss between the network's output prediction result corresponding to the training operation and the labeled result of the second point cloud corresponding to the training operation, with the loss used to update the network parameters; using a second point cloud corresponding to the test operation found in the second point cloud dataset to evaluate the point cloud segmentation network, obtaining the evaluation result corresponding to the test operation. After training the point cloud segmentation network, the process further includes: based on the evaluation results of each round of training, determining the optimal point cloud segmentation network as the 3D point cloud voxel segmentation model.

[0033] In this embodiment, the second point cloud dataset is obtained by scanning the calibration board with a LiDAR device under different postures. Training point clouds selected from the second point cloud dataset are input into the point cloud segmentation network, and the network calculates prediction results. The network's prediction results are compared with the corresponding ground truth labels to calculate the loss value. The model is validated using a validation set to evaluate the performance of the network's current parameters. Based on the validation results, the model parameters are fine-tuned to prevent overfitting or underfitting. The trained model is then finally evaluated using point cloud data from the test set. Evaluation metrics are calculated to assess the model's performance in practical applications. By comparing the evaluation results of each training round, the optimal point cloud segmentation network is selected. The final optimal model is the 3D point cloud voxel segmentation model, capable of efficiently and accurately segmenting new point cloud data.

[0034] The method provided in this embodiment, by using point cloud data collected under different poses, enables the model to better learn the diversity of point cloud data, thereby improving the segmentation accuracy and robustness. The model performs more stably when facing different angles and complex environments. Training with multi-pose data allows the model to adapt to point cloud segmentation tasks in different scenarios, reducing overfitting problems caused by data limitations and thus improving the model's generalization ability. Frequent evaluation and parameter updates of the network using a validation set ensure the model remains in an optimal state during training. This avoids overfitting or underfitting, improving the model's stability and practicality. Automated training and validation processes allow for rapid and accurate adjustment of network parameters. Using loss functions and backpropagation algorithms, the model can be optimized efficiently, reducing the time cost of manual intervention and parameter tuning.

[0035] In one optional implementation, calibration parameters between the camera and the LiDAR are determined based on pixel feature sets and voxel feature sets, including: processing the pixel feature sets and voxel feature sets using a multi-point perspective pose solving method, preset camera intrinsic parameters and preset distortion coefficients to obtain calibration parameters.

[0036] In this embodiment, image and point cloud data are acquired from different perspectives and angles to obtain pixel feature sets and voxel feature sets, respectively. The pixel feature set contains key point features in the image, while the voxel feature set contains 3D spatial features extracted from the point cloud data. During calibration, the acquired pixel and voxel feature sets are jointly processed using a multi-point perspective pose solving method. The multi-point perspective pose solving method is a method for calculating camera pose by using the 3D and 2D correspondences of multiple points, which can optimize the calculation of calibration parameters to a certain extent. To ensure calibration accuracy, the method uses preset camera intrinsic parameters (such as focal length, principal point position, etc.) and distortion coefficients (such as radial distortion, tangential distortion, etc.). These preset values ​​help eliminate the influence of camera lens distortion during calibration, ensuring that the obtained calibration parameters are more accurate. Through a specific processing flow, the pixel feature set and voxel feature set are effectively matched and optimized. Specifically, the features extracted in pixel space are registered with 3D points in voxel space, and the calibration parameters are finally obtained by accurately calculating the relationship between the two.

[0037] The method provided in this embodiment combines pixel and voxel feature sets for calibration. Utilizing a multi-point perspective pose solving method along with preset camera intrinsic parameters and distortion coefficients, it significantly improves the accuracy of the calibration process. Through effective processing and optimization of the feature sets, accurate calibration parameters can be obtained in a shorter time, making it suitable for real-time calibration applications. Because it employs a multi-point perspective pose solving method and considers camera distortion, the robustness of the entire system is enhanced, enabling it to meet calibration requirements in various real-world environments.

[0038] In one optional implementation, after determining the calibration parameters between the camera and the LiDAR based on pixel feature sets and voxel feature sets, the method further includes: based on the calibration parameters, dividing the point cloud data acquired in at least one pose in the first point cloud dataset into different colors according to distance to obtain color-coded point cloud data. The color-coded point cloud data is then projected onto image data acquired in at least one pose in the first image dataset to obtain image data of the projected point cloud data.

[0039] In this embodiment, after obtaining the calibration parameters, the first point cloud data, concentrated in at least one pose, is processed. Based on distance information, the point cloud data is classified, and different colors are assigned to point cloud data at different distances. The obtained color-coded point cloud data is then mapped onto image data acquired in at least one pose in the first image dataset using a projection method. After projection, the projected point cloud data is displayed on the image data. The display of these point cloud data in the image not only preserves their spatial position in three-dimensional space but also presents different depth information through color mapping.

[0040] The method provided in this embodiment, by projecting color-coded point cloud data onto an image, can intuitively display the distribution of point cloud data in the image and highlight depth information, thus enhancing the connection between the point cloud and the image. Using different colors to indicate the depth of the point cloud data makes the image data easier to understand and analyze. Different colors help observers quickly determine the distance and spatial structure of objects. Through the combined application of calibration parameters and projection methods, accurate matching of point cloud data and image data is ensured, reducing errors and improving the overall accuracy of calibration and projection.

[0041] In one optional implementation, the image data includes image color. After determining the calibration parameters between the camera and the LiDAR based on pixel feature sets and voxel feature sets, the method further includes: extracting the image color corresponding to the point cloud data acquired in at least one pose from the image data acquired in at least one pose, based on the calibration parameters and point cloud data acquired in at least one pose. Projecting the image color onto the point cloud data acquired in at least one pose yields point cloud data with the image color.

[0042] In this embodiment, calibration parameters are determined based on pixel feature sets and voxel feature sets. Pixel feature sets refer to pixel information in the image, while voxel feature sets are spatial point information in point cloud data. By comparing and matching these features, calibration parameters, including camera intrinsic and extrinsic parameters and possible distortion coefficients, can be derived. Based on the calibration parameters and point cloud data acquired in at least one pose, image colors corresponding to the point cloud data are further extracted from the image data acquired in at least one pose. This process relies on calibration parameters and pose information to ensure accurate acquisition of color information corresponding to each point cloud data point from the image. The image color information extracted from the image is projected onto the point cloud data acquired in at least one pose. Thus, each point in the point cloud data will no longer only contain position coordinates but will also include the color information of that point in the image, forming point cloud data with image color.

[0043] The method provided in this embodiment enhances the visualization of point cloud data by incorporating image color information into the data. The introduction of color makes the point cloud data richer and more realistic, facilitating further analysis and understanding. By utilizing calibration parameters, it ensures precise matching between the spatial location of image color information and point cloud data, reducing color inconsistencies caused by errors. Combining point cloud data with image color provides more accurate and intuitive representations in various application scenarios. Through the projected colored point cloud data, researchers or automated systems can more easily perform in-depth analysis, identifying the correspondence between objects in the image and the point cloud data, improving the accuracy and efficiency of automated processing.

[0044] This invention provides a joint calibration device for lidar and camera. Figure 2 This is a schematic diagram of a joint calibration device for a lidar and a camera according to an embodiment of the present invention. The device includes: The acquisition module 201 is used to acquire a first image dataset and a first point cloud dataset of the calibration board. The first image dataset is acquired by the camera in at least one pose of the calibration board, and the first point cloud dataset is acquired by the lidar in at least one pose of the calibration board.

[0045] The first recognition module 202 is used to process the first image dataset using a two-dimensional image semantic segmentation model to identify the pixel feature set of the calibration plate in the image.

[0046] The second recognition module 203 is used to process the first point cloud dataset using a three-dimensional point cloud voxel segmentation model to identify the voxel feature set of the calibration plate in the point cloud.

[0047] The determination module 204 is used to determine the calibration parameters between the camera and the LiDAR based on the pixel feature set and the voxel feature set.

[0048] The joint calibration device for lidar and camera provided by this invention processes image data and point cloud data separately, and performs joint analysis using pixel features of the image and voxel features of the point cloud. This enables a more accurate determination of the geometric relationship between the lidar and camera, thereby calculating more accurate extrinsic parameters. Traditional joint calibration typically relies on manual selection of feature points, while this method, by introducing two-dimensional image semantic segmentation and three-dimensional point cloud voxel segmentation models, can automatically identify calibration features in images and point clouds, reducing manual intervention and improving efficiency and reliability. This method enhances its adaptability by processing image and point cloud data under various poses, making it applicable not only to different poses and positions but also capable of handling complex conditions in different scenarios, thus possessing high versatility in practical applications. The use of automated semantic segmentation and voxel segmentation models reduces manual intervention in the calibration process, lowering the time and cost of manual calibration and mitigating potential errors from manual feature point selection.

[0049] Further functional descriptions of the above modules and units are the same as those in the corresponding embodiments described above, and will not be repeated here.

[0050] In this embodiment, the joint calibration device for lidar and camera is presented in the form of a functional unit. Here, a unit refers to an ASIC (Application Specific Integrated Circuit) circuit, a processor and memory that execute one or more software or fixed programs, and / or other devices that can provide the above functions.

[0051] This invention also provides a computer device having the above-described features. Figure 2 The image shows a combined calibration device for a lidar and a camera.

[0052] Please see Figure 3 , Figure 3 This is a schematic diagram of the structure of a computer device provided in an optional embodiment of the present invention, such as... Figure 3As shown, the computer device includes one or more processors 10, memory 20, and interfaces for connecting the components, including high-speed interfaces and low-speed interfaces. The components communicate with each other via different buses and can be mounted on a common motherboard or otherwise installed as needed. The processors can process instructions executed within the computer device, including instructions stored in or on memory to display graphical information of a GUI on external input / output devices (such as display devices coupled to the interfaces). In some alternative implementations, multiple processors and / or multiple buses can be used with multiple memories and multiple memory modules, if desired. Similarly, multiple computer devices can be connected, each providing some of the necessary operations (e.g., as a server array, a group of blade servers, or a multiprocessor system). Figure 3 Take a processor 10 as an example.

[0053] Processor 10 may be a central processing unit, a network processor, or a combination thereof. Processor 10 may further include an integrated circuit. The integrated circuit may be an application-specific integrated circuit (ASIC), a programmable logic device (PLD), or a combination thereof. The programmable logic device may be a complex programmable logic device (CAMP), a field-programmable gate array (FPGA), a general-purpose array logic (GPRS), or any combination thereof.

[0054] The memory 20 stores instructions executable by at least one processor 10 to cause the at least one processor 10 to perform the method shown in the above embodiment.

[0055] The memory 20 may include a program storage area and a data storage area. The program storage area may store the operating system and applications required for at least one function; the data storage area may store data created based on the use of the computer device. Furthermore, the memory 20 may include high-speed random access memory and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some alternative embodiments, the memory 20 may optionally include memory remotely located relative to the processor 10, and these remote memories may be connected to the computer device via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0056] The memory 20 may include volatile memory, such as random access memory; the memory may also include non-volatile memory, such as flash memory, hard disk or solid-state drive; the memory 20 may also include a combination of the above types of memory.

[0057] The computer device also includes a communication interface 30 for communicating with other devices or communication networks.

[0058] This invention also provides a computer-readable storage medium. The methods described above according to embodiments of the invention can be implemented in hardware or firmware, or implemented as computer code that can be recorded on a storage medium, or implemented as computer code downloaded via a network and originally stored on a remote storage medium or a non-transitory machine-readable storage medium and then stored on a local storage medium. Thus, the methods described herein can be processed by software stored on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. The storage medium can be a magnetic disk, optical disk, read-only memory, random access memory, flash memory, hard disk, or solid-state drive, etc.; further, the storage medium can also include combinations of the above types of memory. It is understood that computers, processors, microprocessor controllers, or programmable hardware include storage components capable of storing or receiving software or computer code, which, when accessed and executed by the computer, processor, or hardware, implements the methods shown in the above embodiments.

[0059] A portion of this invention can be applied as a computer program product, such as computer program instructions, which, when executed by a computer, can invoke or provide the methods and / or technical solutions according to the invention through the operation of the computer. Those skilled in the art will understand that the forms in which computer program instructions exist in a computer-readable medium include, but are not limited to, source files, executable files, installation package files, etc. Correspondingly, the ways in which computer program instructions are executed by a computer include, but are not limited to: the computer directly executing the instructions, or the computer compiling the instructions and then executing the corresponding compiled program, or the computer reading and executing the instructions, or the computer reading and installing the instructions and then executing the corresponding installed program. Here, the computer-readable medium can be any available computer-readable storage medium or communication medium accessible to a computer.

[0060] Although embodiments of the invention have been described in conjunction with the accompanying drawings, those skilled in the art can make various modifications and variations without departing from the spirit and scope of the invention, and such modifications and variations all fall within the scope defined by the appended claims.

Claims

1. A joint calibration method for lidar and camera, characterized in that, The method includes: Acquire a first image dataset and a first point cloud dataset for the calibration board. The first image dataset is acquired by the camera in at least one pose of the calibration board, and the first point cloud dataset is acquired by the lidar in at least one pose of the calibration board. The first image dataset is processed using a two-dimensional image semantic segmentation model to identify the pixel feature set of the calibration board in the image; The first point cloud dataset is processed using a 3D point cloud voxel segmentation model to identify the voxel feature set of the calibration board in the point cloud; Based on the pixel feature set and the voxel feature set, the calibration parameters between the camera and the lidar are determined.

2. The method according to claim 1, characterized in that, The two-dimensional image semantic segmentation model is obtained by training the target segmentation network using the second image dataset; The second image dataset was acquired by the camera in multiple poses of the calibration board; The training process for the target segmentation network includes: training and validation sub-processes; The training and validation sub-processes include: After one round of training of the target segmentation network is completed, the parameters of the target segmentation network are updated using the validation set; The second image corresponding to the training operation is found from the second image dataset. The second image is input into the image segmentation network to obtain the prediction result of the image segmentation network corresponding to the training operation. The loss between the prediction result of the image segmentation network corresponding to the training operation and the annotation result of the second image corresponding to the training operation is calculated. The loss is used to update the parameters of the image segmentation network. The target segmentation network is evaluated by finding the second image corresponding to the test operation from the second image dataset, and the evaluation result corresponding to the test operation is obtained. After training the target segmentation network, the following is also included: Based on the evaluation results of each round of training, the optimal target segmentation network is determined as the two-dimensional image semantic segmentation model.

3. The method according to claim 1, characterized in that, The three-dimensional point cloud voxel segmentation model is obtained by training the point cloud segmentation network using the second point cloud dataset. The second point cloud dataset was collected by the lidar under multiple poses of the calibration board; The training process for a point cloud segmentation network includes: training and validation sub-processes; The training and validation sub-processes include: After one round of training of the point cloud segmentation network is completed, the parameters of the point cloud segmentation network are updated using the validation set; The second point cloud corresponding to the training operation is found by finding the second point cloud dataset. The second point cloud is input into the point cloud segmentation network to obtain the prediction result corresponding to the training operation output by the point cloud segmentation network. The loss between the prediction result corresponding to the training operation output by the point cloud segmentation network and the annotation result of the second point cloud corresponding to the training operation is calculated. The loss is used to update the parameters of the point cloud segmentation network. The point cloud segmentation network is evaluated by finding the second point cloud corresponding to the test operation from the second point cloud dataset, and the evaluation result corresponding to the test operation is obtained. After training the point cloud segmentation network, the following steps are also included: Based on the evaluation results of each round of training, the optimal point cloud segmentation network is determined as the three-dimensional point cloud voxel segmentation model.

4. The method according to any one of claims 1 to 3, characterized in that, The step of determining the calibration parameters between the camera and the LiDAR based on the pixel feature set and the voxel feature set includes: The calibration parameters are obtained by processing the pixel feature set and the voxel feature set using a multi-point perspective pose solving method, preset camera intrinsic parameters and preset distortion coefficients.

5. The method according to any one of claims 1 to 3, characterized in that, After determining the calibration parameters between the camera and the LiDAR based on the pixel feature set and the voxel feature set, the method further includes: Based on the calibration parameters, the point cloud data collected in at least one pose in the first point cloud dataset are divided into different colors according to distance to obtain the color-coded point cloud data. The color-coded point cloud data is projected onto image data acquired in at least one pose in the first image dataset to obtain image data of the projected point cloud data.

6. The method according to claim 5, characterized in that, Image data includes image colors; After determining the calibration parameters between the camera and the LiDAR based on the pixel feature set and the voxel feature set, the method further includes: Based on the calibration parameters and the point cloud data collected under the at least one posture, the image color corresponding to the point cloud data collected under the at least one posture is extracted from the image data collected under the at least one posture. The image color is projected onto the point cloud data acquired in at least one pose to obtain point cloud data with the image color.

7. A joint calibration device for a lidar and a camera, characterized in that, The device includes: The acquisition module is used to acquire a first image dataset and a first point cloud dataset of the calibration board. The first image dataset is acquired by the camera in at least one pose of the calibration board, and the first point cloud dataset is acquired by the lidar in at least one pose of the calibration board. The first recognition module is used to process the first image dataset using a two-dimensional image semantic segmentation model to identify the pixel feature set of the calibration board in the image; The second recognition module is used to process the first point cloud dataset using a three-dimensional point cloud voxel segmentation model to identify the voxel feature set of the calibration board in the point cloud. The determination module is used to determine the calibration parameters between the camera and the lidar based on the pixel feature set and the voxel feature set.

8. A computer device, characterized in that, include: A memory and a processor are communicatively connected, the memory storing computer instructions, and the processor executing the computer instructions to perform the joint calibration method of the lidar and camera as described in any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions for causing the computer to perform the joint calibration method of the lidar and camera as described in any one of claims 1 to 6.

10. A computer program product, characterized in that, It includes computer instructions for causing a computer to perform the joint calibration method of the lidar and camera as described in any one of claims 1 to 6.

Images