A method and device for calibrating and correcting external parameters of a camera and a point cloud sensor

Through the visual basic large model and unsupervised segmentation algorithm, and the loss function is optimized by combining typical reference objects, the problems of low generalization and insufficient targeting of foreign parameter calibration in the existing technology are solved, and high-precision Camera and point cloud sensor external parameter calibration, suitable for different autonomous driving systems.

CN117808890BActive Publication Date: 2024-11-26HUIXI INTELLIGENT TECH (SHANGHAI) CO LTD
View PDF 9 Cites 0 Cited by

Patent Information

Application Number
CN202311780521.9
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2023-12-22
Publication Date
2024-11-26
Estimated Expiration
2043-12-22

AI Technical Summary

Technical Problem

The external parameter calibration methods of existing cameras and point cloud sensors are low in generalization, and cannot adapt to different cameras and lidar configurations, and lack targeting, resulting in insufficient external parameter calibration accuracy, affecting the training and deployment of autonomous driving models.

Method used

Image segmentation and point cloud segmentation are used to adopt visual basic model, and the image and point cloud are aligned by timestamps and camera external parameters, and point cloud clusters are formed using unsupervised segmentation and clustering algorithms, and combined with typical reference objects to optimize the loss function to realize the image and Alignment and classification of point cloud targets to improve the accuracy of external parameters alignment.

Benefits of technology

It realizes the external parameter calibration of cameras and point cloud sensors with high generalization and target parameters, and can adapt to cameras and lidar configurations of different autonomous driving systems at no cost, improves the external parameter accuracy and data quality, and reduces the new annotated data. need.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN117808890B_ABST
    Figure CN117808890B_ABST
Patent Text Reader

Abstract

The present invention relates to a method for calibrating and correcting external parameters of a camera and a point cloud sensor, comprising the following steps: S1, inputting an image captured by a camera and a single frame of point cloud data collected by a point cloud sensor; S2, aligning the image and the point cloud by timestamp and camera external parameters; 3, image segmentation; S4, point cloud segmentation; S5, point cloud tracking / fusion / matching; S6, using the image segmentation results and the point cloud segmentation results through the same category distance screening method, aligning and matching the image target and the point cloud target by distance and angle parameters to achieve image target and point cloud target instance alignment, wherein the image target is taken as the true value and the point cloud target is taken as the prediction result to obtain the point cloud image projection; S7, in the successfully matched image target and point cloud target instance, by analyzing the consistency of the point cloud image projection of the typical reference object and the ordinary reference object, the cost function is optimized. The beneficial effects are good generalization and strong pertinence.
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] The present invention relates to the technical field of computer image processing, and in particular to a method and device for calibrating and correcting external parameters of a camera and a point cloud sensor. [Background technology]

[0002] At present, the data of autonomous driving needs to be collected by multiple sensors. The accuracy of external parameter annotation between multiple sensors affects the quality of training data, especially for cameras and point cloud sensors. The low accuracy cannot meet the data requirements of model training, which directly leads to the reduction of online model accuracy and the inability to meet product requirements. In fact, there are many reasons for the insufficient accuracy of sensor annotation, such as sensor jitter of the collection vehicle, thermal expansion and contraction of the sensor bracket, and inaccurate factory calibration.

[0003] The commonly used external parameter calibration and correction scheme for existing cameras and point cloud sensors is to extract targets in camera imaging through deep learning methods, extract targets in point clouds through unsupervised or supervised segmentation methods, and optimize the consistency of point clouds in the projection area by projecting point clouds onto images. The existing scheme has the following problems: 1. Low generalization. Based on the training model of a specific camera, it cannot be adapted as a general model to other camera imaging; 2. No specificity. The existing scheme extracts general targets in the camera, but does not optimize key targets in the scene. The algorithm often focuses on some abnormal areas, resulting in reduced optimization accuracy.

[0004] The following relevant patents were retrieved by searching the Chinese patent database.

[0005] Related search result 1: Patent document, CN114898144A, name, an automatic alignment method based on camera and millimeter wave radar data, first obtain point cloud data through millimeter wave radar scanning, obtain visual image information through camera, then process based on DLA-34 network structure to obtain image feature map, further map effective millimeter wave point cloud to image feature map, then set the maximum number of targets and calculate point cloud feature vector, process through attention fusion module, obtain radar and image fusion feature map after data alignment; finally, send image fusion feature map to RetinaNet target detection subnetwork for processing, obtain positioning and classification results of targets in image, and complete data automatic alignment. Technical points: 1. It is only for processing millimeter wave radar data, and cannot process radar point cloud data universally. It is not universal and cannot adapt to all radar point cloud data and camera image data; 2. Use fusion feature map for positioning and classification, and do not add any weight restrictions to all classified targets, and cannot effectively utilize each classified target.

[0006] Related search result 2: Patent document, CN115393407A, name, external parameter calibration method and device for images and storage medium, the method includes: obtaining camera images and radar images corresponding to the external parameters to be calibrated. Convert the camera image to a preset standard image. The acquisition position of the camera image is consistent with the acquisition position of the standard image. Perform external parameter calibration according to the standard image and the radar image. It is suitable for online external parameter calibration of camera graphics collected by the camera and radar images collected by the laser radar, and the converted standard image and radar image are subjected to external parameter calibration processing to obtain calibrated external parameters matching the camera image and the radar image. By introducing the standard image, when the external parameter calibration is performed by combining the standard image with the radar image, it is only necessary to convert the camera image collected by the camera into the standard image, and there is no need to update and adjust the external parameter calibration algorithm of the standard image and the radar image, thereby improving the scope of application of the external parameter calibration. Technical points: 1. Convert the camera image to a standard image, and uniformly and the radar point cloud are mapped and calculated at each pixel point, but the setting of the standard image is a unified preset value and cannot be used for supervision.

[0007] In view of the technical problems of low generalization and lack of specificity in existing solutions, the present invention makes technical improvements to the camera and point cloud sensor extrinsic parameter calibration and correction method and device. [Summary of the invention]

[0008] The purpose of the present invention is to provide a camera and point cloud sensor extrinsic parameter calibration and correction method with good generalization and strong pertinence.

[0009] To achieve the above object, the technical solution adopted by the present invention is a method for calibrating and correcting external parameters of a camera and a point cloud sensor, comprising the following steps:

[0010] S1, input the image captured by the camera and the single frame data of the point cloud collected by the point cloud sensor;

[0011] S2, align the image and point cloud by timestamp and camera extrinsics;

[0012] S3. In the image, a visual basic large model segmentation scheme with good generalization is used to process the segmented image targets as image segmentation results. The image targets include typical reference objects and ordinary reference objects. The image targets have category labels and instance labels. The selection of the visual basic large model takes the segmentation accuracy as the primary reference indicator.

[0013] S4. In the point cloud, an unsupervised segmentation scheme is used to group the points of the point cloud according to feature similarity through a clustering algorithm to form a point cloud cluster to obtain the segmented point cloud target as the point cloud segmentation result. The point cloud target has an instance label but no clear category label.

[0014] S5. Perform time-series tracking and correlation fusion on the point cloud segmentation results, fuse the segmentation results of multiple time steps, obtain the comprehensive information of the point cloud segmentation results in the entire time series, and classify and identify the point cloud targets based on the fused comprehensive information to obtain the category label;

[0015] S6. The image segmentation results and the point cloud segmentation results are screened by the same category distance, and the distance and angle parameters of the image target and the point cloud target are aligned and matched to achieve the alignment of the image target and the point cloud target instance, wherein the image target is taken as the true value and the point cloud target is taken as the prediction result to obtain the point cloud image projection;

[0016] S7, correcting the camera and point cloud sensor external parameters through point cloud image projection consistency; specifically,

[0017] In the successfully matched image target and point cloud target instances, the following cost function is optimized by analyzing the point cloud image projection consistency of typical reference objects and ordinary reference objects:

[0018]

[0019] represents the loss function corresponding to the kth typical reference object, It represents the loss function corresponding to the i-th common reference object. α and β are hyperparameters, and different weights are adjusted according to different reference objects. n is the number of typical reference objects, and m is the number of common reference objects.

[0020] Preferably, the loss function and The calculation method is the same as:

[0021]

[0022] Among them, w r , w n , w s , w d is an adjustable weight, L R , L N , L S , L l represents the point cloud reflection value, point cloud normal vector, point cloud segmentation loss and point cloud deviation loss, P i Represents the point set of the i-th reference object.

[0023] Preferably, the point cloud segmentation loss and the point cloud deviation loss are calculated as follows:

[0024]

[0025]

[0026] Among them, LS is the point cloud segmentation loss, L d is the point cloud deviation loss, y is the true value label, is the predicted label, and n is the number of reference samples.

[0027] Preferably, static plane / aerial elements are selected to determine reference objects according to the frequency of occurrence of image objects; typical reference objects refer to static objects with relatively clear boundaries and fixed shapes among image objects.

[0028] Preferably, step S1: the image includes videos captured by multiple different cameras or a single camera over a period of time, and the point cloud includes multiple lidars or a single lidar time-series point cloud, wherein the time-series point cloud is a set of many points, each point having spatial position information and other attributes including reflectivity.

[0029] Preferably, step S2: for each timestamp, select the closest timestamp as the basis for matching through their difference, find the closest image and point cloud data, and perform one-to-one matching and alignment of the point cloud and the image in combination with the camera extrinsic parameters. For the matched image and point cloud data, perform a coordinate system transformation according to the camera extrinsic parameters, and transform the point cloud data from the lidar coordinate system to the camera coordinate system.

[0030] Preferably, the visual basis large model uses a public large model of the Grounded-SAM type.

[0031] Another object of the present invention is to provide a camera and point cloud sensor extrinsic parameter calibration and correction device with good generalization and strong pertinence.

[0032] To achieve the above-mentioned another purpose, the technical solution adopted by the present invention is a camera and point cloud sensor extrinsic parameter calibration and correction device, including a camera and a laser radar, which is used to execute the above-mentioned camera and point cloud sensor extrinsic parameter calibration and correction method.

[0033] The beneficial effects of the camera and point cloud sensor external parameter calibration and correction method and device of the present invention are as follows: 1. The zero-sample generalization capability of the universal visual basic large model is reused, and no new annotation data is required, and it can be generalized to the camera and lidar configuration schemes of different autonomous driving systems at no cost; 2. The reference object selection is targeted. In the process of optimizing the loss function, a larger weight is assigned to typical reference objects in the scene, such as traffic lights, signs, and lane lines, and the weight of non-critical targets in other scenes is reduced, thereby improving the alignment accuracy of point clouds and images, thereby improving the accuracy of external parameters.

Brief Description of the Drawings

[0034] Figure 1 The present invention is a flow chart of a method for extrinsic calibration and correction of cameras and point cloud sensors. [Specific implementation method]

[0035] The features and exemplary embodiments of various aspects of the present invention will be described in detail below. In the detailed description below, many specific details are proposed to provide a comprehensive understanding of the present invention. But it is obvious to those skilled in the art that the present invention can be implemented without the need for some of these specific details. The following description of the embodiments is only to provide a better understanding of the present invention by illustrating examples of the present invention. The present invention is by no means limited to any specific configuration and algorithm proposed below, but covers any modification, replacement and improvement of elements, parts and algorithms without departing from the concept of the present invention. In the accompanying drawings and the following description, known structures and technologies are not shown to avoid unnecessary ambiguity to the present invention.

[0036] Example

[0037] This embodiment implements a method for calibrating and correcting external parameters of a camera and a point cloud sensor.

[0038] In order to solve the technical problems of low generalization and lack of targeting in the prior art, this embodiment proposes the use of a large visual basic model to enable the solution to have the ability to generalize to different sensor configuration solutions with zero samples. At the same time, the typical matching targets detected by the large visual basic model are combined with typical obstacles to uniformly optimize the external parameters, and the performance indicators are better than the traditional solutions. The so-called large visual model, in simple terms, is to use a large amount of data and algorithms to build a visual system with high recognition, analysis and expression capabilities. It can imitate the working principle of the human visual system, process and analyze images, and continuously optimize its own algorithms through machine learning. The so-called basic model is defined as "a model trained on large-scale data through self-supervision or semi-supervision, which can adapt to multiple other downstream tasks."

[0039] Due to sensor jitter, thermal expansion and contraction of the sensor bracket, etc., the point cloud and camera projection may not be aligned in the collected data. Such data is of low quality and has three impacts on downstream:

[0040] 1. If low-quality data is allowed to flow through the annotation process, it will need to be reworked in a timely manner after being discovered, which will seriously affect the efficiency of the data link;

[0041] 2. If low-quality data is forcibly labeled and fed into online model training, model performance will deteriorate;

[0042] 3. If the unaligned data is discarded directly, the data utilization rate will be extremely low, wasting a lot of collection, storage and processing resources.

[0043] One method is to use manual quality inspection to filter out the data that cannot be aligned. The specific plan is to extract several segments of projection data and visually check the differences between the point cloud and image of the same target. However, there are individual differences in manual quality inspection, and it is impossible to accurately judge the deviation through visual inspection. At the same time, manual efficiency is also very low, which cannot be expanded to subsequent large-scale mass production scenarios, and a lot of costs are added.

[0044] Another solution is to use automatic optimization algorithms to align key visual and point cloud targets, build loss functions, and optimize camera and lidar external parameters. However, this type of solution has two problems: first, the visual model segmentation is not generalizable, and the image detection and segmentation performance of other data sets is poor; second, the visual model segmentation generally has no categories, resulting in key reference obstacles being submerged in a large number of non-key obstacle losses in the scene, resulting in low optimization accuracy.

[0045] The method of this embodiment is based on the general visual segmentation model and the point cloud segmentation results, and corrects the sensor extrinsic parameters through the consistency of the point cloud image projection. It has zero-sample generalization capability, does not require new labeled data, and can be applied to the camera and lidar configuration schemes of different autonomous driving systems at no cost. Secondly, the reference object selection is targeted. In the process of optimizing the loss function, typical reference objects in the scene, such as traffic lights, signs, and lane lines, are given a larger weight, and the weight of non-critical targets in the scene is reduced to improve the alignment accuracy. Finally, the camera and lidar extrinsic parameters are repaired to obtain high-quality data.

[0046] Figure 1 This is a flow chart of the calibration and correction method of the external parameters of the camera and point cloud sensor. Figure 1 As shown, the method of this embodiment includes the following steps:

[0047] 1. Input image and point cloud single frame data, where the image includes videos taken by multiple cameras over a period of time, and the point cloud includes multiple LiDAR time-series point clouds, which are a collection of many points, each with spatial position information and other attributes including reflectivity. In practice, the input can also be a single camera or a single LiDAR.

[0048] 2. Align the image and point cloud through timestamps and camera extrinsics. For each timestamp, select the closest timestamp as the basis for matching through their difference, find the closest camera image and point cloud data, and use the camera extrinsics to perform one-to-one matching and alignment of point cloud and image. For the matched image and point cloud data, convert the coordinate system according to the camera extrinsics, and convert the point cloud data from the lidar coordinate system to the camera coordinate system.

[0049] 3. In the image, the visual basic large model segmentation solution is used to process static and dynamic targets in the scene. The segmentation results are labeled with categories and instances. The selection of the visual basic large model takes the segmentation accuracy as the primary reference indicator.

[0050] 4. In the point cloud, the unsupervised segmentation scheme is also used to obtain the segmented target. The segmentation result does not have a clear category label, but the points are grouped according to feature similarity through the clustering algorithm to form point cloud clusters.

[0051] 5. Perform time-series tracking and correlation fusion on the point cloud segmentation target, and fuse the segmentation results of multiple time steps to obtain comprehensive information of the target in the entire time series. Based on the fused information, the target can be classified and identified to obtain the category label.

[0052] 6. Align the matching pairs of image visual results and point cloud segmentation results by filtering the distance between the same categories.

[0053] 7. Match the target results on both sides of the point cloud target on the image, and align parameters such as distance and angle. The visual result is regarded as the true value and the point cloud result is regarded as the predicted result.

[0054] 8. Among the successfully matched targets, by analyzing the consistency between typical reference objects and ordinary reference objects, the following cost function is optimized:

[0055]

[0056] represents the loss corresponding to the kth key reference target, It represents the loss corresponding to the i-th common reference object. α and β are hyperparameters, and different weights are adjusted according to different reference objects.

[0057] Loss Function and The calculation method is the same as:

[0058]

[0059] where w r , w n , w s , w d is an adjustable weight, L R , L N , L S , L l Represents the point cloud reflection value, point cloud normal vector, point cloud segmentation loss and point cloud deviation loss, P i Represents the point set of the i-th reference object.

[0060] The segmentation loss and bias loss are calculated as follows:

[0061]

[0062]

[0063] Among them, L S is the segmentation loss, L d is the bias loss, y is the true label, is the predicted label and n is the number of samples.

[0064] Typical reference target selection:

[0065] Typical reference objects refer to static targets in the image, which have relatively clear boundaries and fixed shapes.

[0066]

[0067]

[0068] From the frequency of reference objects, static plane / air elements are selected. Specifically, lane lines and traffic lights appear frequently in various scenes and are suitable as reference objects. Dynamic reference objects appear frequently and can also be used in the solution of this embodiment after the introduction of dynamic target speed estimation.

[0069] This embodiment does not limit the point cloud segmentation algorithm, and any public point cloud segmentation algorithm may be used, such as a traditional clustering method.

[0070] This embodiment does not limit the use of any visual basic large model, as long as the model has good generalization, for example, a public large model such as Grounded-SAM can be used. Grounded-SAM: A fully automated visual tool that only uses the mouth but not the hands.

[0071] Those skilled in the art will appreciate that all or part of the steps to implement the above embodiments may be accomplished by hardware, or may be accomplished by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, wherein the storage medium may be a magnetic disk, an optical disk, a read-only memory (ROM) or a random access memory (RAM), etc.

[0072] The above is only a preferred embodiment of the present invention. It should be pointed out that ordinary technicians in this technical field can make several improvements and supplements without departing from the principle of the present invention. These improvements and supplements should also be regarded as the scope of protection of the present invention.

Claims

1. A method for calibrating and correcting external parameters of a camera and a point cloud sensor, characterized in that The following steps are involved: S1, input the image captured by the camera and the single frame data of the point cloud collected by the point cloud sensor; S2, align the image and point cloud by timestamp and camera extrinsics; S3. In the image, the segmented image target is processed by the visual basic large model segmentation scheme as the image segmentation result, the image target includes typical reference objects and ordinary reference objects, and the image target has category labels and instance labels, wherein the selection of the visual basic large model takes the segmentation accuracy as the primary reference indicator; the typical reference object refers to a static target with a clear boundary and a fixed shape in the image target; S4. In the point cloud, an unsupervised segmentation scheme is used to group the points of the point cloud according to feature similarity through a clustering algorithm to form a point cloud cluster to obtain the segmented point cloud target as the point cloud segmentation result. The point cloud target has an instance label but no clear category label. S5. Perform time-series tracking and correlation fusion on the point cloud segmentation results, fuse the segmentation results of multiple time steps, obtain the comprehensive information of the point cloud segmentation results in the entire time series, and classify and identify the point cloud targets based on the fused comprehensive information to obtain the category label; S6. The image segmentation results and the point cloud segmentation results are screened by the same category distance, and the distance and angle parameters of the image target and the point cloud target are aligned and matched to achieve the alignment of the image target and the point cloud target instance, wherein the image target is taken as the true value and the point cloud target is taken as the prediction result to obtain the point cloud image projection; S7, correcting the camera and point cloud sensor external parameters through point cloud image projection consistency; specifically, In the successfully matched image target and point cloud target instances, the following cost function is optimized by analyzing the point cloud image projection consistency of typical reference objects and ordinary reference objects: represents the loss function corresponding to the kth typical reference object, It represents the loss function corresponding to the i-th common reference object. α and β are hyperparameters, and different weights are adjusted according to different reference objects. n is the number of typical reference objects, and m is the number of common reference objects.

2. The method for calibrating and correcting external parameters of a camera and a point cloud sensor according to claim 1, characterized in that Loss Function and The calculation method is the same as: Among them, w r , w n , w s , w d is an adjustable weight, L R , L N , L S , L d represents the point cloud reflection value, point cloud normal vector, point cloud segmentation loss and point cloud deviation loss, P i Represents the point set of the i-th reference object.

3. A camera and point cloud sensor external parameter calibration correction method according to claim 2, characterized in that The point cloud segmentation loss and point cloud deviation loss are calculated as follows: Among them, L S is the point cloud segmentation loss, L d is the point cloud deviation loss, y is the true value label, is the predicted label, and n is the number of reference samples.

4. The method for calibrating and correcting external parameters of a camera and a point cloud sensor according to claim 1, characterized in that: According to the frequency of occurrence of image targets, static plane / aerial elements are selected to determine the reference objects.

5. The method for calibrating and correcting external parameters of a camera and a point cloud sensor according to claim 1, characterized in that Step S1: The image includes images captured by multiple different cameras or a single camera within a period of time, and the point cloud includes multiple laser radars or a single laser radar time-series point cloud. The time-series point cloud is a collection of many points, each of which has spatial position information and other attributes including reflectivity.

6. The method for calibrating and correcting external parameters of a camera and a point cloud sensor according to claim 1, characterized in that Step S2: For each timestamp, select the closest timestamp as the basis for matching through their difference, find the closest image and point cloud data, and perform one-to-one matching and alignment of the point cloud and image in combination with the camera extrinsic parameters. For the matched image and point cloud data, perform coordinate system conversion according to the camera extrinsic parameters, and convert the point cloud data from the lidar coordinate system to the camera coordinate system.

7. The method for calibrating and correcting external parameters of a camera and a point cloud sensor according to claim 1, characterized in that: The visual basis large model uses a public large model of the Grounded-SAM class.

8. A camera and point cloud sensor external parameter calibration and correction device, comprising a camera and a laser radar, characterized in that: Used to execute a camera and point cloud sensor external parameter calibration and correction method as described in any one of claims 1 to 7.

Citation Information

Patent Citations

  • Automatic alignment method based on camera and millimeter wave radar data

    CN114898144A

  • External parameter calibration method and device for image, equipment and storage medium

    CN115393407A

  • External parameter calibration method and device and electronic equipment

    CN111383279A

  • Target ranging system and method based on image and laser radar fusion

    CN112396650A

  • Automatic driving moving obstacle segmentation method based on laser radar

    CN115830265A