External parameter calibration method and system, computer equipment, storage medium and program product

Abstract

The invention relates to the technical field of automatic driving, and discloses an external parameter calibration method and system, computer equipment, a storage medium and a program product. The method is applied to data processing equipment in an external parameter calibration system. The method comprises the following steps: acquiring each frame of image data acquired by each camera in a multi-camera system and each frame of laser point cloud acquired by a laser radar; performing depth estimation on each frame of image data acquired by each camera to obtain each frame of depth map corresponding to each frame of image data; matching each frame of laser point cloud with each frame of depth map of each camera according to the frame number to obtain a laser point cloud data set and a depth map data set matched with each frame; establishing an external parameter calibration function based on mutual information between the matched laser point cloud data set and depth map data set of each frame; and solving an external parameter optimal solution which enables the external parameter calibration function to be maximized, and taking the external parameter optimal solution as an external parameter calibration result. According to the scheme, when the external parameter calibration function is achieved, convenience and high accuracy are achieved.

Description

Technical Field

[0001] The present invention relates to the field of autonomous driving technology, and in particular to an external parameter calibration method, system, computer equipment, storage medium and program product. Background Art

[0002] LiDAR is a remote sensing technology that uses laser ranging technology to obtain information about target objects or terrain. A surround view camera is a camera that can capture panoramic images and is usually equipped with multiple lenses to achieve all-round photography of the surrounding environment. LiDAR and surround view cameras are two commonly used sensors in autonomous driving. In order to better integrate the information of the two, external parameter calibration is required to determine the relative transformation relationship between the two. External parameter calibration refers to determining the relative position and direction relationship between different sensors so that the data they obtain can be mapped to the same coordinate system.

[0003] In the related art, reference objects such as calibration plates are used for offline extrinsic parameter calibration. However, this method relies on the professional use of calibration plates, is not convenient enough and has low accuracy. Alternatively, online calibration is achieved by matching line features extracted from camera depth maps and lidar depth maps. However, this method is not suitable for scenes with complex or sparse textures. Alternatively, a neural network is used to output extrinsic parameters end-to-end. However, this method relies on a large amount of labeled data for training.

[0004] Therefore, a convenient and accurate external parameter calibration method is urgently needed. Summary of the invention

[0005] In view of this, an object of the present invention is to provide an external parameter calibration method, system, computer device, storage medium and program product to solve the problem that external parameter calibration is not convenient and accurate enough.

[0006] In a first aspect, the present invention provides an external parameter calibration method, which is applied to a data processing device in an external parameter calibration system; the external parameter calibration system also includes a laser radar and a multi-camera system installed on a target vehicle; the method includes: Acquire each frame of image data collected by each camera in the multi-camera system and each frame of laser point cloud collected by the laser radar; Depth estimation is performed on each frame of image data collected by each camera to obtain a depth map of each frame corresponding to each frame of image data; Match each frame of laser point cloud with each frame of depth map of each camera according to the number of frames to obtain a laser point cloud data set and a depth map data set that match each frame; An external parameter calibration function is established based on the mutual information between the laser point cloud data set and the depth map data set that match each frame; The optimal solution of the extrinsic parameters that maximizes the extrinsic calibration function is obtained as the extrinsic parameter calibration result between the lidar and the multi-camera system.

[0007] In an optional implementation, the performing depth estimation on each frame of image data collected by each camera to obtain each frame of depth map corresponding to each frame of image data includes: Determine the target camera coordinate system; Performing external parameter calibration on the coordinate system of each camera in the multi-camera system and the coordinate system of the target camera respectively; Depth estimation is performed on each frame of image data collected by each camera to obtain a depth map to be converted for each frame of image data collected by each camera; The depth maps to be converted of each frame corresponding to each camera are converted to the target camera coordinate system to obtain the depth maps of each frame corresponding to each frame of image data collected by each camera.

[0008] In an optional implementation, matching each frame of laser point cloud with each frame of depth map of each camera according to the number of frames to obtain a laser point cloud data set and a depth map data set matching each frame includes: Group the laser point cloud and the depth map of each camera by frame number; For each group, the laser point cloud is converted to the target camera coordinate system to obtain the laser point cloud after coordinate conversion; Based on the internal parameters of each camera, the laser point cloud after coordinate transformation is projected onto the pixel plane of each camera to obtain the pixel points corresponding to the laser point cloud after coordinate transformation; The pixel points corresponding to the laser point cloud after coordinate transformation are matched with the pixel points in the depth map to obtain a matching laser point cloud data set and depth map data set.

[0009] In an optional implementation, the establishing of an external parameter calibration function based on the mutual information between the laser point cloud data set and the depth map data set matching each frame includes: Obtain the mutual information between the laser point cloud data set and the depth map data set that match each frame respectively; The mutual information between the laser point cloud data set and the depth map data set that match each frame is summed to establish an extrinsic calibration function.

[0010] In an optional implementation, respectively obtaining the mutual information between the laser point cloud data set and the depth map data set that match each frame includes: For each frame, respectively obtain a normalized histogram of the laser point cloud data set and a normalized histogram of the depth map data set of the matching laser point cloud data set; The laser point cloud marginal probability of the normalized histogram of the laser point cloud, the depth map marginal probability of the normalized histogram of the depth map, and the joint probability between the normalized histogram of the laser point cloud and the normalized histogram of the depth map are respectively calculated; Based on the laser point cloud marginal probability, the depth map marginal probability and the joint probability, the mutual information between the matched laser point cloud data set and the depth map data set is obtained.

[0011] In an optional implementation, the mutual information between the laser point cloud data set and the depth map data set matching each frame is obtained by the following formula:

[0012] in, represents the mutual information of the i-th frame, Represents the i-th frame depth map data group, represents the i-th frame laser point cloud data set, represents the entropy of the depth map data set of the i-th frame, represents the entropy of the laser point cloud data set of the i-th frame, It represents the joint entropy of the i-th frame depth map data group and the i-th frame laser point cloud data group.

[0013] In a second aspect, the present invention provides an external parameter calibration system, the system comprising: Laser radar, installed on the target vehicle, is used to collect laser point clouds of each frame; A multi-camera system, installed on the target vehicle, is used to collect image data for each frame; Data processing equipment, including: A data acquisition module, used to acquire each frame of image data collected by each camera in the multi-camera system and each frame of laser point cloud collected by the laser radar; The depth estimation module is used to perform depth estimation on each frame of image data collected by each camera to obtain a depth map of each frame corresponding to each frame of image data; A matching module is used to match each frame of laser point cloud with each frame of depth map of each camera according to the number of frames to obtain a laser point cloud data group and a depth map data group that match each frame; A function building module, used to build an external parameter calibration function based on the mutual information between the laser point cloud data set and the depth map data set that match each frame; The function solving module is used to obtain the optimal solution of the external parameters that maximizes the external parameter calibration function as the external parameter calibration result between the laser radar and the multi-camera system.

[0014] In a third aspect, the present invention provides a computer device, comprising: a memory and a processor, the memory and the processor being communicatively connected to each other, the memory storing computer instructions, and the processor executing the external parameter calibration method of the first aspect or any corresponding embodiment thereof by executing the computer instructions.

[0015] In a fourth aspect, the present invention provides a computer-readable storage medium having computer instructions stored thereon, the computer instructions being used to enable a computer to execute the external parameter calibration method of the first aspect or any corresponding embodiment thereof.

[0016] In a fifth aspect, the present invention provides a computer program product, comprising computer instructions, wherein the computer instructions are used to enable a computer to execute the external parameter calibration method of the first aspect or any corresponding embodiment thereof.

[0017] The technical solution provided by the present invention may include the following beneficial effects: The extrinsic parameter calibration method provided by the present invention first obtains each frame of image data collected by each camera in a multi-camera system and each frame of laser point cloud collected by a laser radar, then respectively estimates the depth of each frame of image data collected by each camera to obtain each frame of depth map corresponding to each frame of image data, then matches each frame of laser point cloud with each frame of depth map of each camera according to the number of frames, obtains a laser point cloud data group and a depth map data group matching each frame, and then establishes an extrinsic parameter calibration function based on the mutual information between the laser point cloud data group and the depth map data group matching each frame, and finally obtains the optimal extrinsic parameter solution that maximizes the extrinsic parameter calibration function as the extrinsic parameter calibration result between the laser radar and the multi-camera system. By matching the depth map corresponding to the image data with the laser point cloud, and determining the most accurate dependency relationship between the laser point cloud data group and the depth map data group based on the mutual information between the two, the optimal extrinsic parameter is obtained, and a convenient and highly accurate extrinsic parameter calibration function is realized without the need for a calibration plate, a scene with suitable texture, or a large amount of training data. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0019] Figure 1 is a flow chart of an external parameter calibration method according to an embodiment of the present invention; Figure 2 is a flow chart of another external parameter calibration method according to an embodiment of the present invention; Figure 3 is a schematic diagram of coordinate conversion of a multi-camera system and a laser radar according to this embodiment; Figure 4 is a structural block diagram of an external parameter calibration system according to an embodiment of the present invention; Figure 5 It is a schematic diagram of the hardware structure of a computer device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0020] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0021] According to an embodiment of the present invention, an embodiment of an external parameter calibration method is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although the logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0022] In this embodiment, an external parameter calibration method is provided, which is applied to a data processing device in an external parameter calibration system, wherein the external parameter calibration system also includes a laser radar and a multi-camera system installed on a target vehicle. The data processing device may be a desktop computer, a laptop computer, or the like. Figure 1 is a flow chart of an external parameter calibration method according to an embodiment of the present invention. Figure 1 As shown, the process includes the following steps: Step S101, obtaining each frame of image data collected by each camera in a multi-camera system and each frame of laser point cloud collected by a laser radar.

[0023] The LiDAR can be installed on the roof of the target vehicle to provide a 360-degree field of view. The cameras in the multi-camera system can be installed on the front, rear, side and roof of the target vehicle, and the appropriate camera type can be selected according to the needs, such as wide-angle, fisheye, high-definition, etc., to meet the field of view and the quality of the collected images.

[0024] When performing external parameter calibration, the multi-camera system is used to collect image data around the target vehicle, and the laser radar is used to collect point cloud data around the target vehicle, so as to obtain each frame of image data collected by each camera in the multi-camera system and each frame of laser point cloud collected by the laser radar.

[0025] Step S102 , performing depth estimation on each frame of image data collected by each camera to obtain a depth map of each frame corresponding to each frame of image data.

[0026] For the target camera, a deep learning algorithm is used to estimate the depth of each frame of image data collected by the target camera, and the depth of each frame of image data is converted into a depth map of each frame under the camera system according to the external parameters of the camera-to-camera system. Among them, the depth map is a two-dimensional array, each element of which corresponds to a pixel in the image, and the value of each pixel represents the distance from the point to the camera. The external parameters of the camera-to-camera system describe the position and orientation of the camera relative to the world coordinate system. Among them, the target camera is one of the cameras in the multi-camera system.

[0027] Step S103, matching each frame of the laser point cloud with each frame of the depth map of each camera according to the number of frames, to obtain a laser point cloud data set and a depth map data set that match each frame.

[0028] By matching each frame of the laser point cloud with each frame of the depth map of each camera according to the frame number, the laser point cloud and depth map at the same time can be obtained, so as to obtain the relative transformation relationship between the laser point cloud and the depth map, and obtain the external parameters between the laser radar and the multi-camera system.

[0029] Step S104: establishing an external parameter calibration function based on the mutual information between the laser point cloud data set and the depth map data set that match each frame.

[0030] Mutual information is used to measure the dependency between two random variables and quantify the amount of information provided by one variable about another variable. Mutual information can reduce the uncertainty of another variable under the premise of knowing one variable. By obtaining the mutual information between the laser point cloud data set and the depth map data set that match each frame, an external parameter calibration function can be established to indicate the dependency between the laser point cloud data set and the depth map data set.

[0031] Step S105, obtaining an optimal solution of the extrinsic parameters that maximizes the extrinsic parameter calibration function as the extrinsic parameter calibration result between the laser radar and the multi-camera system.

[0032] By maximizing the mutual information between the laser point cloud data set and the depth map data set, the optimal extrinsic parameter solution that maximizes the extrinsic parameter calibration function can be obtained as the extrinsic parameter calibration result, so as to select the most accurate dependency relationship between the laser point cloud data set and the depth map data set, so that the data alignment degree between the laser radar and the multi-camera system is the highest and the data alignment effect is the best.

[0033] The external parameter calibration method provided in this embodiment first obtains each frame of image data collected by each camera in the multi-camera system and each frame of laser point cloud collected by the laser radar, then respectively estimates the depth of each frame of image data collected by each camera to obtain each frame of depth map corresponding to each frame of image data, then matches each frame of laser point cloud with each frame of depth map of each camera according to the number of frames, obtains each frame of laser point cloud data group and depth map data group matching, and then establishes an external parameter calibration function based on the mutual information between each frame of laser point cloud data group and depth map data group matching, and finally obtains the external parameter optimal solution that maximizes the external parameter calibration function as the external parameter calibration result between the laser radar and the multi-camera system. By matching the depth map corresponding to the image data with the laser point cloud, and determining the most accurate dependency relationship between the laser point cloud data group and the depth map data group based on the mutual information between the two, the optimal external parameter is obtained, and a convenient and highly accurate external parameter calibration function is realized without the need for a calibration plate, a scene with suitable texture, or a large amount of training data.

[0034] In this embodiment, an external parameter calibration method is provided, which is applied to a data processing device in an external parameter calibration system, wherein the external parameter calibration system also includes a laser radar and a multi-camera system installed on a target vehicle. The data processing device may be a desktop computer, a laptop computer, or the like. Figure 2 is a flow chart of an external parameter calibration method according to an embodiment of the present invention. Figure 2 As shown, the process includes the following steps: Step S201, obtaining each frame of image data collected by each camera in the multi-camera system and each frame of laser point cloud collected by the laser radar.

[0035] Figure 3 This is a schematic diagram of the coordinate transformation of the multi-camera system and the laser radar according to this embodiment. The external parameters (extrinsic parameters) of the multi-camera system and the laser radar refer to the parameters required for the Euclidean transformation from the laser radar coordinate system to the multi-camera system coordinate system, and the rotation matrix from the radar coordinate system to the camera coordinate system. and the displacement of the radar in the camera coordinate system express.

[0036] Step S202 , performing depth estimation on each frame of image data collected by each camera to obtain a depth map of each frame corresponding to each frame of image data.

[0037] Specifically, the above step S202 includes: Step S2021, determine the target camera coordinate system.

[0038] The target camera coordinate system can be a world coordinate system selected according to actual needs, so as to convert the image data collected by each camera in the multi-camera system into a unified coordinate system, ensure that the views of all cameras are under the same reference, reduce the error caused by the difference in viewing angle, and thus improve the accuracy of subsequent depth estimation.

[0039] Step S2022, extrinsic calibration is performed on the coordinate system of each camera in the multi-camera system and the coordinate system of the target camera respectively.

[0040] Get the internal and external parameters of each camera. The internal parameters (intrinsics) of the camera include the focal length and principal point position of the camera, which are used to correct lens distortion and ensure the accuracy of the collected image. They can also indicate the mapping relationship from the three-dimensional world to the two-dimensional image coordinates, project the three-dimensional points onto the image plane, generate the correct image coordinates, and convert the disparity values ​​into depth values ​​when calculating the depth map. When converting images from different cameras to a unified coordinate system, it indicates how to convert the points in the image from the camera coordinate system to the target camera coordinate system for multi-view reconstruction, ensuring that feature points can be accurately connected at different perspectives, improving the accuracy of feature matching, reducing matching errors, and thus improving the accuracy of subsequent depth estimation. The external parameters of the camera include the rotation matrix and translation vector of the camera, which are used to indicate the rotation direction and translation position of the camera in the world coordinate system to convert the image from the camera coordinate system to the target camera coordinate system.

[0041] By calibrating the coordinate system of each camera with the target camera coordinate system, the conversion relationship between the coordinate system of each camera and the target camera coordinate system can be obtained, so as to subsequently convert the image captured by each camera from the camera coordinate system to the target camera coordinate system.

[0042] Step S2023 , performing depth estimation on each frame of image data collected by each camera, to obtain a depth map to be converted for each frame corresponding to each frame of image data collected by each camera.

[0043] Optionally, each frame of image data collected by each camera is first subjected to image preprocessing, such as denoising, geometric correction, etc., and then each frame of image data collected by each camera is processed using a deep learning algorithm to extract features and match features, and the disparity is calculated based on the internal and external parameters of the camera and the matched feature points, and then the disparity is converted into depth to generate a preliminary depth map, and post-processing is performed, such as smoothing, filling in missing values, etc., to obtain depth maps to be converted for each frame of image data collected by each camera. Exemplarily, the deep learning algorithm used can be FisheyeDistanceNet, UnrectDepthNet, SynDistNet, SVDistNet, etc.

[0044] Step S2024: convert each frame of the to-be-converted depth map corresponding to each camera into the target camera coordinate system to obtain each frame of the depth map corresponding to each frame of image data collected by each camera.

[0045] The depth map to be converted is in the corresponding camera coordinate system, so it is also necessary to convert the depth map to be converted corresponding to each camera into the target camera coordinate system to obtain the depth map of each frame corresponding to each frame of image data collected by each camera.

[0046] Step S203, matching each frame of the laser point cloud with each frame of the depth map of each camera according to the number of frames, to obtain a laser point cloud data set and a depth map data set that match each frame.

[0047] Specifically, the above step S203 includes: Step S2031, grouping the laser point cloud and the depth map of each camera according to the frame number.

[0048] For example, the result of grouping the laser point cloud and the depth map of each camera is as follows:

[0049] in, represents the depth map obtained by the image captured by the jth camera in the i-th frame in a multi-camera system, for example Represents the depth map of the third camera in the second frame. represents the laser point cloud collected by the laser radar at the i-th frame, and N represents a total of N frames of data, for example Represents the laser point cloud collected by the laser radar in the Nth frame.

[0050] Step S2032: for each group, convert the laser point cloud to the target camera coordinate system to obtain the laser point cloud after coordinate conversion.

[0051] For example, according to the initial value of the external parameter , ,Will Transform to the target camera coordinate system.

[0052] Step S2033, based on the internal parameters of each camera, project the laser point cloud after coordinate transformation to the pixel plane of each camera to obtain the pixel points corresponding to the laser point cloud after coordinate transformation.

[0053] For the target camera in each camera, first obtain the intrinsic parameters of the target camera and the coordinates of the laser point cloud, and then convert the laser point cloud in the target camera coordinate system into the target camera coordinate system according to the extrinsic parameters of the target camera to obtain the coordinates of the laser point cloud in the target camera coordinate system, and then project it to the two-dimensional pixel plane corresponding to the target camera according to the intrinsic parameters of the target camera to obtain the pixel coordinates of the laser point cloud, and finally round the pixel coordinates of the laser point cloud to obtain the pixel points corresponding to the laser point cloud.

[0054] Step S2034, matching the pixel points corresponding to the laser point cloud after coordinate conversion with the pixel points in the depth map to obtain a matching laser point cloud data set and depth map data set.

[0055] Since the laser point cloud is converted to a pixel plane, and the depth map is also a pixel plane, the pixel points corresponding to the laser point cloud can be matched with the pixel points in the depth map to obtain the corresponding relationship between the pixel points in the laser point cloud and the depth estimation information contained in the pixel points in the depth map. For example, taking the i-th frame as an example, the matching results are as follows:

[0056]

[0057] in, Represents the i-th frame depth map data group, represents the i-th frame laser point cloud data set, It represents the number of matching point pairs between the laser point cloud of the i-th frame and the depth map of the i-th frame, that is, the number of matching point pairs between the laser point cloud of the i-th frame and the depth map of the i-th frame. Figure 1 A total of matches Matching point pairs.

[0058] Step S204: establishing an external parameter calibration function based on the mutual information between the laser point cloud data set and the depth map data set that match each frame.

[0059] Specifically, the above step S204 includes: Step S2041, respectively obtain the mutual information between the laser point cloud data set and the depth map data set that match each frame.

[0060] Specifically, for each frame, the normalized histogram of the laser point cloud of the matching laser point cloud data group and the normalized histogram of the depth map of the depth map data group are obtained respectively; then, the laser point cloud marginal probability of the laser point cloud normalized histogram, the depth map marginal probability of the depth map normalized histogram and the joint probability between the normalized histogram of the laser point cloud and the normalized histogram of the depth map are obtained respectively; finally, based on the laser point cloud marginal probability, the depth map marginal probability and the joint probability, the mutual information between the matching laser point cloud data group and the depth map data group is obtained.

[0061] For example, given two random variables X and Y, decomposing the entropy of the random variables can define the mutual information for:

[0062]

[0063]

[0064]

[0065] in, is the marginal probability of the random variable X, is the marginal probability of the random variable Y, is the joint probability of random variables X and Y, is the entropy of the random variable X, is the entropy of the random variable Y, is the joint entropy of random variable X and random variable Y.

[0066] In conjunction with the external parameter calibration problem of this embodiment, the mutual information between the laser point cloud data set and the depth map data set matching each frame is obtained by the following formula:

[0067]

[0068]

[0069]

[0070] in, represents the mutual information of the i-th frame, Represents the i-th frame depth map data group, Represents the ath element in the i-th frame depth map data group, represents the i-th frame laser point cloud data set, represents the bth element in the i-th frame laser point cloud data set, express The marginal probability of express The marginal probability of express and The joint probability of represents the entropy of the depth map data set of the i-th frame, represents the entropy of the laser point cloud data set of the i-th frame, It represents the joint entropy of the i-th frame depth map data group and the i-th frame laser point cloud data group.

[0071] Step S2042, summing the mutual information between the laser point cloud data set and the depth map data set that match each frame to establish an extrinsic calibration function.

[0072] Exemplarily, the established external parameter calibration function is as follows:

[0073] Step S205, obtaining an optimal solution of the extrinsic parameters that maximizes the extrinsic parameter calibration function as the extrinsic parameter calibration result between the laser radar and the multi-camera system.

[0074] Specifically, different mutual information sum values ​​can be obtained by using different external parameters between the multi-camera system and the laser radar. When the external parameters between the multi-camera system and the laser radar are the most accurate, that is, when the external parameters take the optimal solution, the corresponding mutual information sum value is the largest, that is, the extrinsic parameter calibration function is maximized when the extrinsic parameter takes the optimal solution. Therefore, the optimal solution of the external parameters that maximizes the extrinsic parameter calibration function can be obtained as the external parameter calibration result.

[0075] Exemplarily, the optimal solution of the external parameters is expressed as follows:

[0076] Among them, arg represents the variable, express When the maximum value is taken (i.e. the external parameter calibration function is maximized), the external parameter The value of . Finally, the optimal solution of the extrinsic parameters is used as the extrinsic parameter calibration result between the lidar and the multi-camera system.

[0077] The optimal solution of the external parameters is the external parameters (external parameters) between the laser radar and the multi-camera system. For example, the Powell BOBYQA algorithm is used to solve the external parameter calibration function online to obtain the external parameters of the multi-camera system and the laser radar. .

[0078] The external parameter calibration method provided in this embodiment first obtains each frame of image data collected by each camera in the multi-camera system and each frame of laser point cloud collected by the laser radar, then respectively estimates the depth of each frame of image data collected by each camera to obtain each frame of depth map corresponding to each frame of image data, then matches each frame of laser point cloud with each frame of depth map of each camera according to the number of frames, obtains each frame of laser point cloud data group and depth map data group matching, and then establishes an external parameter calibration function based on the mutual information between each frame of laser point cloud data group and depth map data group matching, and finally obtains the external parameter optimal solution that maximizes the external parameter calibration function as the external parameter calibration result between the laser radar and the multi-camera system. By matching the depth map corresponding to the image data with the laser point cloud, and determining the most accurate dependency relationship between the laser point cloud data group and the depth map data group based on the mutual information between the two, the optimal external parameter is obtained, and a convenient and highly accurate external parameter calibration function is realized without the need for a calibration plate, a scene with suitable texture, or a large amount of training data.

[0079] In this embodiment, an external parameter calibration system is also provided, and the data processing device in the system is used to implement the above-mentioned embodiments and preferred implementation modes, and the descriptions that have been made will not be repeated. As used below, the term "module" can implement a combination of software and / or hardware of a predetermined function. Although the data processing device described in the following embodiments is preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceivable.

[0080] This embodiment provides an external parameter calibration system, such as Figure 4 As shown, including: Laser radar, installed on the target vehicle, is used to collect laser point clouds of each frame; A multi-camera system, installed on the target vehicle, is used to collect image data for each frame; Data processing equipment, including: A data acquisition module, used to acquire each frame of image data collected by each camera in the multi-camera system and each frame of laser point cloud collected by the laser radar; The depth estimation module is used to perform depth estimation on each frame of image data collected by each camera to obtain a depth map of each frame corresponding to each frame of image data; A matching module is used to match each frame of laser point cloud with each frame of depth map of each camera according to the number of frames to obtain a laser point cloud data group and a depth map data group that match each frame; A function building module, used to build an external parameter calibration function based on the mutual information between the laser point cloud data set and the depth map data set that match each frame; The function solving module is used to obtain the optimal solution of the external parameters that maximizes the external parameter calibration function as the external parameter calibration result between the laser radar and the multi-camera system.

[0081] In an optional embodiment, the depth estimation module is also used to: determine the target camera coordinate system; perform external parameter calibrating the coordinate system of each camera in the multi-camera system with the target camera coordinate system; perform depth estimation on each frame of image data captured by each camera to obtain depth maps for each frame to be converted corresponding to each frame of image data captured by each camera; convert the depth maps for each frame to be converted corresponding to each camera to the target camera coordinate system to obtain depth maps for each frame corresponding to each frame of image data captured by each camera.

[0082] In an optional embodiment, the matching module is also used to: group the laser point cloud and the depth map of each camera according to the number of frames; for each group, convert the laser point cloud to the target camera coordinate system to obtain the laser point cloud after coordinate conversion; based on the internal parameters of each camera, project the laser point cloud after coordinate conversion to the pixel plane of each camera to obtain the pixel points corresponding to the laser point cloud after coordinate conversion; match the pixel points corresponding to the laser point cloud after coordinate conversion with the pixel points in the depth map to obtain matching laser point cloud data groups and depth map data groups.

[0083] In an optional embodiment, the function establishment module is also used to: respectively obtain the mutual information between the laser point cloud data group and the depth map data group that match each frame; sum the mutual information between the laser point cloud data group and the depth map data group that match each frame to establish an external parameter calibration function.

[0084] In an optional embodiment, the function establishment module is also used to: for each frame, respectively obtain the laser point cloud normalized histogram of the matching laser point cloud data group and the depth map normalized histogram of the depth map data group; respectively obtain the laser point cloud marginal probability of the laser point cloud normalized histogram, the depth map marginal probability of the depth map normalized histogram, and the joint probability between the laser point cloud normalized histogram and the depth map normalized histogram; based on the laser point cloud marginal probability, the depth map marginal probability and the joint probability, obtain the mutual information between the matching laser point cloud data group and the depth map data group.

[0085] In an optional implementation, the mutual information between the laser point cloud data set and the depth map data set matching each frame is obtained by the following formula:

[0086] in, represents the mutual information of the i-th frame, Represents the i-th frame depth map data group, represents the i-th frame laser point cloud data set, represents the entropy of the depth map data set of the i-th frame, represents the entropy of the laser point cloud data set of the i-th frame, It represents the joint entropy of the i-th frame depth map data group and the i-th frame laser point cloud data group.

[0087] The further functional description of each of the above modules and units is the same as that of the above corresponding embodiments and will not be repeated here.

[0088] The data processing device in the external parameter calibration system in this embodiment is presented in the form of a functional unit, where the unit refers to an ASIC (Application Specific Integrated Circuit) circuit, a processor and memory that executes one or more software or fixed programs, and / or other devices that can provide the above-mentioned functions.

[0089] The embodiment of the present invention also provides a computer device having the above Figure 4 The data processing equipment in the external parameter calibration system shown.

[0090] See also Figure 5 , Figure 5 is a schematic diagram of the structure of a computer device provided by an optional embodiment of the present invention, such as Figure 5 As shown, the computer device comprises: one or more processors 10, memory 20, and interfaces for connecting various components, including high-speed interfaces and low-speed interfaces. Various components utilize different buses to communicate with each other, and can be installed on a common mainboard or installed in other ways as required. The processor can process the instructions executed in the computer device, including instructions stored in the memory or on the memory to display the graphical information of the GUI on an external input / output device (such as, a display device coupled to the interface). In some optional embodiments, if necessary, multiple processors and / or multiple buses can be used together with multiple memories and multiple memories. Equally, multiple computer devices can be connected, and each device provides part of the necessary operation (for example, as a server array, a group of blade servers, or a multi-processor system). Figure 5 A processor 10 is taken as an example.

[0091] The processor 10 may be a central processing unit, a network processor or a combination thereof. The processor 10 may further include a hardware chip. The hardware chip may be a dedicated integrated circuit, a programmable logic device or a combination thereof. The programmable logic device may be a complex programmable logic device, a field programmable gate array, a general purpose array logic or any combination thereof.

[0092] The memory 20 stores instructions executable by at least one processor 10, so that the at least one processor 10 executes the method shown in the above embodiment.

[0093] The memory 20 may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created according to the use of the computer device, etc. In addition, the memory 20 may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some optional embodiments, the memory 20 may optionally include a memory remotely arranged relative to the processor 10, and these remote memories may be connected to the computer device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0094] The memory 20 may include a volatile memory, such as a random access memory; the memory may also include a non-volatile memory, such as a flash memory, a hard disk or a solid state drive; the memory 20 may also include a combination of the above types of memory.

[0095] The computer device also includes an input device 30 and an output device 40. The processor 10, the memory 20, the input device 30 and the output device 40 may be connected via a bus or other means. Figure 5 The example of connecting through bus is taken in the following.

[0096] The input device 30 can receive input digital or character information, and generate key signal input related to the user settings and function control of the computer device, such as a touch screen, a keypad, a mouse, a track pad, a touch pad, an indicator bar, one or more mouse buttons, a trackball, a joystick, etc. The output device 40 may include a display device, an auxiliary lighting device (e.g., an LED) and a tactile feedback device (e.g., a vibration motor), etc. The above-mentioned display device includes but is not limited to a liquid crystal display, a light emitting diode, a display and a plasma display. In some optional embodiments, the display device may be a touch screen.

[0097] The embodiment of the present invention also provides a computer-readable storage medium. The method according to the embodiment of the present invention can be implemented in hardware, firmware, or can be implemented as a computer code that can be recorded in a storage medium, or can be implemented as a computer code that is originally stored in a remote storage medium or a non-temporary machine-readable storage medium and will be stored in a local storage medium through a network download, so that the method described herein can be stored in such software processing on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. Among them, the storage medium can be a magnetic disk, an optical disk, a read-only storage memory, a random access memory, a flash memory, a hard disk or a solid-state hard disk, etc.; further, the storage medium can also include a combination of the above types of memories. It can be understood that a computer, a processor, a microprocessor controller, or programmable hardware includes a storage component that can store or receive software or computer code. When the software or computer code is accessed and executed by a computer, a processor, or hardware, the method shown in the above embodiment is implemented.

[0098] A part of the present invention may be applied as a computer program product, such as a computer program instruction, which, when executed by a computer, can call or provide the method and / or technical solution according to the present invention through the operation of the computer. Those skilled in the art should understand that the existence of computer program instructions in computer-readable media includes, but is not limited to, source files, executable files, installation package files, etc., and accordingly, the way in which computer program instructions are executed by a computer includes, but is not limited to: the computer directly executes the instruction, or the computer compiles the instruction and then executes the corresponding compiled program, or the computer reads and executes the instruction, or the computer reads and installs the instruction and then executes the corresponding installed program. Here, the computer-readable medium may be any available computer-readable storage medium or communication medium accessible to the computer.

[0099] Although the embodiments of the present invention have been described in conjunction with the accompanying drawings, those skilled in the art may make various modifications and variations without departing from the spirit and scope of the present invention, and such modifications and variations shall all fall within the scope of protection of the present invention.

Claims

1. A method for calibrating an external parameter, characterized in that: A data processing device used in an external parameter calibration system; the external parameter calibration system also includes a laser radar and a multi-camera system installed on a target vehicle; the method includes: Acquire each frame of image data collected by each camera in the multi-camera system and each frame of laser point cloud collected by the laser radar; Depth estimation is performed on each frame of image data collected by each camera to obtain a depth map of each frame corresponding to each frame of image data; Match each frame of laser point cloud with each frame of depth map of each camera according to the number of frames to obtain a laser point cloud data set and a depth map data set that match each frame; An external parameter calibration function is established based on the mutual information between the laser point cloud data set and the depth map data set that match each frame; The optimal solution of the extrinsic parameters that maximizes the extrinsic calibration function is obtained as the extrinsic parameter calibration result between the lidar and the multi-camera system.

2. The method according to claim 1, characterized in that The depth estimation is performed on each frame of image data collected by each camera to obtain a depth map of each frame corresponding to each frame of image data, including: Determine the target camera coordinate system; Performing external parameter calibration on the coordinate system of each camera in the multi-camera system and the coordinate system of the target camera respectively; Depth estimation is performed on each frame of image data collected by each camera to obtain a depth map to be converted for each frame of image data collected by each camera; The depth maps to be converted of each frame corresponding to each camera are converted to the target camera coordinate system to obtain the depth maps of each frame corresponding to each frame of image data collected by each camera.

3. The method according to claim 2, characterized in that The step of matching each frame of laser point cloud with each frame of depth map of each camera according to the number of frames to obtain a laser point cloud data set and a depth map data set matching each frame includes: Group the laser point cloud and the depth map of each camera by frame number; For each group, the laser point cloud is converted to the target camera coordinate system to obtain the laser point cloud after coordinate conversion; Based on the internal parameters of each camera, the laser point cloud after coordinate transformation is projected onto the pixel plane of each camera to obtain the pixel points corresponding to the laser point cloud after coordinate transformation; The pixel points corresponding to the laser point cloud after coordinate transformation are matched with the pixel points in the depth map to obtain a matching laser point cloud data set and depth map data set.

4. The method according to any one of claims 1 to 3, characterized in that: The method of establishing an external parameter calibration function based on the mutual information between the laser point cloud data set and the depth map data set that match each frame includes: Obtain the mutual information between the laser point cloud data set and the depth map data set that match each frame respectively; The mutual information between the laser point cloud data set and the depth map data set that match each frame is summed to establish an extrinsic calibration function.

5. The method according to claim 4, characterized in that The step of respectively obtaining the mutual information between the laser point cloud data set and the depth map data set that match each frame includes: For each frame, respectively obtain a normalized histogram of the laser point cloud data set and a normalized histogram of the depth map data set of the matching laser point cloud data set; The laser point cloud marginal probability of the normalized histogram of the laser point cloud, the depth map marginal probability of the normalized histogram of the depth map, and the joint probability between the normalized histogram of the laser point cloud and the normalized histogram of the depth map are respectively calculated; Based on the laser point cloud marginal probability, the depth map marginal probability and the joint probability, the mutual information between the matched laser point cloud data set and the depth map data set is obtained.

6. The method according to claim 5, characterized in that The mutual information between the laser point cloud data set and the depth map data set matching each frame is calculated using the following formula: in, represents the mutual information of the i-th frame, Represents the i-th frame depth map data group, represents the i-th frame laser point cloud data set, represents the entropy of the depth map data set of the i-th frame, represents the entropy of the i-th frame laser point cloud data group, It represents the joint entropy of the i-th frame depth map data group and the i-th frame laser point cloud data group.

7. An external parameter calibration system, characterized in that: The system comprises: Laser radar, installed on the target vehicle, is used to collect laser point clouds of each frame; A multi-camera system, installed on the target vehicle, is used to collect image data for each frame; Data processing equipment, including: A data acquisition module, used to acquire each frame of image data collected by each camera in the multi-camera system and each frame of laser point cloud collected by the laser radar; The depth estimation module is used to perform depth estimation on each frame of image data collected by each camera to obtain a depth map of each frame corresponding to each frame of image data; A matching module is used to match each frame of laser point cloud with each frame of depth map of each camera according to the number of frames to obtain a laser point cloud data group and a depth map data group that match each frame; A function building module, used to build an external parameter calibration function based on the mutual information between the laser point cloud data set and the depth map data set that match each frame; The function solving module is used to obtain the optimal solution of the external parameters that maximizes the external parameter calibration function as the external parameter calibration result between the laser radar and the multi-camera system.

8. A computer device, characterized in that: include: A memory and a processor, wherein the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the external parameter calibration method according to any one of claims 1 to 6 by executing the computer instructions.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a computer to execute the external parameter calibration method according to any one of claims 1 to 6.

10. A computer program product, characterized in that The method comprises computer instructions, wherein the computer instructions are used to enable a computer to execute the external parameter calibration method according to any one of claims 1 to 6.

Images