Method, device, equipment and storage medium for determining extrinsic parameters of monocular camera

Abstract

The present invention discloses a method, apparatus, device, and storage medium for determining the extrinsic parameters of a monocular camera. The method comprises: obtaining a video image captured by the monocular camera and a depth image captured by a laser radar on the same vehicle as the monocular camera; extracting first feature information corresponding to the video image; projecting the depth image onto the coordinate system of the video image to obtain a projected image corresponding to the depth image; extracting second feature information corresponding to the projected image; concatenating the first and second feature information to obtain joint feature information, and performing a global regression on the projected image using the joint feature information to obtain the extrinsic parameters of the monocular camera. The present invention solves the technical problem of manual multi-sensor calibration, which is time-consuming, labor-intensive, and inefficient.

Description

Technical Field

[0001] The present invention relates to the field of autonomous driving technology, and in particular to a method, device, equipment, and storage medium for determining extrinsic parameters of a monocular camera. Background Art

[0002] Sensor calibration is a fundamental requirement for autonomous driving, and accurate calibration is the foundation of multi-sensor fusion. Calibration confirms the mutual conversion relationship between different coordinate systems, thereby achieving better sensor data fusion and improving the ability to identify objects in the autonomous driving environment. Currently, multi-sensor calibration requires manual intervention. This requires an operator to hold a checkerboard and move it around the field, changing the position of the checkerboard so that multiple sensors collect data corresponding to the checkerboard in different positions. The operator then manually processes the data collected by the sensors to achieve multi-sensor calibration. Therefore, manual multi-sensor calibration is not only time-consuming and labor-intensive, but also inefficient. Summary of the Invention

[0003] The embodiments of the present invention provide a method, apparatus, device, and storage medium for determining the extrinsic parameters of a monocular camera, aiming to solve the technical problem that manual multi-sensor calibration is time-consuming, labor-intensive, and inefficient.

[0004] An embodiment of the present invention provides a method for determining extrinsic parameters of a monocular camera, the method comprising:

[0005] Obtaining a video image captured by a monocular camera and a depth image captured by a laser radar on the same vehicle as the monocular camera; wherein the video image and the depth image are images captured by the monocular camera and the laser radar for the same environment;

[0006] Extracting first feature information corresponding to the video image;

[0007] Projecting the depth image onto the coordinate system of the video image to obtain a projected image corresponding to the depth image;

[0008] extracting second feature information corresponding to the projection image;

[0009] The first feature information and the second feature information are spliced ​​to obtain joint feature information, and the projected image is globally regressed using the joint feature information to obtain extrinsic parameters of the monocular camera.

[0010] In one embodiment, the step of acquiring a video image captured by a monocular camera and a depth image captured by a laser radar on the same vehicle as the monocular camera includes:

[0011] Obtain the original video images captured by the monocular camera and the point cloud data collected by the lidar;

[0012] Converting the point cloud data into a sparse depth map;

[0013] Normalizing the original video image to obtain the video image;

[0014] The sparse depth map is normalized, and the depth image is generated according to the normalized sparse depth map.

[0015] In one embodiment, the step of generating the depth image according to the normalized sparse depth map includes:

[0016] Performing maximum pooling on the normalized sparse depth map to obtain a semi-dense depth map;

[0017] The semi-dense depth map is used as the depth image.

[0018] In one embodiment, the step of projecting the depth image onto the coordinate system where the video image is located to obtain a projected image corresponding to the depth image includes:

[0019] Calibrate the depth image using a preset initial calibration and an intrinsic parameter matrix of the monocular camera;

[0020] The depth image is projected onto the coordinate system of the calibrated video image to obtain the projected image.

[0021] In one embodiment, the step of concatenating the first feature information and the second feature information to obtain joint feature information includes:

[0022] Establishing a feature mapping between the first feature information and the second feature information;

[0023] The first feature information and the second feature information are concatenated based on the feature mapping to obtain the joint feature information.

[0024] In one embodiment, the step of performing global regression on the projected image using the joint feature information to obtain the extrinsic parameters of the monocular camera includes:

[0025] Determining a plurality of first preset calibration matrices corresponding to the joint feature information and a first matrix score of each of the first preset calibration matrices using a pre-trained neural network;

[0026] The first preset calibration matrix with the highest first matrix score is determined as the extrinsic parameter.

[0027] In one embodiment, after the step of performing global regression on the projected image using the joint feature information to obtain the extrinsic parameters of the monocular camera, the method further includes:

[0028] Iteratively updating the weight parameters of the neural network using the extrinsic parameters;

[0029] When the number of iterations reaches a set number, using the deep neural network to determine a plurality of second preset calibration matrices corresponding to the joint feature information and a second matrix score of each of the second preset calibration matrices;

[0030] A second preset calibration matrix with the highest second matrix score is determined as the extrinsic parameters of the monocular camera.

[0031] In addition, to achieve the above-mentioned object, the present invention further provides a device for determining extrinsic parameters of a monocular camera, the device comprising:

[0032] An image acquisition module, configured to acquire a video image captured by a monocular camera and a depth image captured by a laser radar on the same vehicle as the monocular camera; wherein the video image and the depth image are images captured by the monocular camera and the laser radar for the same environment;

[0033] a feature extraction module, configured to extract first feature information corresponding to the video image, and project the depth image onto the coordinate system where the video image is located to obtain a projected image corresponding to the depth image; and project the depth image onto the coordinate system where the video image is located to obtain a projected image corresponding to the depth image, and extract second feature information corresponding to the projected image;

[0034] an information splicing module, configured to splice the first feature information and the second feature information to obtain joint feature information;

[0035] A data regression module is used to perform global regression on the projected image using the joint feature information to obtain the extrinsic parameters of the monocular camera.

[0036] In addition, to achieve the above-mentioned purpose, the present invention also provides a terminal device comprising: a memory, a processor, and an external parameter determination program for a monocular camera stored on the memory and runnable on the processor, wherein the external parameter determination program for the monocular camera implements the steps of the above-mentioned method for determining the external parameters of the monocular camera when executed by the processor.

[0037] In addition, to achieve the above-mentioned purpose, the present invention also provides a storage medium on which a program for determining the external parameters of a monocular camera is stored. When the program for determining the external parameters of a monocular camera is executed by a processor, the steps of the above-mentioned method for determining the external parameters of a monocular camera are implemented.

[0038] The technical solutions of a method, apparatus, device, and storage medium for determining extrinsic parameters of a monocular camera provided in the embodiments of the present invention have at least the following technical effects or advantages:

[0039] When determining the extrinsic parameters of a monocular camera in the context of autonomous driving, the present invention obtains a video image captured by the monocular camera and a depth image captured by a laser radar on the same vehicle as the monocular camera based on the RegNet network, extracts first feature information corresponding to the video image, projects the depth image to the coordinate system where the video image is located, obtains a projected image corresponding to the depth image, extracts second feature information corresponding to the projected image, splices the first feature information and the second feature information to obtain joint feature information, and uses the joint feature information to perform global regression on the projected image to obtain the extrinsic parameters of the monocular camera. This technical solution realizes automatic calibration of the monocular camera and the laser radar and automatic determination of the extrinsic parameters of the monocular camera, solves the technical problem of manual multi-sensor calibration, which is time-consuming, labor-intensive and inefficient, and not only improves the work efficiency of determining the extrinsic parameters of the monocular camera, but also saves labor costs and time. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 1. A flow chart of an embodiment of a method for determining extrinsic parameters of a monocular camera according to the present invention;

[0041] Figure 2 This is a schematic diagram of the deep neural network structure of the present invention;

[0042] Figure 3 1. A schematic flow chart of another embodiment of a method for determining extrinsic parameters of a monocular camera according to the present invention;

[0043] Figure 4 Schematic diagram of the specific process of step S210 in the method for determining the extrinsic parameters of a monocular camera of the present invention;

[0044] Figure 5 A schematic diagram of the hardware operating environment involved in an embodiment of the present invention;

[0045] Figure 6 This is a functional module diagram of the device for determining the extrinsic parameters of a monocular camera according to the present invention. DETAILED DESCRIPTION

[0046] To better understand the above technical solutions, exemplary embodiments of the present invention will be described in more detail below with reference to the accompanying drawings. Although exemplary embodiments of the present invention are shown in the accompanying drawings, it should be understood that the present invention can be implemented in various forms and should not be limited by the embodiments described herein. Instead, these embodiments are provided to enable a more thorough understanding of the present invention and to fully convey the scope of the present invention to those skilled in the art.

[0047] An embodiment of the present invention provides an embodiment of a method for determining extrinsic parameters of a monocular camera. It should be noted that although a logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in an order different from that shown here.

[0048] like Figure 1 As shown, in one embodiment of the present invention, the method for determining the extrinsic parameters of a monocular camera of the present invention includes the following steps:

[0049] Step S210: Acquire a video image captured by a monocular camera and a depth image captured by a laser radar on the same vehicle as the monocular camera.

[0050] In this embodiment, a monocular camera and a lidar are installed on the same vehicle, which has an autonomous driving function. The monocular camera is used to capture video images, and the lidar is used to capture depth images. The video images can be understood as RGB images, and the depth images can reflect the geometric shape of the visible surface of objects. Specifically, in the context of autonomous driving, the monocular camera video image and the lidar depth image are obtained. The video image and depth image are images collected by the monocular camera and the lidar for the same environment, and the video image and depth image correspond to images of objects in the same location or area in the real environment.

[0051] Step S220: extracting first feature information corresponding to the video image.

[0052] In this embodiment, the first feature information can be extracted by traditional manual extraction methods or by deep neural network extraction. Figure 2 As shown, the deep neural network is pre-trained. The deep neural network refers to the RegNet network, which includes three deep network structures and one regression network structure. Among them, there are two parallel deep network structures for feature extraction, namely the first deep network structure (NIN1) and the second deep network structure (NIN2). Another deep network structure is called the third deep network structure (NIN3), which is set after the first and second deep network structures. The regression network structure (FC) is set after the third deep network structure. The deep network structure refers to the Network in Network, abbreviated as NIN. Each NIN includes a k×k convolution and multiple 1×1 convolutions. RGB represents the video image, Deph represents the depth image, and Project represents the projection of the depth image onto the coordinate system of the video image.

[0053] After obtaining the video image, the video image is used as the input of the RegNet network and input into the RegNet network. The RegNet network extracts rich feature representations through NIN1. That is, the video image is input into NIN1, and NIN1 extracts the points of interest in the video image to obtain the first feature information. The first feature information refers to RGB feature information, such as the RGB component values ​​of the image, texture information, etc.

[0054] Step S230: Projecting the depth image to the coordinate system where the video image is located to obtain a projected image corresponding to the depth image.

[0055] Step S240: extracting second feature information corresponding to the projection image.

[0056] Based on the preset initial calibration Hinit and the intrinsic parameter matrix of the monocular camera, the depth image is projected into the coordinate system of the video image to obtain a projection image corresponding to the depth image. Some pixels on the projection image will obtain the depth feature information of the depth image. After obtaining the depth image, the depth image is also used as the input of the RegNet network and input into the RegNet network. The RegNet network extracts rich feature representations through NIN2. That is, the depth image is input into NIN2, and NIN2 extracts points of interest in the depth image to obtain secondary feature information. The secondary feature information refers to depth feature information, such as depth points.

[0057] The projecting of the depth image to the coordinate system of the video image to obtain the projection image corresponding to the depth image includes: using a preset initial calibration H init The depth image is calibrated with the intrinsic parameter matrix of the monocular camera, and the depth image is projected to the coordinate system of the calibrated video image to obtain the projected image. Before the depth image is input to NIN2, the preset initial calibration H is used. init The video image is calibrated using the intrinsic parameter matrix of the monocular camera, thereby achieving initial calibration of the video image. Multiple preset initial calibrations are used for video image calibration, each of which is randomly assigned. The depth image is then projected onto the coordinate system of the calibrated video image, and the resulting image is the projected image.

[0058] Step S250: concatenate the first feature information and the second feature information to obtain joint feature information.

[0059] In this embodiment, after obtaining the first feature information and the second feature information, NIN3 concatenates the first feature information and the second feature information to generate a joint representation, which is the joint feature information.

[0060] Specifically, step S250 includes: establishing a feature mapping between the first feature information and the second feature information; splicing the first feature information and the second feature information based on the feature mapping to obtain the joint feature information. Wherein, the obtained projection image and video image have overlapping content, so the extracted first feature information and the second feature information have common feature information. Some pixels on the projection image will obtain the depth points of the depth image, and the depth points corresponding to some pixels on the projection image correspond to some depth points in the depth image, that is, the first feature information and the second feature information have common feature information. Based on the common feature information, a feature mapping between the first feature information and the second feature information can be established, and then the first feature information and the second feature information can be connected according to the feature mapping to achieve splicing and fusion of the two feature information, thereby obtaining joint feature information, which can represent the feature information of the video image and the depth image together.

[0061] Step S260: performing global regression on the projected image using the joint feature information to obtain extrinsic parameters of the monocular camera.

[0062] In this embodiment, global regression is performed on the projected image using the joint feature information. Global regression is the process of predicting a preset calibration matrix corresponding to the joint feature information to obtain the preset calibration matrix corresponding to the joint feature information. The obtained preset calibration matrix is ​​then used to determine the extrinsic parameters of the monocular camera. The data in the preset calibration matrix is ​​the extrinsic parameters of the monocular camera.

[0063] Specifically, step S260 includes using a pre-trained neural network to determine multiple first preset calibration matrices corresponding to the joint feature information and first matrix scores for each of the first preset calibration matrices, and determining the first preset calibration matrix with the highest first matrix score as an extrinsic parameter. Furthermore, the joint feature information can be used to determine the probability that a depth point in the depth image correctly falls within the video image, or alternatively, the probability that a pixel on the projected image accurately captures the depth point.

[0064] A pre-trained neural network can be understood as a classifier. Its training process includes training the network to be trained using a large number of image samples, where each image sample includes a video image sample and a depth image sample. Each image sample carries a manually calibrated preset calibration matrix and a matrix score for the preset calibration matrix. The data in the preset calibration matrix are the manually set extrinsic parameters of the monocular camera. If the error between the extrinsic parameters of the monocular camera output by the network to be trained and the data in the preset calibration matrix is ​​less than or equal to a preset error threshold, training of the network to be trained is stopped, and the network to be trained is saved as a deep neural network.

[0065] After obtaining the joint feature information, the classifier processes the joint feature information to obtain multiple preset calibration matrices and matrix scores for each preset calibration matrix. Here, the obtained preset calibration matrix and the matrix of the preset calibration matrix are respectively referred to as the first preset calibration matrix and the first matrix score of the first preset calibration matrix. That is, the classifier outputs multiple first preset calibration matrices after processing the joint feature information, and scores each first preset calibration matrix, and simultaneously outputs the first matrix score of each first preset calibration matrix. Then, the data in the first preset calibration matrix with the highest first matrix score is determined as the extrinsic parameter. For example, the multiple first preset calibration matrices obtained are A, B, C, D, E, and F, and the matrix scores corresponding to A, B, C, D, E, and F are a, b, c, d, e, and f, respectively. Among them, e is the largest, and the data in E is determined to be the extrinsic parameter of the monocular camera.

[0066] It is worth noting that the neural network can be an independent network model or belong to a RegNet network. When the neural network belongs to a RegNet network, the neural network is the regression network structure in the RegNet network.

[0067] According to the above technical solution, this embodiment realizes automatic calibration of the monocular camera and the lidar and automatic determination of the extrinsic parameters of the monocular camera through the RegNet network in the context of autonomous driving. This not only improves the efficiency of determining the extrinsic parameters of the monocular camera, but also saves labor costs and time.

[0068] like Figure 3 In another embodiment of the present invention, the method for determining the extrinsic parameters of a monocular camera of the present invention comprises the following steps:

[0069] Step S210: Acquire a video image captured by a monocular camera and a depth image captured by a laser radar on the same vehicle as the monocular camera;

[0070] Step S220: extracting first feature information corresponding to the video image;

[0071] Step S230: Projecting the depth image to the coordinate system where the video image is located to obtain a projected image corresponding to the depth image;

[0072] Step S240: extracting second feature information corresponding to the projection image;

[0073] Step S250: concatenating the first feature information and the second feature information to obtain joint feature information;

[0074] Step S260: performing global regression on the projected image using the joint feature information to obtain extrinsic parameters of the monocular camera;

[0075] Step S270: iteratively updating the weight parameters of the neural network using the extrinsic parameters;

[0076] Step S280: When the number of iterations reaches a set number, using the deep neural network to determine a plurality of second preset calibration matrices corresponding to the joint feature information and a second matrix score of each of the second preset calibration matrices;

[0077] Step S290: Determine the second preset calibration matrix with the highest second matrix score as the extrinsic parameters of the monocular camera.

[0078] The specific implementation of steps S210-S260 in this embodiment is the same as the specific implementation of steps S210-S260 in the above embodiment and will not be repeated in this embodiment. Because the projection of the depth image varies significantly with a given preset initial calibration, most of the depth image projection cannot be projected onto the video image. This results in only a limited correspondence between the depth image and the video image. If the currently obtained extrinsic parameters are used for practical application, they are not accurate enough. Therefore, further calibration is required, that is, calibrating the extrinsic parameters using a loss function, for example, a Euclidean loss function. The error of the extrinsic parameters of the monocular camera is inferred using the loss function. If the error is too large, the extrinsic parameters of the monocular camera need to be calibrated. The calibration process includes: iteratively updating the weight parameters of the neural network using the extrinsic parameters. When the number of iterations reaches a set number, the neural network is used to determine multiple second preset calibration matrices corresponding to the joint feature information and the second matrix scores of each second preset calibration matrix. The second preset calibration matrix with the highest second matrix score is determined as the target extrinsic parameter of the monocular camera. Iteratively updating the weight parameters of the neural network using extrinsic parameters is a process of modifying the weight parameters of the neural network. After the weight parameters are modified, the neural network becomes more accurate. A set number of iterations is pre-set. When the number of iterations reaches the set number, the neural network is considered relatively accurate. The updated neural network then processes the joint feature information to obtain multiple preset calibration matrices and matrix scores for each preset calibration matrix. The obtained preset calibration matrices and the matrix of the preset calibration matrices are referred to herein as the second preset calibration matrix and the first matrix score of the second preset calibration matrix, respectively. Similarly, the updated neural network processes the joint feature information to output multiple second preset calibration matrices, assigns a score to each second preset calibration matrix, and outputs a second matrix score for each second preset calibration matrix. The data in the second preset calibration matrix with the highest second matrix score is then determined as the extrinsic parameters, thereby improving the accuracy of determining the extrinsic parameters of the monocular camera. For example, multiple second preset calibration matrices are obtained, namely A', B', C', D', E' and F', and the matrix scores corresponding to A', B', C', D', E' and F' are a', b', c', d', e' and f', respectively. Among them, a' is the largest, and the data in A' is determined to be the external parameters of the monocular camera.

[0079] like Figure 4 As shown, based on the above embodiment, step S210 includes the following steps:

[0080] Step S211: obtaining the original video image captured by the monocular camera and the point cloud data captured by the lidar;

[0081] Step S212: normalizing the original video image to obtain the video image;

[0082] Step S213: converting the point cloud data into a sparse depth map;

[0083] Step S214: normalizing the sparse depth map, and generating the depth image according to the normalized sparse depth map.

[0084] Specifically, the original video image is an unprocessed image captured by a monocular camera, and the point cloud data is the original data captured by a lidar. Projecting the point cloud data onto a plane can convert the point cloud data into a sparse depth map. Due to the inaccuracy of the preset initial calibration, if the sparse depth map is projected onto the image plane, the sparse depth map will be inconsistent with the image. Therefore, the original video image and the sparse depth map need to be normalized separately so that the original video image and the sparse depth map are standardized within the range of ±1. The normalized original video image is the video image. After obtaining the normalized sparse depth map, a depth image is generated based on the normalized sparse depth map, that is, the normalized sparse depth map is used as the depth image.

[0085] Considering that if the normalized sparse depth map is used as a depth image, the depth points in the depth image are relatively sparse, which will also affect the effect of generating the projection image. Furthermore, the step of generating the depth image based on the normalized sparse depth map includes: performing maximum pooling processing on the normalized sparse depth map to obtain a semi-dense depth map; and using the semi-dense depth map as the depth image. Specifically, a 5x5 maximum pooling window can be used to maximum pool the normalized sparse depth map into a semi-dense depth map, and the semi-dense depth map can be used as the depth image, which is conducive to speeding up the calculation speed of the RegNet network and improving the generation effect of the projection image.

[0086] like Figure 5 As shown, Figure 5 This is a schematic diagram of the structure of the hardware operating environment involved in the embodiment of the present invention.

[0087] It should be noted that Figure 5 This is a structural diagram of the hardware operating environment of the terminal device.

[0088] like Figure 5As shown, the terminal device may include: a processor 1001, such as a CPU, a memory 1005, a user interface 1003, a network interface 1004, and a communication bus 1002. Among them, the communication bus 1002 is used to realize the connection and communication between these components. The user interface 1003 may include a display screen (Display), an input unit such as a keyboard (Keyboard), and the user interface 1003 may optionally include a standard wired interface and a wireless interface. The network interface 1004 may optionally include a standard wired interface and a wireless interface (such as a WI-FI interface). The memory 1005 may be a high-speed RAM memory or a stable memory (non-volatile memory), such as a disk memory. The memory 1005 may optionally be a storage device independent of the aforementioned processor 1001.

[0089] Those skilled in the art will understand that Figure 5 The terminal device structure shown in the figure does not constitute a limitation on the terminal device, and may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently.

[0090] like Figure 5 As shown, the memory 1005, which is a storage medium, may include an operating system, a network communication module, a user interface module, and a program for determining the extrinsic parameters of a monocular camera. The operating system is a program that manages and controls the hardware and software resources of a terminal device, and the program for determining the extrinsic parameters of a monocular camera and other software or programs are executed.

[0091] exist Figure 5 In the terminal device shown, the user interface 1003 is mainly used to connect to the terminal and communicate data with the terminal; the network interface 1004 is mainly used for the background server and communicates data with the background server; the processor 1001 can be used to call the external parameter determination program of the monocular camera stored in the memory 1005.

[0092] In this embodiment, the terminal device includes: a memory 1005, a processor 1001, and a program for determining the extrinsic parameters of a monocular camera stored in the memory 1005 and executable on the processor, wherein:

[0093] When the processor 1001 calls the extrinsic parameter determination program of the monocular camera stored in the memory 1005, the following operations are performed:

[0094] Obtaining a video image captured by a monocular camera and a depth image captured by a laser radar on the same vehicle as the monocular camera; wherein the video image and the depth image are images captured by the monocular camera and the laser radar for the same environment;

[0095] Extracting first feature information corresponding to the video image;

[0096] Projecting the depth image onto the coordinate system of the video image to obtain a projected image corresponding to the depth image;

[0097] extracting second feature information corresponding to the projection image;

[0098] The first feature information and the second feature information are spliced ​​to obtain joint feature information, and the projected image is globally regressed using the joint feature information to obtain extrinsic parameters of the monocular camera.

[0099] When the processor 1001 calls the extrinsic parameter determination program of the monocular camera stored in the memory 1005, it also performs the following operations:

[0100] Obtain the original video images captured by the monocular camera and the point cloud data collected by the lidar;

[0101] Converting the point cloud data into a sparse depth map;

[0102] Normalizing the original video image to obtain the video image;

[0103] The sparse depth map is normalized, and the depth image is generated according to the normalized sparse depth map.

[0104] When the processor 1001 calls the extrinsic parameter determination program of the monocular camera stored in the memory 1005, it also performs the following operations:

[0105] Performing maximum pooling on the normalized sparse depth map to obtain a semi-dense depth map;

[0106] The semi-dense depth map is used as the depth image.

[0107] When the processor 1001 calls the extrinsic parameter determination program of the monocular camera stored in the memory 1005, it also performs the following operations:

[0108] Calibrate the depth image using a preset initial calibration and an intrinsic parameter matrix of the monocular camera;

[0109] The depth image is projected onto the coordinate system of the calibrated video image to obtain the projected image.

[0110] When the processor 1001 calls the extrinsic parameter determination program of the monocular camera stored in the memory 1005, it also performs the following operations:

[0111] Establishing a feature mapping between the first feature information and the second feature information;

[0112] The first feature information and the second feature information are concatenated based on the feature mapping to obtain the joint feature information.

[0113] When the processor 1001 calls the extrinsic parameter determination program of the monocular camera stored in the memory 1005, it also performs the following operations:

[0114] Determining a plurality of first preset calibration matrices corresponding to the joint feature information and a first matrix score of each of the first preset calibration matrices using a pre-trained neural network;

[0115] The first preset calibration matrix with the highest first matrix score is determined as the extrinsic parameter.

[0116] When the processor 1001 calls the extrinsic parameter determination program of the monocular camera stored in the memory 1005, it also performs the following operations:

[0117] Iteratively updating the weight parameters of the neural network using the extrinsic parameters;

[0118] When the number of iterations reaches a set number, using the deep neural network to determine a plurality of second preset calibration matrices corresponding to the joint feature information and a second matrix score of each of the second preset calibration matrices;

[0119] A second preset calibration matrix with the highest second matrix score is determined as the extrinsic parameters of the monocular camera.

[0120] When determining the extrinsic parameters of a monocular camera in the context of autonomous driving, the present invention obtains a video image captured by the monocular camera and a depth image captured by a laser radar on the same vehicle as the monocular camera based on the RegNet network, extracts first feature information corresponding to the video image, projects the depth image to the coordinate system where the video image is located, obtains a projected image corresponding to the depth image, extracts second feature information corresponding to the projected image, splices the first feature information and the second feature information to obtain joint feature information, and uses the joint feature information to perform global regression on the projected image to obtain the extrinsic parameters of the monocular camera. This technical solution realizes automatic calibration of the monocular camera and the laser radar and automatic determination of the extrinsic parameters of the monocular camera, solves the technical problem of manual multi-sensor calibration, which is time-consuming, labor-intensive and inefficient, and not only improves the work efficiency of determining the extrinsic parameters of the monocular camera, but also saves labor costs and time.

[0121] like Figure 6 As shown, the present invention also provides an external parameter determination device for a monocular camera, the external parameter determination device for a monocular camera comprising:

[0122] An image acquisition module 310 is configured to acquire a video image captured by a monocular camera and a depth image captured by a laser radar on the same vehicle as the monocular camera; wherein the video image and the depth image are images captured by the monocular camera and the laser radar for the same environment;

[0123] A feature extraction module 320 is configured to extract first feature information corresponding to the video image, and project the depth image onto the coordinate system of the video image to obtain a projected image corresponding to the depth image; and project the depth image onto the coordinate system of the video image to obtain a projected image corresponding to the depth image, and extract second feature information corresponding to the projected image;

[0124] An information splicing module 330 is configured to splice the first feature information and the second feature information to obtain joint feature information;

[0125] The data regression module 340 is configured to perform global regression on the projected image using the joint feature information to obtain extrinsic parameters of the monocular camera.

[0126] The specific implementation of the device for determining extrinsic parameters of a monocular camera of the present invention is basically the same as the embodiments of the method for determining extrinsic parameters of a monocular camera described above, and will not be repeated here.

[0127] Furthermore, the present invention also provides a terminal device comprising: a memory, a processor, and a monocular camera external parameter determination program stored in the memory and executable on the processor, wherein the monocular camera external parameter determination program implements the steps of the above-mentioned monocular camera external parameter determination method when executed by the processor.

[0128] Furthermore, the present invention also provides a storage medium storing a program for determining the extrinsic parameters of a monocular camera. When the program for determining the extrinsic parameters of a monocular camera is executed by a processor, the steps of the above-mentioned method for determining the extrinsic parameters of a monocular camera are implemented.

[0129] It will be understood by those skilled in the art that embodiments of the present invention may be provided as methods, systems, or computer program products. Thus, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0130] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowcharts and / or block diagrams, as well as combinations of processes and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowcharts and / or block diagrams. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0131] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0132] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0133] It should be noted that in the claims, any reference signs placed between parentheses shall not be construed as limiting the claims. The word "comprising" does not exclude the presence of components or steps not listed in the claim. The word "a" or "an" preceding a component does not exclude the presence of a plurality of such components. The invention can be implemented by means of hardware comprising several different components and by means of a suitably programmed computer. In a unit claim enumerating several means, several of these means may be embodied by one and the same item of hardware. The use of the words first, second, third etc. does not indicate any order. These words may be interpreted as names.

[0134] Although the preferred embodiments of the present invention have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present invention.

[0135] Obviously, those skilled in the art may make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if such changes and modifications fall within the scope of the claims and their equivalents, the present invention is intended to include such changes and modifications.

Claims

1. A method for determining the external parameters of a monocular camera, characterized in that: The method comprises: Obtaining a video image captured by a monocular camera and a depth image captured by a laser radar on the same vehicle as the monocular camera; wherein the video image and the depth image are images captured by the monocular camera and the laser radar for the same environment; Extracting first feature information corresponding to the video image; Based on a preset initial calibration and an intrinsic parameter matrix of a monocular camera, projecting the depth image into a coordinate system where the video image is located to obtain a projected image corresponding to the depth image; extracting second feature information corresponding to the projection image; Extracting common feature information between the first feature information and the second feature information, so that some pixels on the projected image obtain depth points of the depth image; establishing a feature mapping between the first feature information and the second feature information based on the common feature information; splicing the first feature information and the second feature information based on the feature mapping to obtain joint feature information; Determining a plurality of first preset calibration matrices corresponding to the joint feature information and a first matrix score of each of the first preset calibration matrices using a pre-trained neural network; Determine a first preset calibration matrix having the highest first matrix score as the extrinsic parameters of the monocular camera; Iteratively updating the weight parameters of the neural network using the extrinsic parameters; When the number of iterations reaches a set number, using the deep neural network to determine a plurality of second preset calibration matrices corresponding to the joint feature information and a second matrix score of each of the second preset calibration matrices; A second preset calibration matrix with the highest second matrix score is determined as the extrinsic parameters of the monocular camera.

2. The method according to claim 1, wherein The step of acquiring a video image captured by a monocular camera and a depth image captured by a laser radar on the same vehicle as the monocular camera comprises: Obtain the original video images captured by the monocular camera and the point cloud data collected by the lidar; Converting the point cloud data into a sparse depth map; Normalizing the original video image to obtain the video image; The sparse depth map is normalized, and the depth image is generated according to the normalized sparse depth map.

3. The method according to claim 2, wherein The step of generating the depth image according to the normalized sparse depth map comprises: Performing maximum pooling on the normalized sparse depth map to obtain a semi-dense depth map; The semi-dense depth map is used as the depth image.

4. The method according to claim 1, wherein The step of projecting the depth image to the coordinate system where the video image is located based on the preset initial calibration and the intrinsic parameter matrix of the monocular camera to obtain a projected image corresponding to the depth image includes: Calibrate the depth image using the preset initial calibration and the intrinsic parameter matrix of the monocular camera; The depth image is projected onto the coordinate system of the calibrated video image to obtain the projected image.

5. A device for determining the external parameters of a monocular camera, characterized in that: The device for determining the external parameters of the monocular camera includes: An image acquisition module, configured to acquire a video image captured by a monocular camera and a depth image captured by a laser radar on the same vehicle as the monocular camera; wherein the video image and the depth image are images captured by the monocular camera and the laser radar for the same environment; a feature extraction module, configured to extract first feature information corresponding to the video image, and project the depth image onto a coordinate system where the video image is located based on a preset initial calibration and an intrinsic parameter matrix of a monocular camera to obtain a projected image corresponding to the depth image, wherein the preset initial calibration is randomly given, and extract second feature information corresponding to the projected image; an information splicing module, configured to extract common feature information between the first feature information and the second feature information so that some pixels on the projected image obtain depth points of the depth image; establish a feature mapping between the first feature information and the second feature information based on the common feature information; and splice the first feature information and the second feature information based on the feature mapping to obtain joint feature information; A data regression module is configured to use a pre-trained neural network to determine multiple first preset calibration matrices corresponding to the joint feature information and a first matrix score of each of the first preset calibration matrices; determine the first preset calibration matrix with the highest first matrix score as the extrinsic parameter of the monocular camera; use the extrinsic parameters to iteratively update the weight parameters of the neural network; when the number of iterations reaches a set number, use the deep neural network to determine multiple second preset calibration matrices corresponding to the joint feature information and a second matrix score of each of the second preset calibration matrices; and determine the second preset calibration matrix with the highest second matrix score as the target extrinsic parameter of the monocular camera.

6. A terminal device, characterized in that: include: A memory, a processor, and a monocular camera extrinsic parameter determination program stored in the memory and executable on the processor, wherein the monocular camera extrinsic parameter determination program, when executed by the processor, implements the steps of the monocular camera extrinsic parameter determination method according to any one of claims 1 to 4.

7. A storage medium, characterized in that: A monocular camera extrinsic parameter determination program is stored thereon, and when the monocular camera extrinsic parameter determination program is executed by a processor, the steps of the monocular camera extrinsic parameter determination method according to any one of claims 1 to 4 are implemented.

Images