A method for spatial extrinsic parameter calibration of binocular cameras and lidar based on backpropagation neural networks.

Abstract

This invention discloses a spatial extrinsic parameter calibration method for a binocular camera and a lidar based on a backpropagation (BP) neural network, belonging to the field of sensor calibration. The method involves: synchronously acquiring left and right image information and point cloud information of N calibration boards at different locations using a lidar and a binocular camera with known intrinsic parameters. Corner points of the calibration boards are extracted from the left and right images, and the coordinate parameters of the calibration board's center point in the left pixel coordinate system are calculated. The point cloud data obtained from the lidar is segmented and filtered, and the coordinate parameters of the calibration board's center point in the lidar coordinate system are calculated. The BP neural network model is optimized by constructing a loss function. The BP neural network model is iteratively trained to obtain the estimated extrinsic parameters of the lidar and the binocular camera. Baseline length error combined with reprojection error is introduced as an evaluation index to improve the accuracy and effectiveness of the calibration result evaluation.

Description

TECHNICAL FIELD

[0001] The application relates to a method for calibrating the spatial external parameters of a binocular camera and a laser radar, in particular to a method for calibrating the spatial external parameters of a binocular camera and a laser radar based on a BP neural network, and belongs to the field of sensor calibration. BACKGROUND

[0002] In recent years, with the rapid development of unmanned system technology, the environmental perception technology of unmanned systems has also developed greatly. Laser radar-binocular camera systems are increasingly applied to various scenes. Laser radar can effectively obtain the depth information of the surrounding environment, and has high measurement accuracy and strong anti-interference capability. However, laser radar cannot obtain color and texture information. Cameras can obtain color and texture information, and binocular cameras can obtain the depth information of the co-view part of the left and right cameras, but cameras are susceptible to light and have poor depth information accuracy. Therefore, the two sensors form a strong complementary relationship, and the environmental perception system that fuses laser radar and binocular cameras can obtain better detection effect than a single sensor. The premise of data fusion of the two sensors is accurate external parameter calibration of the two sensors. Therefore, accurate and robust calibration of the binocular camera and the laser radar has important research value and significance.

[0003] The existing calibration method of laser radar and binocular camera mainly uses PnP to calibrate the left and right cameras and the laser radar respectively to obtain the external parameters, and the baseline length and the re-projection accuracy obtained are limited. Currently, there are few methods for simultaneously calibrating binocular cameras and lasers using neural networks. The traditional separate calibration method is difficult to obtain external parameters with small re-projection error and baseline length error. Moreover, the error of the traditional PnP method will significantly increase when the number of available calibration board positions is small, and the robustness is not strong. Therefore, it is necessary to develop a method that can effectively reduce the re-projection error and baseline length error of the external parameters of the binocular camera and the laser radar calibration. SUMMARY

[0004] To solve the problem that the error will significantly increase and the robustness is not strong when the number of available calibration board positions is small in the prior art, the main purpose of the present application is to provide a method for calibrating the spatial external parameters of a binocular camera and a laser radar based on a BP neural network, which can effectively reduce the re-projection error and baseline length error of the external parameters of the binocular camera and the laser radar calibration, improve the efficiency of the calibration of the spatial external parameters of the binocular camera and the laser radar, and improve the accuracy and robustness of the calibration results for sensor information fusion.

[0005] The purpose of the present application is achieved by the following technical solutions:

[0006] The application discloses a binocular camera and laser radar space external parameter calibration method based on a BP neural network, which comprises the following steps.

[0007] The application discloses a binocular camera and laser radar space external parameter calibration method based on a BP neural network, which comprises the following steps.

[0008] S1. Setting a laser radar, a binocular camera and a chessboard calibration board, collecting left and right image information and point cloud information of N groups of calibration boards at different positions by the laser radar and the binocular camera with known internal parameters synchronously. n ,…,P N} and P n represents the calibration board center point of the nth group. The N groups of data contain N groups of calibration board corner points C = {C1, C2, … C n ,…,C N} and C n represents the calibration board corner point of the nth group, and the nth group has X corner points C n ={C n1 ,C n2 ,…C nx ,…,C nX}.

[0009] In order to improve the calibration accuracy, preferably, N is greater than or equal to 6, and X is 35.

[0010] S2. Extracting the corner points of the calibration board in the left image, obtaining the coordinates (u l ,v l ) of the N groups of calibration board center points P in the left pixel coordinate system. Extracting the corner points of the calibration board in the right image, obtaining the coordinates (ur ,v r ); using PnP method and coordinate system conversion to calculate N sets of center points P of the calibration board in the left camera coordinate system (x Cl ,y Cl ,z Cl ) and in the right camera coordinate system (x Cr ,y Cr ,z Cr ).

[0011] S21. Obtain the coordinates of the center point of each set of data calibration board and each corner point in the pixel coordinate system, define the coordinates of the center point of the calibration board in the world coordinate system, and calculate the coordinates of each corner point in the world coordinate system.

[0012] For each set of data left image, use the detection function to extract the corner points in the checkerboard calibration board, obtain the coordinates of each corner point C nx in the left pixel coordinate system, and obtain the center point P n of each set of data calibration board in the left pixel coordinate system through the positional relationship between the corner points (u nl ,v nl ), and finally obtain the coordinates of N sets of checkerboard calibration board center points P in the left pixel coordinate system (u l ,v l ); for each set of data right image, use the detection function to extract the corner points C in the checkerboard calibration board, which can obtain the coordinates of each corner point C nx in the right pixel coordinate system, and obtain the center point P n of each set of data calibration board in the right pixel coordinate system (u nr ,v nr ), and finally obtain the coordinates of N sets of checkerboard calibration board center points P in the left pixel coordinate system (u r ,v r ). Define the coordinates of the center point P n of the calibration board in the world coordinate system as (0, 0, 0), the X axis of the world coordinate system is parallel to the short side of the calibration board, the Y axis of the world coordinate system is parallel to the long side of the calibration board, and the Z axis of the world coordinate system points from the non-checkerboard surface of the calibration board to the checkerboard surface. According to the length of the checkerboard calibration board grid, the coordinates of each corner point C nx in the world coordinate system are calculated.

[0013] S22. For each set of data, using PnP method, according to the coordinates of each corner point C nx in the left pixel coordinate system and its corresponding coordinates in the world coordinate system, the left camera coordinate system (O cl X cl Ycl Z cl ) relative to the world coordinate system (O w X w Y w Z w ) relative to the world coordinate system (O translation vector For each group of data, using the PnP method, the coordinates of each corner point C nx in the right pixel coordinate system and its corresponding coordinates in the world coordinate system are used to solve the rotation matrix of the right camera coordinate system (O cr X cr Y cr Z cr ) relative to the world coordinate system (O w X w Y w Z w ) relative to the world coordinate system (O translation vector

[0014] S23. Calculate the coordinates (x Cl ,y Cl ,z Cl ) of the N sets of checkerboard center points P in the left camera coordinate system and the coordinates (x Cr ,y Cr ,z Cr ) in the right camera coordinate system through coordinate system conversion.

[0015] For each group of data, multiply the coordinates (0, 0, 0) of the center point P n of the calibration board in the world coordinate system by the extrinsic matrix of the left camera coordinate system relative to the world coordinate system to obtain the coordinates (x nCl ,y nCl ,z nCl ) of the center point P n of the calibration board in the left camera coordinate system. Finally, the coordinates (x Cl ,y Cl ,z Cl ) of the N sets of checkerboard center points P in the left camera coordinate system are obtained. The extrinsic matrix is obtained from the rotation matrix translation vector in S22.

[0016]

[0017]

[0018]

[0019] For each set of data, the center point P of the calibration board for that set of data is determined. n The coordinates (0,0,0) in the world coordinate system and the extrinsic parameter matrix of the right camera coordinate system relative to the world coordinate system for this set of data. Multiplying the data yields the center point P of the calibration board for this set of data. n Coordinates (x) in the right camera coordinate system nCl ,y nCl ,z nCl ), and then obtain the coordinates (x, y) of the center point P of the N sets of chessboard calibration boards in the right camera coordinate system. Cr ,y Cr ,z Cr ). Extrinsic parameter matrix Rotation matrix in S22 Translation vector get.

[0020]

[0021]

[0022]

[0023] To ensure the accuracy of corner point extraction, as a preferred method, the cv2.findChessboardCorners detection function is used to extract the corner points of the chessboard calibration board from the left and right images.

[0024] S3. Acquire point cloud data of the calibration board using a lidar system; apply a pass-through filter to each set of data to remove irrelevant point clouds, retaining only the valid calibration board point cloud. Based on the valid calibration board point cloud, fit the plane where the calibration board is located using a random sampling consensus method, projecting the calibration board point cloud onto this plane to form a new point cloud. Fit the spatial equations of the four sides of the new calibration board point cloud, find the intersection points of the four sides, which form a quadrilateral. Solve for the intersection points of the two diagonals of the quadrilateral using geometric relationships; these intersection points are the center point P of the calibration board for this set of data. n The coordinates in the lidar coordinate system are then used to obtain the coordinates (x, y) of the center point P of the N calibration plates in the lidar coordinate system. L ,y L ,z L ).

[0025] S4. Define the left camera coordinate system, the right camera coordinate system, and the LiDAR coordinate system. Define the Euler angles and translation vectors used for rotating the LiDAR coordinate system relative to the left camera coordinate system, and represent the extrinsic parameter matrix of the LiDAR coordinate system relative to the left camera coordinate system using the aforementioned Euler angles and translation vectors. The extrinsic matrix The spatial extrinsic parameter of the lidar and the left camera. The Euler angles and the translation vector used for rotating the lidar coordinate system relative to the right camera coordinate system are defined, and the extrinsic parameter matrix of the lidar coordinate system relative to the right camera coordinate system is represented by the Euler angles and the translation vector The extrinsic parameter matrix The spatial extrinsic parameter of the lidar and the right camera.

[0026] In the transformation process of the lidar coordinate system to the left and right camera coordinate systems, three coordinate systems are involved, which are the left camera coordinate system, the right camera coordinate system and the lidar coordinate system. The point cloud collected by the lidar is a point in the lidar coordinate system, which needs to be converted into a point in the camera coordinate system through rotation and translation transformation.

[0027] The rotation and translation of the lidar coordinate system relative to the left camera coordinate system can be represented by the rotation matrix and the translation vector is a 3x3 vector, t xl , t yl , t zl The three elements are independent of each other.

[0028]

[0029] is a 3x3 matrix, but the 9 elements are not independent of each other. The above 9 elements are decoupled by using Euler angles. The method of expressing the rotation matrix as 3 independent Euler angles decouples the 9 variables, reduces the variables, and simplifies the BP neural network structure. The rotation matrix is converted to the following form by Euler angles:

[0030]

[0031]

[0032] An extrinsic parameter matrix can be represented by the six parameters of φ l , θ l , ψ l , t xl , t yl , t zl The extrinsic parameter matrix is used to convert a three-dimensional point between the lidar coordinate system and the left camera coordinate system, which is the spatial extrinsic parameter of the lidar and the left camera.

[0033]

[0034] ​The rotation and translation of the LiDAR coordinate system relative to the right camera coordinate system can be represented by the rotation matrix and the translation vector is a 3x 3 vector, t xr , t yr , t zr are independent of each other.

[0035]

[0036] is a 3x 3 matrix, but the 9 elements are not independent of each other. The 9 elements are decoupled by using the Euler angle representation. By expressing the rotation matrix as a form of 3 independent Euler angles, the number of variables is reduced, and the BP neural network structure is simplified. The method of expressing the rotation matrix as a form of 3 independent Euler angles decouples the 9 variables, reduces the number of variables, and further simplifies the BP neural network structure. By Euler angles, the rotation matrix is converted to the following form:

[0037]

[0038]

[0039] A extrinsic matrix can be represented by φ r , θ r , ψ r , t xr , t yr , t zr is used to convert three-dimensional points between the LiDAR coordinate system and the left camera coordinate system, i.e. the spatial extrinsic parameters of the LiDAR and the left camera.

[0040]

[0041] For convenience of calculation, as a preferred, Euler angles in the order of Z-Y-X are used.

[0042] S5. Introduce baseline length error re-projection error and the spatial projection error of the center point of the calibration board in the camera coordinate system ​​And build a BP neural network model and loss function: due to the input and output characteristics of binocular camera and laser radar calibration, as preferred, the BP neural network model includes a 3-layer structure: input layer, intermediate hidden layer and output layer respectively; the input layer contains 3 neurons, the hidden layer contains 6 neurons and 6 biases, and the output layer contains 5 neurons. The input layer is the coordinate value of the center point of the calibration board in the laser radar coordinate system, the hidden layer obtains the coordinate value of the center point of the calibration board in the left camera coordinate system calculated by the model (x ClP ,y ClP ,z ClP ) and the coordinate value of the center point of the calibration board in the left camera coordinate system calculated by the model (x CrP ,y CrP ,z CrP ), and the output layer is the coordinate (u lP ,v lP ) of the center point of the calibration board in the left pixel coordinate system calculated by the model, the coordinate (u rP ,v rP ) of the center point of the calibration board in the right pixel coordinate system calculated by the model and the calculated baseline length Baseline P . The baseline length error , the re-projection error and the spatial projection error of the center point of the calibration board in the camera coordinate system Baseline T is the actual length of the baseline of the binocular camera, (x ClP ,y ClP ,z ClP ) is the coordinate value of the center point of the calibration board in the left camera coordinate system, (x CrP ,y CrP ,z CrP ) is the coordinate value of the center point of the calibration board in the left camera coordinate system, (u r ,v r ) is the coordinate of the center point of the calibration board in the left pixel coordinate system, (u r ,v r ) is the coordinate of the center point of the calibration board in the right pixel coordinate system. Through three proportional coefficients α, β, χ, a loss function is constructed which contains the baseline length error, the re-projection error and the spatial projection error of the center point of the calibration board in the camera coordinate system In the BP neural network, the loss function is introduced into the synchronous back propagation from the hidden layer to the input layer, the baseline length error is reduced by the synchronous back propagation of the baseline length error in the loss function, and the re-projection error can be reduced; the re-projection error is further eliminated by the synchronous back propagation of the re-projection error in the loss function; that is, the precision of the spatial extrinsic parameter calibration of the binocular camera and the laser radar is improved by introducing the loss function into the synchronous back propagation from the hidden layer to the input layer. The mapping relationship of the left and right cameras is constructed in the hidden layer by the loss function, and the training efficiency and the spatial extrinsic parameter calibration precision are improved by synchronously training the binocular camera data and the laser radar data by fusing the mapping relationship of the left and right cameras constructed in the hidden layer by the loss function.

[0043] The baseline length Baseline P As shown in formula (15).

[0044]

[0045] The baseline length error is introduced The re-projection error And the spatial projection error of the center point of the calibration board in the camera coordinate system Baseline T is the actual length of the baseline of the binocular camera, (x ClP , y ClP , z ClP ) is the coordinate value of the center point of the calibration board in the left camera coordinate system, (x CrP , y CrP , z CrP ) is the coordinate value of the center point of the calibration board in the left camera coordinate system, (u r , v r ) is the coordinate of the center point of the calibration board in the left pixel coordinate system, (u r , v r ) is the coordinate of the center point of the calibration board in the right pixel coordinate system.

[0046]

[0047]

[0048]

[0049] The loss function containing the baseline length error, the re-projection error and the spatial projection error of the center point of the calibration board in the camera coordinate system is constructed by three scale coefficients α, β, χ The loss function As shown in formula (19).

[0050]

[0051] S6. Configure the parameters required for network training and train. Set the learning rate, batch size, weight initialization method, weight decay coefficient, maximum number of iterations, and loss function target size. Input the coordinates of the N sets of center points P of the calibration board in the laser radar coordinate system (x L ,y L ,z L ), the coordinates in the left and right camera coordinate systems, and the coordinates in the left and right pixel coordinate systems into the improved BP neural network for training. Update the weights during training, and obtain the extrinsic matrix of the laser radar coordinate system to the left camera coordinate system and the extrinsic matrix of the laser radar coordinate system to the right camera coordinate system after the loss function meets the conditions, that is, the spatial extrinsic calibration of the binocular camera and the laser radar is realized based on the BP neural network.

[0052] It also includes S7. The extrinsic matrix of the laser radar and the left and right cameras obtained in S6 and Project the laser point cloud into the left and right camera coordinate systems respectively and calculate the re-projection error Calculate the baseline length and the baseline length of the camera itself through the extrinsic matrix and , and obtain the baseline length error The two indicators are used to comprehensively evaluate the results of the spatial extrinsic calibration of the binocular camera and the laser radar, that is, by introducing the baseline length error and the re-projection error as evaluation indicators, the accuracy and effectiveness of the evaluation of the calibration results are improved, and the accuracy and robustness of the calibration results for sensor information fusion are improved.

[0053] Advantages:

[0054] 1. For the problem that the baseline length and the re-projection accuracy obtained by separately calibrating the left and right cameras and the laser radar to obtain the extrinsic parameters are limited, the disclosed binocular camera and laser radar spatial extrinsic calibration method based on BP neural network effectively reduces the re-projection error and the baseline length error of the laser radar and the left and right cameras in the spatial extrinsic calibration when the number of poses is small, and improves the spatial extrinsic calibration accuracy of the binocular camera and the laser radar.

[0055] 2. For the problem that the number of unknown quantities in the rotation matrix is large and not mutually independent during the spatial extrinsic calibration, the disclosed binocular camera and laser radar spatial extrinsic calibration method based on BP neural network decouples the 9 variables of the rotation matrix, reduces the variables, simplifies the BP neural network structure, and improves the efficiency of the spatial extrinsic calibration of the binocular camera and the laser radar.

[0056] 3. Addressing the issue of the limited evaluation methods for current space extrinsic parameter calibration results, this invention discloses a space extrinsic parameter calibration method for binocular cameras and lidar based on a BP neural network. This method introduces baseline length error in conjunction with reprojection error as a new evaluation metric. Specifically, the baseline length of the binocular camera is indirectly calculated based on the extrinsic parameters of the lidar and the left and right cameras, and then subtracted from the actual baseline length. By using baseline length error and reprojection error, the calibration results can be effectively and reliably evaluated, resulting in higher accuracy and stronger robustness when using the calibration results for sensor information fusion in subsequent applications. Attached Figure Description

[0057] Figure 1 This is a flowchart of a spatial extrinsic parameter calibration method for a binocular camera and lidar based on a BP neural network according to the present invention.

[0058] Figure 2 This is the checkerboard calibration plate used in this invention;

[0059] Figure 3 This is a schematic diagram illustrating the definitions of the left camera coordinate system, right camera coordinate system, and lidar coordinate system used in this invention.

[0060] Figure 4 The ZYX sequential Euler angle form used in this invention;

[0061] Figure 5 This is the network structure of the BP neural network of the present invention;

[0062] Figure 6 This is a flowchart of the joint calibration method based on BP neural network of the present invention;

[0063] Figure 7 This is the backpropagation mode of the BP neural network of this invention.

[0064] Figure 8 The baseline length error obtained by the BP neural network algorithm of this invention is compared with the baseline length error obtained by the PnP algorithm. The vertical axis represents the baseline error length (mm), and the horizontal axis represents the experiment number.

[0065] Figure 9 This graph compares the left camera reprojection error obtained by the BP neural network algorithm of this invention with the left camera reprojection error obtained by the PnP algorithm. The vertical axis represents the camera reprojection error (pixel), and the horizontal axis represents the experiment number.

[0066] Figure 10 The image shows a comparison between the right camera reprojection error obtained by the BP neural network algorithm and the right camera reprojection error obtained by the PnP algorithm. The vertical axis represents the camera reprojection error (pixel), and the horizontal axis represents the experiment number. Detailed Implementation

[0067] To better illustrate the purpose and advantages of the present invention, the invention will be further described below with reference to the accompanying drawings and examples:

[0068] like Figure 1 As shown in the figure, this embodiment discloses a spatial extrinsic parameter calibration method for binocular cameras and lidar based on BP neural networks. The specific implementation steps are as follows:

[0069] (I) Preparation of calibration materials and setup of calibration scenario

[0070] like Figure 2 As shown, this invention uses a checkerboard calibration board as the calibration object. The checkerboard calibration board measures 594mm × 841mm, and each checkerboard square has a side length of 95mm. The calibration scene should be open and free of clutter, and there should be no other objects within 1m of the calibration object. Due to the scanning and data characteristics of LiDAR, the four sides of the calibration board should not be parallel to the ground; otherwise, the calibration results will be inaccurate.

[0071] (II) Spatial Exterior Parameter Calibration of Binocular Camera and LiDAR

[0072] like Figure 3 As shown, this process involves the left camera coordinate system (O). Cl X Cl Y Cl Z Cl ), right camera coordinate system (O) Cr X Cr Y Cr Z Cr ) and lidar coordinate system (O L X L Y L Z L ).

[0073] S1. Set up the LiDAR, binocular camera, and checkerboard calibration board. The LiDAR used is a Velodyne HDL-64ES2 model; the binocular camera is a ZED binocular camera with a baseline length of 120mm. Simultaneously acquire N sets of left and right image information and point cloud information of the calibration board at different locations using the LiDAR and the binocular camera with known intrinsic parameters. The N sets of data contain N calibration board center points P = {P1, P2, ..., P...} n ,…,P N}, P n The point represents the center point of the calibration plate in the nth group. The N groups of data contain N groups of calibration plate corner points C = {C1, C2, ... C...} n ,…,C N}, C n C represents the corner point of the calibration plate in group n, and there are X corner points in group n. n ={C n1 Cn2 ,…C nx ,…,C nX To improve calibration accuracy, N≥6, X=35.

[0074] S2. Extract the corner points of the calibration board in the left image to obtain N sets of coordinates (u, u) of the calibration board center point P in the left pixel coordinate system. l ,v l Extract the corner points of the calibration board from the right image to obtain N sets of coordinates (u, u) of the calibration board center point P in the right pixel coordinate system. r ,v r The PnP method and coordinate system transformation are used to calculate the coordinates (x, y) of N sets of calibration plate center point P in the left camera coordinate system. Cl ,y Cl ,z Cl ) and the coordinates (x) in the right camera coordinate system Cr ,y Cr ,z Cr ):

[0075] S21. Obtain the coordinates of the center point of the calibration board and each corner point of each data set in the pixel coordinate system, define the coordinates of the center point of the calibration board in the world coordinate system, and calculate the coordinates of each corner point of each data set in the world coordinate system.

[0076] For each set of data, the detection function is used to extract the corner points of the checkerboard calibration board from the left image, thus obtaining C for each corner point in each set of data. nx The center point P of each data calibration board is obtained by using the coordinates in the left pixel coordinate system and the positional relationship between the corner points. n Coordinates in the left pixel coordinate system (u) nl ,v nl Finally, the coordinates (u) of the center point P of the N sets of chessboard calibration boards in the left pixel coordinate system are obtained. l ,v l For each set of data, the detection function is used to extract the corner points C in the chessboard calibration board from the right image, thus obtaining the corner point C for each set of data. nx The center point P of each data calibration board is obtained by using the coordinates in the right pixel coordinate system and the positional relationship between the corner points. n Coordinates in the right pixel coordinate system (u) nr ,v nr Finally, the coordinates (u) of the center point P of the N sets of chessboard calibration boards in the left pixel coordinate system are obtained. r ,v r Define the center point P of the calibration plate. nCoordinates in the world coordinate system. In order to ensure the accuracy of the corner extraction, the left and right images are detected using the cv2.findChessboardCorners function to extract the corners of the chessboard calibration board. nx Coordinates in the world coordinate system. In order to ensure the accuracy of the corner extraction, the left and right images are detected using the cv2.findChessboardCorners function to extract the corners of the chessboard calibration board.

[0077] S22. For each set of data, use the PnP method to use each corner point C nx Coordinates in the left pixel coordinate system and its corresponding coordinates in the world coordinate system, and the left camera coordinate system (O cl X cl Y cl Z cl ) relative to the world coordinate system (O w X w Y w Z w ) rotation matrix Translation vector For each set of data, use the PnP method to use each corner point C nx Coordinates in the right pixel coordinate system and its corresponding coordinates in the world coordinate system, and the right camera coordinate system (O cr X cr Y cr Z cr ) relative to the world coordinate (O w X w Y w Z w ) rotation matrix Translation vector

[0078] S23. Calculate the coordinates (x Cl ,y Cl ,z Cl ) of the center point P of the N sets of chessboard calibration board in the left camera coordinate system and the coordinates (x Cr ,y Cr ,z Cr ) in the right camera coordinate system through coordinate system conversion.

[0079] For each set of data, multiply the center point P of the calibration board in this set of data n Coordinates in the world coordinate system (0, 0, 0) and the extrinsic matrix of the left camera coordinate system relative to the world coordinate system , get the center point P of the calibration board in this set of data nCoordinates (x) in the left camera coordinate system nCl ,y nCl ,z nCl Finally, the coordinates (x, y) of the center point P of the N sets of chessboard calibration boards in the left camera coordinate system are obtained. Cl ,y Cl ,z Cl ). Extrinsic parameter matrix Rotation matrix in S22 Translation vector get.

[0080]

[0081]

[0082]

[0083] For each set of data, the center point P of the calibration board for that set of data is determined. n The coordinates (0,0,0) in the world coordinate system and the extrinsic parameter matrix of the right camera coordinate system relative to the world coordinate system for this set of data. Multiplying the data yields the center point P of the calibration board for this set of data. n Coordinates (x) in the right camera coordinate system nCl ,y nCl ,z nCl Finally, the coordinates (x, y) of the center point P of the N sets of chessboard calibration boards in the right camera coordinate system are obtained. Cr ,y Cr ,z Cr ). Extrinsic parameter matrix Rotation matrix in S22 Translation vector get.

[0084]

[0085]

[0086]

[0087] S3. Acquire point cloud data of the calibration board using a lidar system; apply a pass-through filter to each set of data to remove irrelevant point clouds, retaining only the valid calibration board point cloud. Based on the valid calibration board point cloud, fit the plane where the calibration board is located using a random sampling consensus method, projecting the calibration board point cloud onto this plane to form a new point cloud. Fit the spatial equations of the four sides of the new calibration board point cloud, find the intersection points of the four sides, which form a quadrilateral. Solve for the intersection points of the two diagonals of the quadrilateral using geometric relationships; these intersection points are the center point P of the calibration board for this set of data. nCoordinates in the laser radar coordinate system. Finally, N sets of coordinates of the center points P of the calibration plates in the laser radar coordinate system (x L ,y L ,z L ) are obtained.

[0088] S4. Define the left camera coordinate system, the right camera coordinate system, and the laser radar coordinate system. Define the Euler angles and the translation vector used for the rotation of the laser radar coordinate system relative to the left camera coordinate system, and express the extrinsic matrix of the laser radar coordinate system relative to the left camera coordinate system by using the above-mentioned Euler angles and translation vector The extrinsic matrix is the spatial extrinsic parameter of the laser radar and the left camera. Define the Euler angles and the translation vector used for the rotation of the laser radar coordinate system relative to the right camera coordinate system, and express the extrinsic matrix of the laser radar coordinate system relative to the right camera coordinate system by using the above-mentioned Euler angles and translation vector The extrinsic matrix is the spatial extrinsic parameter of the laser radar and the right camera.

[0089] In the transformation process from the laser radar coordinate system to the left and right camera coordinate systems, three coordinate systems are involved, namely the left camera coordinate system, the right camera coordinate system, and the laser radar coordinate system. The point cloud collected by the laser radar is in the laser radar coordinate system, and needs to be converted into a point in the camera coordinate system through rotation and translation.

[0090] The rotation and translation of the laser radar coordinate system relative to the left camera coordinate system can be represented by the rotation matrix and the translation vector is a 3x 3 vector, t xl , t yl , and t zl are independent of each other.

[0091]

[0092] is a 3x 3 matrix, but the 9 elements are not independent of each other. The above-mentioned 9 elements are decoupled by using the Euler angle expression. The method of expressing the rotation matrix as 3 independent Euler angles decouples the 9 variables, reduces the variables, and simplifies the BP neural network structure. The rotation matrix is converted into the following form by using Euler angles:

[0093]

[0094]

[0095] φ l , θ​l , ψ l , t xl , t yl , t zl Six parameters can represent an extrinsic parameter matrix. extrinsic parameter matrix Used to transform 3D points between the lidar coordinate system and the left camera coordinate system. That is, the spatial extrinsic parameters of the lidar and the left camera.

[0096]

[0097] The rotation and translation of the lidar coordinate system relative to the right camera coordinate system can be achieved using rotation matrices. Translation vector express. It is a 3x3 vector, t xr , t yr , t zr The three elements are independent of each other.

[0098]

[0099] It is a 3x3 matrix, but its 9 elements are not independent. Euler angles are used to decouple these 9 elements. By expressing the rotation matrix as 3 independent Euler angles, the number of variables is reduced, simplifying the BP neural network structure. The method of expressing the rotation matrix as 3 independent Euler angles decouples the 9 variables, reduces the number of variables, and simplifies the BP neural network structure. The rotation matrix is ​​expressed using Euler angles. Transform into the following form:

[0100]

[0101]

[0102] From φ r θ r , ψ r , t xr , t yr , t zr Six parameters can represent an extrinsic parameter matrix. extrinsic parameter matrix Used to transform 3D points between the lidar coordinate system and the left camera coordinate system. That is, the spatial extrinsic parameters of the lidar and the left camera.

[0103]

[0104] like Figure 4Euler angles in the order of Z-Y-X are used.

[0105] S5. Introducing baseline length error re-projection error and spatial projection error of the center point of the calibration board in the camera coordinate system and loss function: Since the input and output characteristics of the binocular camera and the laser radar calibration, such as Figure 5 As shown in the figure, the BP neural network model includes a 3-layer structure: input layer, intermediate hidden layer, and output layer; the input layer contains 3 neurons, the hidden layer contains 6 neurons and 6 biases, and the output layer contains 5 neurons. The input layer is the coordinate value of the center point of the calibration board in the laser radar coordinate system, the hidden layer obtains the coordinate value of the center point of the calibration board in the left camera coordinate system calculated by the model (x ClP ,y ClP ,z ClP ) and the coordinate value of the center point of the calibration board in the left camera coordinate system calculated by the model (x CrP ,y CrP ,z CrP ), and the output layer is the coordinate of the center point of the calibration board in the left pixel coordinate system calculated by the model (u lP ,v lP ), the coordinate of the center point of the calibration board in the right pixel coordinate system calculated by the model (u rP ,v rP ) and the calculated baseline length Baseline P .

[0106]

[0107] Introducing baseline length error re-projection error and spatial projection error of the center point of the calibration board in the camera coordinate system Baseline T is the actual length of the baseline of the binocular camera, (x ClP ,y ClP ,z ClP ) is the coordinate value of the center point of the calibration board in the left camera coordinate system, (x CrP ,y CrP ,z CrP ) is the coordinate value of the center point of the calibration board in the left camera coordinate system, (u r ,v r ) is the coordinate of the center point of the calibration board in the left pixel coordinate system, and (u r ,v r ) is the coordinate of the center point of the calibration board in the right pixel coordinate system.

[0108]

[0109]

[0110]

[0111] The loss function including the baseline length error, the re-projection error and the spatial projection error of the center point of the calibration board in the camera coordinate system is constructed by three scale coefficients a, b, c

[0112]

[0113] In the BP neural network, the loss function is introduced into the synchronous back propagation from the hidden layer to the input layer, the baseline length error is reduced by the synchronous back propagation of the baseline length error in the loss function, and the re-projection error can be reduced; the re-projection error is further eliminated by the synchronous back propagation of the re-projection error in the loss function; that is, the precision of the spatial extrinsic parameter calibration of the binocular camera and the laser radar is improved by introducing the loss function into the synchronous back propagation from the hidden layer to the input layer. And the mapping relationship of the left and right cameras is constructed in the hidden layer by the loss function, and the training of the binocular camera data and the laser radar data is synchronized by the fusion of the mapping relationship of the left and right cameras constructed in the hidden layer by the loss function, so as to improve the training efficiency and the precision of the spatial extrinsic parameter calibration.

[0114] S6. Configuration of network training parameters and training. Set the learning rate, batch size, weight initialization method, weight decay coefficient, maximum iteration number and loss function target size. Input the coordinates (x L ,y L ,z L ) of N sets of center points P of the calibration board in the laser radar coordinate system, the coordinates in the left and right camera coordinate systems, and the coordinates in the left and right pixel coordinate systems into the improved BP neural network for training. As shown in Figure 6 Figure 7 , the weight is updated during training, and the extrinsic parameter matrix of the laser radar coordinate system to the left camera coordinate system and the extrinsic parameter matrix of the laser radar coordinate system to the right camera coordinate system

[0115] (III) Calibration result evaluation

[0116] According to the extrinsic parameter matrices of the laser radar and the left and right cameras obtained in S6 and , the laser point cloud is projected into the left and right camera coordinate systems respectively, and the re-projection error is calculated , the baseline length error is obtained by comparing the baseline length with the baseline length of the camera itself through the extrinsic parameters and ​ The evaluation method can effectively and reliably evaluate the calibration result, and the accuracy is higher and the robustness is stronger when the calibration result is used for subsequent sensor information fusion.

[0117] As shown in Figure 8 , the baseline length error obtained by the BP neural network algorithm of the application is compared with the baseline length error obtained by the PnP algorithm. Figure 9 As shown in Figure 10 , the right camera reprojection error obtained by the BP neural network algorithm of the application is compared with the right camera reprojection error obtained by the PnP algorithm.

[0118] The above specific description further details the purpose, technical scheme and beneficial effects of the application, and it should be understood that the above description is only a specific embodiment of the application and is not used to limit the protection scope of the application, and any modification, equivalent replacement, improvement, etc. within the spirit and principles of the application should be included in the protection scope of the application.

Claims

1. A method for calibrating space parameters of a binocular camera and a laser radar based on a BP neural network, characterized in that: The method comprises the following steps, S1. Set up the laser radar, binocular camera and checkerboard calibration board; collect N sets of left and right image information and point cloud information of the calibration board at different positions by the laser radar and the binocular camera with known internal parameters; N sets of data contain N center points P = {P1, P2,..., P n ,...,P N} of the calibration board, P n n represents the center point of the nth set of calibration board; N sets of data contain N sets of corner points C = {C1, C2,..., C n ,...,C N} of the calibration board, C n n represents the corner point of the nth set of calibration board, and the nth set has X corner points C n = {C n1 ,C n2 ,...C nx ,...,C nX}. S2. Extract the corner points of the calibration board in the left image to obtain the coordinates (u l ,v l ) of the N sets of center points P of the calibration board in the left pixel coordinate system; extract the corner points of the calibration board in the right image to obtain the coordinates (u r ,v r ) of the N sets of center points P of the calibration board in the right pixel coordinate system; use the PnP method and coordinate system conversion to solve the coordinates (x Cl ,y Cl ,z Cl ) of the N sets of center points P of the calibration board in the left camera coordinate system and the coordinates (x Cr ,y Cr ,z Cr ) of the N sets of center points P of the calibration board in the right camera coordinate system; S3. Obtain the point cloud data of the calibration board by using the laser radar; use straight-through filtering to remove irrelevant point clouds for each group of data, and only keep the effective calibration board point cloud; according to the effective calibration board point cloud, use the random sample consensus method to fit the plane where the calibration board is located, project the point cloud of the calibration board onto the plane to form a new point cloud; fit the spatial equation of the four edges of the new calibration board point cloud, find the intersection point of the four edges, the intersection point of the four edges constitutes a quadrilateral, and the intersection point of the two diagonal lines of the quadrilateral is obtained through geometric relationship, which is the center point P of the calibration board of the group of data n In the coordinate system of the laser radar, and then obtain the coordinates (x L ,y L ,z L ) of the N groups of center points P of the calibration board in the coordinate system of the laser radar; S4. Define the left camera coordinate system, the right camera coordinate system and the laser radar coordinate system; define the Euler angles and the translation vector used for rotating the laser radar coordinate system relative to the left camera coordinate system, and express the extrinsic matrix of the laser radar coordinate system relative to the left camera coordinate system through the above-mentioned Euler angles and translation vector The extrinsic matrix i.e. the spatial extrinsic parameter of the laser radar and the left camera; define the Euler angles and the translation vector used for rotating the laser radar coordinate system relative to the right camera coordinate system, and express the extrinsic matrix of the laser radar coordinate system relative to the right camera coordinate system through the above-mentioned Euler angles and translation vector The extrinsic matrix i.e. the spatial extrinsic parameter of the laser radar and the right camera; The step S4 is implemented by, In the transformation process of the laser radar coordinate system to the left and right camera coordinate systems, three coordinate systems are involved, which are the left camera coordinate system, the right camera coordinate system and the laser radar coordinate system; the point cloud collected by the laser radar is a point in the laser radar coordinate system, and needs to be converted into a point in the camera coordinate system through rotation and translation transformation; The rotation and translation of the LiDAR coordinate system with respect to the left camera coordinate system can be represented by the rotation matrix and the translation vector respectively; is a 3 x 3 vector, t xl , t yl , t zl are independent of each other; is a 3x 3 matrix, but the 9 elements are not independent of each other, and the 9 elements are decoupled by using Euler angle expression; the method of expressing the rotation matrix as 3 independent Euler angles reduces the number of variables and simplifies the structure of the BP neural network; the rotation matrix is converted into the following form by Euler angle: ​ by φ l , θ l , ψ l , t xl , t yl , t zl six parameters can represent an extrinsic matrix extrinsic matrix is used to convert a three-dimensional point between the laser radar coordinate system and the left camera coordinate system, that is, the spatial extrinsic parameter of the laser radar and the left camera; The rotation and translation of the LiDAR coordinate system with respect to the right camera coordinate system can be represented by the rotation matrix and the translation vector respectively; is a 3x3 vector, t xr , t yr , t zr are independent of each other. is a 3x 3 matrix, but the 9 elements are not independent of each other, and the 9 elements are decoupled by using the Euler angle expression; by expressing the rotation matrix as 3 independent Euler angles, the variable is reduced, and the BP neural network structure is simplified; the method of expressing the rotation matrix as 3 independent Euler angles decouples the 9 variables, reduces the variable, and further simplifies the BP neural network structure; by Euler angle, the rotation matrix is converted into the following form: by φ r , θ r , ψ r , t xr , t yr , t zr six parameters can represent an extrinsic matrix extrinsic matrix is used to convert a three-dimensional point between the laser radar coordinate system and the left camera coordinate system, that is, the spatial extrinsic parameter of the laser radar and the right camera; S5. Introducing baseline length error re-projection error and spatial projection error of the center point of the calibration board under the camera coordinate system and constructing a BP neural network model and a loss function: the BP neural network model includes a 3-layer structure: an input layer, an intermediate hidden layer, and an output layer; the input layer contains 3 neurons, the hidden layer contains 6 neurons and 6 biases, and the output layer contains 5 neurons; the input layer is the coordinate value of the center point of the calibration board under the laser radar coordinate system, the hidden layer obtains the coordinate value (x ClP ,y ClP ,z ClP ) of the center point of the calibration board calculated by the model under the left camera coordinate system and the coordinate value (x CrP ,y CrP ,z CrP ) of the center point of the calibration board calculated by the model under the right camera coordinate system, and the output layer is the coordinate (u lP ,v lP ) of the center point of the calibration board calculated by the model under the left pixel coordinate system, the coordinate (u rP ,v rP ) of the center point of the calibration board calculated by the model under the right pixel coordinate system, and the baseline length Baseline P calculated by the model; introducing baseline length error re-projection error and spatial projection error of the center point of the calibration board under the camera coordinate system Baseline T is the actual length of the baseline of the binocular camera, (x ClP ,y ClP ,z ClP ) is the coordinate value of the center point of the calibration board under the left camera coordinate system, (x CrP ,y CrP ,z CrP ) is the coordinate value of the center point of the calibration board under the right camera coordinate system, (u r ,v r ) is the coordinate of the center point of the calibration board under the left pixel coordinate system, and (u r ,v r ) is the coordinate of the center point of the calibration board under the right pixel coordinate system; a loss function containing the baseline length error, the re-projection error, and the spatial projection error of the center point of the calibration board under the camera coordinate system is constructed through three proportional coefficients a, b, and c In the BP neural network, the loss function is introduced into the synchronous back propagation from the hidden layer to the input layer, the baseline length error is reduced through the synchronous back propagation of the baseline length error in the loss function, and the re-projection error can be reduced; the re-projection error is further eliminated through the synchronous back propagation of the re-projection error in the loss function; that is, the precision of the spatial extrinsic parameter calibration of the binocular camera and the laser radar is improved by introducing the loss function into the synchronous back propagation from the hidden layer to the input layer; the training efficiency and the spatial extrinsic parameter calibration precision are improved by synchronously training the binocular camera data and the laser radar data through the fusion of the mapping relationship of the left and right cameras constructed by the loss function in the hidden layer. S6. Configure the parameters required for network training and train; set the learning rate, batch size, weight initialization method, weight decay coefficient, maximum iteration number and loss function target size; The coordinates (x L ,y L ,z L ) of the center points P of N sets of calibration boards in the laser radar coordinate system, the coordinates in the left and right camera coordinate systems, and the coordinates in the left and right pixel coordinate systems are input into the improved BP neural network for training; The weight is updated during training, and the following is obtained when the loss function meets the condition: an extrinsic matrix of a laser radar coordinate system to a left camera coordinate system and an extrinsic matrix of a laser radar light radar coordinate system to a right camera coordinate system That is, the spatial extrinsic parameter calibration of the binocular camera and the laser radar is realized based on the BP neural network.

2. The method of claim 1, wherein the method is based on a BP neural network. Also includes S7, the extrinsic matrix of the laser radar and the left and right cameras obtained according to S6 and Project the laser point cloud into the left and right camera coordinate systems respectively and calculate the re-projection error Through the extrinsic matrix and Calculate the baseline length and compare it with the baseline length of the camera itself to obtain the baseline length error The results of the binocular camera and the laser radar spatial extrinsic parameter calibration are comprehensively evaluated through two indicators. The baseline length error and the re-projection error are introduced as evaluation indicators.

3. The method of claim 1 or 2, wherein: The step S2 is implemented by, S21. Obtain the coordinates of the center point of each group of data calibration board and each corner point in the pixel coordinate system, define the coordinates of the center point of the calibration board in the world coordinate system and calculate the coordinates of each corner point in the world coordinate system; For each set of data, the detection function is used to extract the corner points of the checkerboard calibration board from the left image, resulting in C for each corner point in each set of data. nx The center point P of each data calibration board is obtained by using the coordinates in the left pixel coordinate system and the positional relationship between the corner points. n Coordinates in the left pixel coordinate system (u) nl ,v nl Finally, the coordinates (u) of the center point P of the N sets of chessboard calibration boards in the left pixel coordinate system are obtained. l ,v l For each set of data, the detection function is used to extract the corner points C in the chessboard calibration board from the right image, thus obtaining the corner point C for each set of data. nx The center point P of each data calibration board is obtained by using the coordinates in the right pixel coordinate system and the positional relationship between the corner points. n Coordinates in the right pixel coordinate system (u) nr ,v nr Finally, the coordinates (u) of the center point P of the N sets of chessboard calibration boards in the left pixel coordinate system are obtained. r ,v r Define the center point P of the calibration plate. n The coordinates in the world coordinate system are (0,0,0). The X-axis of the world coordinate system is parallel to the short side of the calibration plate, the Y-axis is parallel to the long side of the calibration plate, and the Z-axis points from the non-chessboard surface to the chessboard surface of the calibration plate. The corner point C of each data set is calculated based on the side lengths of the chessboard grid on the calibration plate. nx Coordinates in the world coordinate system; S22. For each set of data, using the PnP method, the rotation matrix of the left camera coordinate system (O nx X cl Y cl Z cl ) relative to the world coordinate system (O cl X w Y w Z w ) is calculated from the corresponding coordinates of each corner point C w translation vector For each set of data, using the PnP method, the rotation matrix of the left camera coordinate system (O nx X cr Y cr Z cr ) relative to the world coordinate (O cr X w Y w Z w ) system is calculated from the corresponding coordinates of each corner point C w translation vector ​​ S23. Calculate the coordinates (x, y) of the center point P of the N sets of chessboard calibration boards in the left camera coordinate system through coordinate system transformation. Cl ,y Cl ,z Cl ) and the coordinates (x) in the right camera coordinate system Cr ,y Cr ,z Cr ); For each group of data, the center point P of the group of data is calibrated n The coordinate (0, 0, 0) in the world coordinate system and the extrinsic matrix of the left camera coordinate system of the group of data relative to the world coordinate system are multiplied to obtain the center point P of the group of data n The coordinate (x nCl ,y nCl ,z nCl ) in the left camera coordinate system; finally, the coordinates (x Cl ,y Cl ,z Cl ) of the N groups of center points P of the checkerboard calibration board in the left camera coordinate system are obtained; and the extrinsic matrix is obtained from the rotation matrix and the translation vector in S22. For each group of data, the center point P of the group of data is calibrated on the calibration board n The coordinate (0, 0, 0) in the world coordinate system is multiplied by the extrinsic matrix of the right camera coordinate system relative to the world coordinate system to obtain the center point P of the group of data on the calibration board n The coordinate (x nCl ,y nCl ,z nCl ) in the right camera coordinate system is obtained, and then the coordinates (x Cr ,y Cr ,z Cr ) of the N groups of center points P of the checkerboard calibration board in the right camera coordinate system are obtained; the extrinsic matrix is obtained from the rotation matrix and the translation vector in S22.

4. The method of claim 3, wherein: In step S5, Baseline P As shown in equation (15); introducing baseline length error re-projection error and the spatial projection error of the calibration board center point in the camera coordinate system Baseline T is the actual length of the baseline of the binocular camera, (x ClP , y ClP , z ClP ) is the coordinate value of the center point of the calibration board in the left camera coordinate system, (x CrP , y CrP , z CrP ) is the coordinate value of the center point of the calibration board in the right camera coordinate system, (u l , v l ) is the coordinate of the center point of the calibration board in the left pixel coordinate system, (u r , v r ) is the coordinate of the center point of the calibration board in the right pixel coordinate system; The loss function containing the baseline length error, the re-projection error and the spatial projection error of the center point of the calibration board in the camera coordinate system is constructed by three proportionality coefficients α, β, χ The loss function As shown in formula (19); 5. The method of claim 4, wherein: N≥6, X=35.

6. The method of claim 5, wherein: The corner points in the checkerboard calibration board are extracted using the cv2.findChessboardCorners detection function for the left and right images.

7. The method of claim 6, wherein: The Euler angle uses the Euler angle in the Z-Y-X sequence form. The Euler angle uses the Euler angle in the Z-Y-X sequence form.

Images