Unmanned vehicle environment perception multi-sensor space-time registration method

Abstract

The invention provides an unmanned vehicle environment perception multi-sensor space-time registration method, which comprises the following steps of: obtaining internal and external parameters in a binocular camera mathematical model by utilizing the binocular camera mathematical model and a projection transformation principle, and then, setting a known calibration object and taking a binocular camera detection image with a relatively high data rate as a benchmark to obtain a multi-sensor space-time registration result. And carrying out coordinate transformation on the laser radar point cloud data and the millimeter wave radar plot data, and finally unifying the data to image data to complete spatial registration of the three sensors. The problems that a traditional time registration method is poor in real-time performance and low in space registration precision due to the fact that the traditional time registration method excessively depends on a high-resolution laser radar are effectively solved.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of unmanned vehicles, and particularly to a sensor registration method. BACKGROUND

[0002] An unmanned vehicle is a kind of ground vehicle. It has a very broad application prospect in the military and civilian fields due to its ability to automatically drive without human control and perform corresponding tasks, and there are successful application cases. Real-time environment perception and understanding are the prerequisite for ground unmanned vehicle systems to perform autonomous tasks. In particular, in the military field, due to the complexity of the ground environment, especially the complex conditions of the field of battle and autonomous navigation, ground unmanned systems are still in the research stage for autonomous environment perception under complex field conditions. In particular, obstacle detection in the surrounding environment has always been a very challenging research topic and a difficult problem worldwide. Multi-sensor fusion unmanned vehicle obstacle detection is the main development direction of current unmanned vehicle environment perception technology. However, due to the differences in the working platforms of multiple sensors or the sampling intervals of the observation information provided, the existence of sensor self-bias and observation error causes the need for data registration before multi-sensor fusion, i.e., the time and space registration process required for multi-sensor data "error-free" conversion, mainly including time registration and space registration. At the same time, high-precision time and space registration is the prerequisite and foundation for subsequent multi-sensor information fusion.

[0003] The traditional method of multi-sensor fusion detection and perception time and space registration mainly uses the GPS / Beidou navigation positioning method to realize time synchronization and space alignment. The basic idea is that all sensors in the system add time stamps to the real-time detection and perception of target or environmental information based on GPS / Beidou timing, realizing time synchronization and registration during information fusion. Space registration is also based on the latitude and longitude information provided by the navigation positioning device on each sensor, which is converted to the same platform coordinate system or geodetic coordinate system through coordinate conversion, realizing space registration.

[0004] The defects of the traditional multi-sensor fusion detection time synchronization and registration method are that the sampling frequencies of different sensors are not the same, the information collected by the sensors is not necessarily the information at the same time, the time stamp of the heterogeneous sensor measurement information is in reverse order, and when the high data rate sensor loses the target and the low data rate sensor keeps tracking the target, the fused composite tracking track appears batch interruption phenomenon. The traditional spatial registration mainly depends on the high resolution laser radar to obtain 3D point coordinates as the reference, and then the measurement information of other sensors is matched with the 3D point coordinates of the laser radar, that is, based on the high resolution of the laser radar, the corresponding camera 2D point and laser radar 3D point are used to realize that the camera 2D point coordinates are obtained by fitting the vertex 3D coordinates of the calibration board (corner key point extraction), and the laser radar 3D point is obtained by fitting the boundary straight line of the calibration board. The corresponding coordinate transformation matrix is obtained by solving the PNP (positioning three-dimensional space objects from an image) problem as follows:

[0005]

[0006] The problem is converted into an optimization problem of radar coordinate system to camera coordinate system re-projection to solve the final R, T as follows.

[0007]

[0008] However, such a processing method depends on the high resolution laser radar, otherwise the 3D point coordinates obtained have large errors, so that the final data fusion result is not very accurate. SUMMARY

[0009] In order to overcome the defects of the prior art, the present application provides a multi-sensor space-time registration method for environment perception of unmanned vehicles.

[0010] The application discloses a multi-sensor space-time registration method for environment perception of unmanned vehicles, which is based on laser radar, binocular camera and millimeter wave radar three kinds of sensors, and realizes time registration of information of the three kinds of sensors through software. The internal and external parameters in the mathematical model of the binocular camera are obtained by using the mathematical model of the binocular camera and the principle of projection transformation, then, the known calibration object is set, the laser radar point cloud data and the millimeter wave radar track data are coordinate-converted by taking the image detected by the binocular camera with higher data rate as the reference, and finally the space registration of the three kinds of sensors is completed by unifying to the image data. The real-time performance of the traditional time registration method is effectively improved, and the problem of low spatial registration accuracy caused by excessive dependence on the high resolution laser radar is solved.

[0011] The technical scheme adopted by the application to solve the technical problems comprises the following steps:

[0012] Step 1: Binocular camera calibration

[0013] Step 1.1: Perform binocular camera three-dimensional image to two-dimensional image projection;

[0014] Step 1.2: Calculate the internal parameters K of the binocular camera;

[0015] Step 1.3: Calculate the external parameters R and t;

[0016] Step 2: Laser radar and binocular camera joint calibration;

[0017] Step 2.1: Set up the calibration board;

[0018] As shown in Figure 2 , two rectangular paper boards with the same height as the unmanned vehicle binocular camera are placed 15 m in front of the unmanned vehicle as calibration boards, one of the edges of the calibration board is placed at a 45° angle with the ground, and the distance between the two paper boards is within the detection range of the laser radar and binocular vision, about 4-5 m apart;

[0019] Step 2.2: Obtain the 3D coordinates of the calibration board vertices under the binocular camera;

[0020] Two-dimensional code (ArUco) markers are placed on the two rectangular boards, and since the size of the paper board and the position of the ArUco marker are known, the positions of the calibration board vertices under the ArUco marker coordinate system are calculated; the ArUco marker provides the rotation and translation matrix between the camera coordinate system and the marker coordinate system, thereby obtaining the 3D coordinates P of the eight vertices of the two calibration boards under the binocular camera i = [u i , v i , w i , 1] T , i = 1, 2, …, 8;

[0021] Step 2.3: Obtain the 3D vertex coordinates of the calibration board vertices under the laser radar;

[0022] Step 2.4: Laser radar and binocular camera joint calibration;

[0023] After obtaining the 3D coordinates of the eight vertices of the two calibration boards under the binocular camera detection and the 3D coordinates of the eight vertices of the two calibration boards under the laser radar detection, eight groups of point cloud pairs (P, Q) of the binocular camera 3D points to the laser radar 3D points are obtained, and then the objective function shown in the following formula is minimized:

[0024]

[0025] The rotation matrix R and translation matrix t between the binocular camera and the lidar are obtained using the Iterative Nearest Neighbor (ICP) method. Finally, the lidar 3D points are projected onto the image plane based on the obtained rotation matrix R and translation matrix t to complete the joint calibration of lidar and binocular camera.

[0026] Step 3: Joint calibration of millimeter-wave radar and binocular camera;

[0027] Step 3.1: Set the calibration target;

[0028] like Figure 4 As shown, a metal plate target point is placed 15m in front of the unmanned vehicle. The size of the metal plate meets the requirements of millimeter-wave radar detection to facilitate millimeter-wave radar detection.

[0029] The target point on the metal plate is set to 1m×1m. The size of the target point is set so that the information obtained by the millimeter-wave radar and the binocular camera is clearer, which is beneficial to improving the calibration effect and accuracy.

[0030] Step 3.2: Perform coordinate transformation between millimeter-wave radar and binocular camera;

[0031] Step 3.3: Calculation of the calibration matrix;

[0032] The six parameters of the calibration matrix are solved using the following calculations;

[0033] Let T i =[t i1 t i2 t i3 ]',U=[u1 u2 ... u n ]', V=[v1 v2 ... v n ]', I n×1 =[1 1 ...1]', Where n is the number of alignment points. This refers to the position of the alignment point in the radar coordinate system, j = 1, 2, ..., n, n ≥ 4, and the calibration matrix. Calculated using the linear least squares (LS) method:

[0034] T1 = (PP) T ) -1 P T U

[0035] T2=(PP T ) -1 P T V

[0036] T3=(PP T ) -1 P T In×1

[0037] It can complete the spatial registration of three types of sensors: lidar, binocular camera, and millimeter-wave radar.

[0038] The specific steps for projecting the binocular camera's 3D image into a 2D image in step 1.1 are as follows:

[0039] like Figure 1 As shown, in the mathematical model of a binocular camera, the projection center is O, the image plane is W, and any point p in the world coordinate system has a corresponding projection point n on the image plane W. n is the intersection of the extension lines of p and O with the image plane W.

[0040] The mathematical expression for projecting the 3D image acquired by the binocular camera onto a 2D plane is as follows:

[0041] in, These are homogeneous coordinates in the world coordinate system, where X, Y, and Z are three-dimensional coordinates in the world coordinate system, and n = [u, v, 1]. T λ represents the homogeneous coordinates of the image plane, u and v are the two-dimensional position coordinates of each pixel in an image, and λ is the scaling factor. The perspective projection matrix is ​​a 3×4 matrix, determined by the intrinsic and extrinsic parameters of the binocular camera. The perspective projection matrix can be decomposed into the following equation:

[0042]

[0043] K is the intrinsic parameter of the stereo camera, R is the rotation matrix of the stereo camera's extrinsic parameters, and t is the translation vector of the stereo camera's extrinsic parameters. The two extrinsic parameters R and t represent the geometric transformation relationship between the camera coordinate system and the world coordinate system, respectively. R and t also determine the orientation and position of the camera installation.

[0044] The steps for calculating the intrinsic parameters K of the stereo camera in step 1.2 are as follows:

[0045] The intrinsic parameter K of the stereo camera represents the projection relationship between the object point and the image point in the stereo camera coordinate system. It is obtained through Zhang's calibration method, and the expression is as follows:

[0046]

[0047] Among them, f x with f y , respectively, are the equivalent focal lengths of the binocular camera in the x and y directions, and u0 and v0 are the coordinates of the center of the image pixel, i.e., the intersection of the optical axis and the image plane.

[0048] The specific steps for obtaining the 3D vertex coordinates of the calibration board vertex under the LiDAR in step 2.3 are as follows:

[0049] The 3D coordinates of the calibration plate vertex under the lidar are obtained by straight line fitting. Due to the horizontal characteristics of the lidar scanning line, if one side of the marker is kept parallel to the ground, a vertical edge can be obtained, but a horizontal edge may not be obtained. To overcome this problem, the calibration plate is tilted so that one side of the calibration plate forms a 45° angle with the ground plane.

[0050] Two calibration boards were probed using a lidar system, resulting in two sets of point cloud pairs representing the eight boundaries of the two calibration boards. Figure 3 The circle representing the boundary of the calibration board is shown. Then, using the point clouds of the four boundaries on each calibration board, the edge points are fitted to the four boundary lines of the calibration board using the Random Sample Consensus (RanSaC) method. The intersection of every two boundary lines is then obtained, which is the vertex of the calibration board. Two sets of point cloud pairs can be fitted to obtain the eight boundary lines of the two calibration boards. The coordinates Q of the eight vertices of the two calibration boards can be obtained through the intersection of every two boundary lines. i =[X i ,Y i Z i ,1] T ,i=1,2,…,8.

[0051] In step 3.1, the clustering method used by the millimeter-wave radar is nearest neighbor clustering. Assuming the target point detected by the radar has centroid coordinates of the panel in the image:

[0052]

[0053] Where w and h are the width and height of the panel in the image, respectively, and I w,h These are pixel values; the total width and height of the panel in the image are represented by W×H.

[0054] The steps in step 3.2 for coordinate transformation between the millimeter-wave radar and the binocular camera are as follows:

[0055] like Figure 5 As shown, the coordinates of the same target point in the millimeter-wave radar coordinate system are represented as (X... r ,Y r Z r The coordinates of the point in the stereo camera coordinate system are represented as (X...). c ,Y c Z c (u,v) represents pixel coordinates, the distance of the target point in the radar coordinate system is represented by r, and the azimuth angle is represented by α;

[0056] Assume the position of the target point in the two-dimensional coordinate system of the stereo camera image is (x r ,y r), millimeter wave radar only gives the target point two-dimensional azimuth information (X r ,Y r ), the calibration matrix is obtained by the following transformation relationship:

[0057]

[0058] Wherein, is a 3x3 dimensional calibration matrix, x r = rsin alpha, y r = rcos alpha, directly convert the radar coordinates into pixel coordinates.

[0059] An electronic device comprising: one or more processors; memory; one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, the one or more programs are configured to perform the method as described above.

[0060] A computer readable storage medium, the computer readable storage medium stores program code, the program code can be called by a processor to execute the method as described above.

[0061] The beneficial effects of the present application are based on three types of sensors such as laser radar, binocular camera, millimeter wave radar, in the time registration method, from the real-time point of view, without using GPS timing method, instead of using thread synchronization method, the real-time of time registration is higher. In the method of space registration, the present application proposes a 3D point corresponding to 3D point calibration method, which does not use the traditional binocular camera 2D point and laser radar 3D point corresponding to realize, overcome the defect that the final data fusion result is not so accurate caused by excessive dependence on high resolution laser radar, the 3D point coordinates obtained have great error. BRIEF DESCRIPTION OF DRAWINGS

[0062] Figure 1 is a schematic diagram of three-dimensional image projection to two-dimensional plane in binocular camera calibration of the present application.

[0063] Figure 2 The schematic diagram of the calibration board setting when laser radar and binocular camera are jointly calibrated.

[0064] Figure 3 is a schematic diagram of laser radar and binocular camera joint calibration.

[0065] Figure 4 is a target point setting schematic diagram of millimeter wave radar and binocular camera joint calibration.

[0066] Figure 5 is a principle schematic diagram of millimeter wave radar and binocular camera joint calibration. DETAILED DESCRIPTION

[0067] The application will be further described in connection with the accompanying drawings and examples.

[0068] The application provides a multi-sensor space-time registration method for environment perception of an unmanned vehicle based on a laser radar, binocular camera and millimeter wave radar, including time registration and space registration. In the time registration method, a thread synchronization method is used instead of a GPS timing method, and the time registration has higher real-time performance. In the space registration method, a 3D point-to-3D point calibration method is proposed, which does not use the traditional binocular camera 2D point-to-laser radar 3D point correspondence to achieve the space registration, and overcomes the defects caused by excessive dependence on high-resolution laser radars.

[0069] The data fusion of sensors in time is the synchronization of sensor data in time. Since the sampling frequencies of different sensors are not the same, the information collected by the sensors is not necessarily the information at the same time. In this project, a thread synchronization method is used instead of a GPS timing method from the perspective of real-time performance. In the program, a radar data receiving thread and a camera data receiving thread are created, and the current radar data is obtained when the current frame image is collected each time. In this way, the radar data and the camera data are synchronized in time.

[0070] The principle of space registration is to unify the radar coordinate system, the camera coordinate system, the image pixel coordinate system and the vehicle body coordinate system. The unification and establishment of the coordinate system are beneficial to the measurement of the specific distance and direction of the environment and obstacles by the environment perception sensor, and the calibration of the camera external parameters can realize the inverse perspective projection transformation and provide important parameters for vehicle visual navigation. At the same time, in order to project the radar scanning points onto the image to form a dynamic region of interest (ROI) in the image coordinate system, this is beneficial to reducing the search area for the identification and tracking of the front obstacles, thereby reducing the system calculation time and improving the system real-time performance. At the same time, combined with the respective perception characteristics of the camera and the radar, the robustness of the system can be enhanced, the detection accuracy can be effectively improved, and the false alarm rate of identification can be reduced.

[0071] LiDAR and cameras are the two most important sensors for autonomous perception in the field. Their respective characteristics make sensor fusion a superior detection solution. LiDAR offers high resolution, high ranging accuracy, and good real-time performance, but it is significantly affected by obstacle material properties and weather factors (such as rain and fog). Cameras provide rich color and feature information and can detect objects of interest using state-of-the-art algorithms, but the two-dimensional images acquired by cameras lack depth information and cannot obtain three-dimensional obstacle information from the images. Furthermore, camera detection accuracy is low and affected by weather, lighting, texture, and shadows, making it difficult to obtain accurate obstacle information. Therefore, this invention employs LiDAR and camera data fusion to map the three-dimensional coordinates of point clouds in space to pixels in the camera, obtaining a transformation matrix between the radar coordinate system and the camera coordinate system.

[0072] The embodiments of the present invention are as follows:

[0073] Step 1: Binocular camera calibration

[0074] Step 1.1: Project the 3D image from the binocular camera into a 2D image;

[0075] like Figure 1 As shown, in the mathematical model of a binocular camera, the projection center is O, the image plane is W, and any point p in the world coordinate system has a corresponding projection point n on the image plane W. n is the intersection of the extension lines of p and O with the image plane W.

[0076] The mathematical expression for projecting the 3D image acquired by the binocular camera onto a 2D plane is as follows:

[0077] in, These are homogeneous coordinates in the world coordinate system, where X, Y, and Z are three-dimensional coordinates in the world coordinate system, and n = [u, v, 1]. T λ represents the homogeneous coordinates of the image plane, u and v are the two-dimensional position coordinates of each pixel in an image, and λ is the scaling factor. The perspective projection matrix is ​​a 3×4 matrix, determined by the intrinsic and extrinsic parameters of the binocular camera. The perspective projection matrix can be decomposed into the following equation:

[0078]

[0079] K is the intrinsic parameter of the stereo camera, R is the rotation matrix of the extrinsic parameter of the stereo camera, and t is the translation vector of the extrinsic parameter of the stereo camera. The two extrinsic parameters R and t represent the geometric transformation relationship between the camera coordinate system and the world coordinate system, respectively. R and t also determine the orientation and position of the camera installation.

[0080] Step 1.2: Calculate the intrinsic parameters K of the stereo camera;

[0081] The intrinsic parameter K of the stereo camera represents the projection relationship between the object point and the image point in the stereo camera coordinate system. It is obtained through Zhang's calibration method, and the expression is as follows:

[0082]

[0083] Among them, f x with f y , respectively, are the equivalent focal lengths of the binocular camera in the x and y directions, and u0 and v0 are the coordinates of the center of the image pixel, i.e., the intersection of the optical axis and the image plane;

[0084] Step 1.3: Calculate the extrinsic parameters R and t;

[0085] Step 2: Joint calibration of LiDAR and binocular camera;

[0086] Step 2.1: Set up the calibration board

[0087] like Figure 2 As shown, two rectangular cardboard pieces with the same height as the binocular camera on the autonomous vehicle are placed 15m in front of the vehicle as calibration boards. One side of the calibration board is placed at a 45° angle to the ground. The two cardboard pieces are 4-5m apart, within the detection range of the lidar and binocular vision.

[0088] Step 2.2: Obtain the 3D coordinates of the calibration board vertices under the stereo camera;

[0089] Two rectangular plates are marked with ArUco QR codes. Since the dimensions of the cardboard and the positions of the ArUco marks are known, the positions of the vertices of the calibration plates in the ArUco mark coordinate system are calculated. The ArUco marks provide the rotation and translation matrices between the camera coordinate system and the mark coordinate system, thus obtaining the 3D coordinates P of the eight vertices of the two calibration plates under the stereo camera. i =[u i ,v i ,w i ,1] T i = 1, 2, ..., 8;

[0090] Step 2.3: Obtain the 3D vertex coordinates of the calibration board vertices under the LiDAR;

[0091] The 3D coordinates of the calibration plate vertex under the lidar are obtained by straight line fitting. Due to the horizontal characteristics of the lidar scanning line, if one side of the marker is kept parallel to the ground, a vertical edge can be obtained, but a horizontal edge may not be obtained. To overcome this problem, the calibration plate is tilted so that one side of the calibration plate forms a 45° angle with the ground plane.

[0092] Two calibration boards were probed using a lidar system, resulting in two sets of point cloud pairs representing the eight boundaries of the two calibration boards. Figure 3 The diagram shows the circles representing the boundaries of the calibration plates. Then, using the point clouds of the four boundaries on each calibration plate, the Random Sample Consensus (RanSaC) method is used to fit the edge points to the four boundary lines of the calibration plate. The intersection of every two boundary lines is then determined, which represents the vertex of the calibration plate. Two sets of point clouds can be used to fit the eight boundary lines of the two calibration plates. The coordinates Q of the eight vertices of the two calibration plates can be obtained through the intersection of every two boundary lines. i =[X i ,Y i Z i ,1] T i = 1, 2, ..., 8;

[0093] Step 2.4: Joint calibration of LiDAR and binocular camera;

[0094] After obtaining the 3D coordinates of the eight vertices of the two calibration boards under the binocular camera detection and the 3D coordinates of the eight vertices of the two calibration boards under the lidar detection, eight pairs of point clouds (P, Q) from the binocular camera 3D points to the lidar 3D points are obtained. Then, the objective function shown in the following equation is minimized:

[0095]

[0096] The rotation matrix R and translation matrix t between the binocular camera and the lidar are obtained using the Iterative Nearest Neighbor (ICP) method. Finally, the lidar 3D points are projected onto the image plane based on the obtained rotation matrix R and translation matrix t to complete the joint calibration of lidar and binocular camera.

[0097] Step 3: Joint calibration of millimeter-wave radar and binocular camera;

[0098] Step 3.1: Set the calibration target;

[0099] like Figure 4 As shown, a metal plate is placed 15m in front of the unmanned vehicle as a target point. The size of the metal plate meets the requirements for millimeter-wave radar detection. The millimeter-wave radar uses nearest neighbor clustering. Assuming the target point detected by the radar is the centroid coordinate of the panel in the image, the coordinates are:

[0100]

[0101] Where w and h are the width and height of the panel in the image, respectively, and I w,h These are pixel values; the total width and height of the panel in the image are represented by W×H.

[0102] The target point size of the metal plate is set to 1m*1m, and the target point size is set to make the information obtained by the millimeter wave radar and the binocular camera clearer, so as to improve the calibration effect and precision;

[0103] Step 3.2: Coordinate conversion of the millimeter wave radar and the binocular camera is performed;

[0104] As shown in Figure 5 , the coordinate point of the same target point in the millimeter wave radar coordinate system is represented as (X r ,Y r ,Z r ), and the coordinate point in the binocular camera coordinate system is represented as (X c ,Y c ,Z c ), (u, v) represents the pixel coordinate, the distance of the target point in the radar coordinate system is represented by r, and the azimuth angle is represented by a;

[0105] Suppose the position of the target point in the two-dimensional coordinate system of the binocular camera image is (x r ,y r ), the millimeter wave radar only gives the two-dimensional azimuth information (X r ,Y r ) of the target point, and the calibration matrix is obtained through the following transformation relationship:

[0106]

[0107] wherein, is a 3*3 dimensional calibration matrix, x r =rsin a, y r =rcos a, and the radar coordinate is directly converted into the pixel coordinate;

[0108] Step 3.3: Calculation of the calibration matrix;

[0109] Through the following calculation, the six parameters of the calibration matrix are solved;

[0110] Let T i =[t i1 t i2 t i3 ]', U=[u1 u2... u n ]', V=[v1 v2... v n ]', I n×1 =[1 1...1]', wherein n is the number of alignment points, is the position of the alignment point in the radar coordinate system, j=1, 2,..., n, n>=4, and the calibration matrix is calculated by the linear least square (LS) method:

[0111] T1 = (PP T ) -1 P T U

[0112] T2 = (PP T ) -1 P T V

[0113] T3 = (PP T ) -1 P T I n×1

[0114] That is, the spatial registration of the three sensors of the laser radar, binocular camera and millimeter wave radar can be completed.

Claims

1. An unmanned vehicle environment perception multi-sensor spatio-temporal registration method, characterized in that Comprising the following steps: Step 1: binocular camera calibration; Step 1.1: Perform binocular camera three-dimensional image to two-dimensional image projection; Step 1.2: Calculate the internal parameters K of the binocular camera; Step 1.3: Calculate the external parameters R and t; Step 2: Laser radar and binocular camera joint calibration; Step 2.1: Set up the calibration board; Place two rectangular paper boards with the same height as the unmanned vehicle binocular camera mounting height 15m in front of the unmanned vehicle as the calibration board, one of the edges of the calibration board is placed at a 45° angle with the ground, and the distance between the two paper boards is within the detection range of the laser radar and binocular vision, about 4-5m apart; Step 2.2: Obtain the 3D coordinates of the calibration board vertices under the binocular camera; Two-dimensional code ArUco markers are performed on two rectangular plates. Since the size of the paper plate and the position of the ArUco marker are known, the positions of the vertices of the calibration plate in the coordinate system of the ArUco marker are calculated. The ArUco marker provides a rotation and translation matrix between the camera coordinate system and the marker coordinate system, so that the 3D coordinates P of the 8 vertices of the two calibration plates under the binocular camera are obtained i = [u i ,v i ,w i ,1] T ,i = 1, 2, …, 8; Step 2.3: Obtain the 3D vertex coordinates of the calibration board under the laser radar; Step 2.4: Laser radar and binocular camera joint calibration; After obtaining the 3D coordinates of the 8 vertices of the two calibration boards under the binocular camera detection and the 3D coordinates of the 8 vertices of the two calibration boards under the laser radar detection, 8 groups of point cloud pairs (P, Q) of binocular camera 3D points to laser radar 3D points are obtained, and then the target function shown in the following formula is minimized: The rotation matrix R and the translation matrix t between the binocular camera and the laser radar are obtained by using the iterative nearest neighbor method, and finally the laser radar 3D points are projected onto the image plane according to the obtained rotation matrix R and translation matrix t, completing the joint calibration of the laser radar and the binocular camera; Step 3: Joint calibration of millimeter wave radar and binocular camera; Step 3.1: Set the calibration target; As shown in Figure 4, place a metal plate target point 15m in front of the unmanned vehicle, and the size of the metal plate meets the detection requirements of the millimeter wave radar, so as to facilitate the detection of the millimeter wave radar; The size of the metal plate target point is set to 1m x 1m, and the size of the target point is set to make the information obtained by the millimeter wave radar and the binocular camera clearer, which is conducive to improving the calibration effect and accuracy; Step 3.2: Coordinate conversion between millimeter wave radar and binocular camera; Step 3.3: Calculation of calibration matrix; Through the following calculation, the six parameters of the calibration matrix are solved; Let T i =[t i1 t i2 t i3 ]',U=[u1 u2 ... u n ]', V=[v1 v2 ... v n ]', I n×1 =[1 1 ... 1]', Where n is the number of alignment points. This refers to the position of the alignment point in the radar coordinate system, j = 1, 2, ..., n, n ≥ 4, and the calibration matrix. Calculated using the linear least squares (LS) method: T1 = (PP T ) -1 P T U T2 = (PP T ) -1 P T V T3 = (PP T ) -1 P T I n×1 The spatial registration of the laser radar, binocular camera and millimeter wave radar three sensors is completed.

2. The unmanned vehicle environment perception multi-sensor space-time registration method according to claim 1, characterized in that: The specific steps of step 1.1 for performing binocular camera three-dimensional image to two-dimensional image projection are: In the mathematical model of the binocular camera, the projection center is O, the image plane is W, and any point p in the world coordinate system has a corresponding projection point n on the image plane W, which is the intersection point of the extension line of p and O with the image plane W; The three-dimensional image acquired by the binocular camera is projected onto a two-dimensional plane, and the mathematical expression of the projection process is as follows: wherein, is the homogeneous coordinate of the world coordinate system, X, Y and Z are three-dimensional coordinates of the world coordinate system, n = [u, v, 1] T is the homogeneous coordinate of the image plane, u, v are two-dimensional position coordinates of each pixel point in an image, λ is a scaling factor, is a 3x4 perspective projection matrix determined by the internal and external parameters of the binocular camera, the perspective projection matrix is decomposed as follows: K is the internal parameter of the binocular camera, R is the rotation matrix of the external parameter of the binocular camera, t is the translation vector of the external parameter of the binocular camera, and R and t are two external parameters representing the geometric transformation relationship between the camera coordinate system and the world coordinate system, and R and t determine the orientation and position of the camera installation at the same time.

3. The unmanned vehicle environment perception multi-sensor space-time registration method according to claim 2, characterized in that: The step of calculating the internal parameters K of the binocular camera in step 1.2 is: The intrinsic parameter K of the binocular camera represents the projection relationship between the object point and the image point in the binocular camera coordinate system, and is obtained by Zhang's calibration method, and the expression is as follows: where f x and f y are the effective focal lengths of the binocular camera in the x and y directions, respectively, and u0and v0are the coordinates of the image pixel centers, i.e., the intersection of the optical axis and the image plane.

4. The unmanned vehicle environment perception multi-sensor space-time registration method according to claim 1, characterized in that: The specific steps of step 2.3 for obtaining the 3D vertex coordinates of the calibration board vertex under the laser radar are: The 3D coordinates of the calibration board vertex under the laser radar are obtained by straight line fitting. Due to the horizontal characteristics of the laser radar scanning line, if one side of the marker remains parallel to the ground, a vertical edge can be obtained, but a horizontal edge may not be obtained; in order to overcome this problem, the calibration board is tilted to form a 45° angle between one side of the calibration board and the ground; Two sets of point clouds of eight boundaries of two calibration plates are obtained by detecting the two calibration plates using a laser radar, then the edge points are fitted into four boundary lines of the calibration plate by using the random sample consensus method with the point clouds of four boundaries on each calibration plate, and the intersection of each two boundary lines is calculated, i.e. the vertex of the calibration plate, so that eight boundary lines of two calibration plates are fitted with two sets of point clouds, and eight vertex coordinates Q of two calibration plates are obtained through the intersection of each two boundary lines i = [X i , Y i , Z i , 1] T , i = 1, 2, …, 8.

5. The unmanned vehicle environment perception multi-sensor space-time registration method according to claim 1, characterized in that: In step 3.1, the clustering method used by the millimeter wave radar is nearest neighbor clustering, assuming that the target point detected by the radar is the centroid coordinate of the panel in the image: where w, h are the width and height of the panel in the image, I w,h is the pixel value, and the total width and height of the panel in the image are denoted by W x H.

6. The unmanned vehicle environment perception multi-sensor space-time registration method according to claim 1, characterized in that: The step 3.2 is to perform the coordinate conversion between the millimeter wave radar and the binocular camera, and the steps are as follows: The coordinate point of the same target point in the millimeter wave radar coordinate system is represented as (X r ,Y r ,Z r ), the coordinate point in the binocular camera coordinate system is represented as (X c ,Y c ,Z c ), (u, v) represents a pixel coordinate, the distance of the target point in the radar coordinate system is represented by r, and the azimuth angle is represented by α; Assume the position of the target point in the two-dimensional coordinate system of the binocular camera image is (x r ,y r ), the millimeter wave radar only gives the two-dimensional azimuth information of the target point (X r ,Y r ), and the calibration matrix is obtained through the following transformation relationship: wherein, is a 3x3 dimensional calibration matrix, x r = rsin a, y r = rcos a, directly converts radar coordinates to pixel coordinates.

7. An electronic device, comprising: Comprise: One or more processors; Memory; One or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, the one or more programs are configured to perform the method of any one of claims 1-6.

8. A computer-readable storage medium, characterized in that, The computer readable storage medium stores program code, and the program code can be called and executed by the processor to perform the method of any one of claims 1-6.

Images