Perception fusion method for camera and laser radar sensor

Abstract

The invention discloses a perception fusion method for a camera and a laser radar sensor, which does not directly correspond and fuse image / point cloud features, but establishes a virtual intermediate space for feature fusion outside a bird's-eye view space and a pixel space. The calculation amount of the decoupling feature fusion process is associated with the image pixel space / point cloud aerial view space size, the calculation amount required by fusion is obviously reduced, and the calculation efficiency is improved; meanwhile, dense point clouds collected by a high-performance laser radar commonly used on the intelligent driving vehicle are better utilized; the intermediate space comprises a sparse space, a dense space and a small-size dense space, but the intermediate space and the aerial view space where the point cloud is located can be directly fused finally, a dense feature map containing sufficient point cloud information is given, the algorithm performance and the feature universality are improved, and unified feature support is provided for multiple subsequent prediction algorithms.

Description

Technical Field

[0001] The present invention belongs to the technical field of autonomous driving, and specifically relates to a perception fusion method for a camera and a lidar sensor. Background Art

[0002] For smart vehicles with autonomous driving functions, environmental understanding and semantic perception are the basis of their work, so improving their perception intelligence is the key to their autonomous driving functions. Considering that smart vehicles can be installed with a large number of sensors of various types, enhancing the fusion understanding of multi-sensor information will inevitably become an industry development trend.

[0003] However, existing multi-sensor fusion methods are often feature-intensive, and their computational complexity depends directly on the feature size of the image or radar point cloud; this places limitations on driving scenarios that require a large sensing range (hence a large radar point cloud feature size) or fine sensing results (hence a large image feature size). Most of the newly proposed sparse fusion solutions in the passenger car industry focus on a single type of low-cost sensor (mainly the fusion of multiple camera images), so they cannot fully utilize the scale invariance and good control of spatial position of the increasingly popular high-performance lidar point cloud perception, which may ultimately affect efficiency and safety, and cannot solve this problem universally. Summary of the invention

[0004] In response to the above problems, the main purpose of the present invention is to design a perception fusion method for cameras and lidar sensors, which fully takes into account the characteristics of new high-performance sensors, so as to make full use of the dense point cloud collected by high-performance lidar, improve the perception algorithm's ability to understand spatial geometry, enhance traffic operation efficiency and safety, and solve the problem of heavy computational burden of existing methods when faced with large perception range and fine perception requirements.

[0005] In order to achieve the above purpose, the present invention adopts the following technical solutions:

[0006] A perception fusion method for a camera and a lidar sensor, the fusion method comprising a bird's-eye view space for mapping point cloud features and a pixel space for mapping image features, and an intermediate space for image feature fusion is set outside the bird's-eye view space and the pixel space;

[0007] The perception fusion method comprises the following steps:

[0008] S1: Obtain the three-dimensional point cloud data collected by the laser radar and the color image data collected by the camera, perform point cloud feature extraction and image feature extraction respectively, and map the point cloud features to the bird's-eye view space to form the point cloud bird's-eye view space, and map the image features to the pixel space to form the image pixel space;

[0009] S2: Using camera extrinsic parameters, intrinsic parameters and lidar extrinsic parameters, the mapping relationship between the position of the 3D point cloud coordinate system and the position of the image pixel space is calculated;

[0010] S3: According to the three-dimensional point cloud coordinate system, an intermediate space for image feature fusion is established to obtain image features filled in the intermediate space;

[0011] S4: using the correspondence between the three-dimensional point cloud coordinate system and the point cloud bird's-eye view space, the image features corresponding to the k sparse coordinates in the intermediate space are converted into image bird's-eye view space features;

[0012] S5: Fuse the features of the point cloud bird's-eye view space in step S1 with the features of the image bird's-eye view space obtained in step S4 to obtain fused bird's-eye view space features for downstream algorithm processing.

[0013] As a further description of the present invention, in step S1, the three-dimensional point cloud data is divided into a plurality of point cloud columns according to the coordinates, and the highest point, the lowest point, the centroid point, and the number of point clouds in each point cloud column are statistically counted, and the statistics are used as the initial features of each point cloud column. By inputting the initial features of the point cloud column into the backbone network of deep learning, the point cloud features of the three-dimensional point cloud data are obtained;

[0014] For color image data, the image features of the color image data are obtained by inputting the regularized color image into the deep learning backbone network.

[0015] As a further description of the present invention, in step S2, the mapping of the position of the three-dimensional point cloud coordinate system to the position of the pixel space includes the following steps:

[0016] S21: Multiply the image external parameter matrix and the internal parameter matrix, and combine the laser radar external parameters to obtain the mapping matrix from the pixel space position to the three-dimensional point cloud coordinate system position. The expression is: P = K c [R c ∣t c ]·P l ;

[0017] The total camera matrix P c For: P c =K c [R c ∣t c ]; among them, K c is the internal parameter matrix, [R c ∣t c ] is the external parameter matrix, R c is the rotation matrix, t c is the translation vector;

[0018] The external parameter P of the laser radar l For: Pl =[R l ∣t l ]; where R l is the rotation matrix of the lidar, t l is the translation vector of the lidar;

[0019] S22: Invert the total camera matrix to obtain the mapping matrix from the 3D point cloud coordinate system to the pixel space position, which is expressed as: P -1 =(K c [R c ∣t c ]·P l ) -1 ;

[0020] S23: Use the augmented coordinates of the three-dimensional point cloud coordinate system to multiply the matrix obtained in step S22 to obtain a four-dimensional vector, where the point X in the three-dimensional point cloud coordinate system is 3d =[X,Y,Z,1] T , and P -1 Multiply to get the coordinate X in pixel space pixel =P -1 X 3d ;

[0021] S24: Determine whether the third dimension of the four-dimensional vector in step S23 can be mapped; if X pixel The third dimension z pixel <0, then this point cannot be mapped;

[0022] S25: For the points that can be mapped, divide the first two dimensions by the third dimension to obtain the image pixel space coordinates;

[0023] The coordinate vector of the point cloud is X pixel =[u,v,w,1] T , then the image pixel space coordinates (u pixel ,v pixel ), the expression is: pixel =u / w,v pixel =v / w;

[0024] S26: Divide the image pixel space coordinates obtained in step S25 by the image resolution to obtain the normalized image pixel space coordinates (u normalized ,v normalized ), the expression is: normalized =u pixel / W,v normalized =v pixel / H; where (W,H) is the resolution of the image;

[0025] S27: For points whose coordinate values ​​exceed the range of (0,1), it is considered that their mapping exceeds the image range and the point is removed. The normalized image pixel space coordinates are: 0≤u normalized ≤1,0≤v normalized ≤1;

[0026] If the condition is not met, the point is removed.

[0027] As a further description of the present invention, the intermediate space in step S3 includes a sparse space, and the establishment process of the sparse space is:

[0028] S31: normalize the horizontal and vertical coordinates of the BEV space to between 0 and 1, and divide it into grid units smaller than 0.02×0.02;

[0029] S32: Calculate the grids divided in step S31 according to the process of step S2 to obtain a list of grids that can be mapped into the image;

[0030] S33: randomly initialize k coordinate pairs whose values ​​are between 0 and 1. If they are in the grid list obtained in step S32, they are retained, otherwise they continue to be reinitialized;

[0031] S34: The obtained k coordinates are used as anchor points of the initialized intermediate space, and corresponding image features in the image pixel space are obtained for the k coordinates and the position mapping relationship calculated in step S2, and the image features constitute a sparse space.

[0032] As a further description of the present invention, the intermediate space in step S3 also includes a dense space, and the establishment process of the dense space is:

[0033] S35: normalize the horizontal and vertical coordinates of the BEV space to between 0 and 1, and divide it into grid units smaller than 0.02×0.02;

[0034] S36: Calculate the grids divided in step S35 according to the process of step S2 to obtain a list of grids that can be mapped into the image;

[0035] S37: Initialize coordinate pairs between 0 and 1 in one-to-one correspondence with the grid units in step S35. If they are in the grid list obtained in step S35, keep them, otherwise continue to reinitialize;

[0036] S38: Obtain the one-to-one coordinates of the grid units as the anchor points of the initialized intermediate space, obtain the corresponding image features in the image pixel space through the one-to-one coordinates of the grid units and the position mapping relationship calculated in step S2, and the image features constitute a dense space.

[0037] As a further description of the present invention, the intermediate space in step S3 also includes a small-sized dense space, and the small-sized dense space establishment process is:

[0038] Before step S35, the divided grid units are downsampled to obtain a small-size dense space divided grid, and steps S36-S38 are executed;

[0039] Through the above steps S36-S38, corresponding image features in the image pixel space are obtained, and the image features constitute a small-sized dense space.

[0040] As a further description of the present invention, in step S4, the image features are converted into image bird's-eye view spatial features, including the following steps:

[0041] S41: Calculate the distance weights of k sparse coordinates;

[0042] For each continuous coordinate (xi,yi), its four nearby discrete coordinates are:

[0043] For (x i ,y i ) and (x ij ,y ij ), whose weight w ij The expression is:

[0044]

[0045]

[0046]

[0047]

[0048] Among them, kk sparse coordinates {(x i ,y i ,z i )} k i＝1 and its corresponding image features {f i} k i＝1 ;

[0049] S42: For the continuous coordinates (x i ,y i ) The corresponding image feature f i , assign the weights calculated in step S41 to the four discrete coordinates, and the expression is:

[0050] F(x i1 ,yi1 )+=w i1 ·f i ,

[0051] F(x i2 ,y i2 )+=w i2 ·f i ,

[0052] F(x i3 ,y i3 )+=w i3 ·f i ,

[0053] F(x i4 ,y i4 )+=w i4 ·f i ;

[0054] After the above processing, the bird's-eye view spatial features of the image are obtained.

[0055] As a further description of the present invention, in step S5, the features of the point cloud bird's-eye view space are fused with the features of the image bird's-eye view space, including the following steps:

[0056] S51: concatenate the features and then reduce the dimension through the convolutional neural network layer;

[0057] Among them, the point cloud bird’s-eye view spatial feature is F point , the image bird's-eye view spatial feature is F image , the concatenated feature F concat , F concat =concat(F point ,F image );

[0058] The convolution kernel of the convolutional neural network layer is W conv , the bias is b, then the dimension reduction output feature is F fused , F fused =Conv(F concat ,W conv ,b);

[0059] Among them, Conv represents the convolution operation;

[0060] S52: Point cloud bird's eye view spatial feature F point and the image bird's-eye view spatial feature F image Add together to get the fused feature F fused :F fused =F point +F image .

[0061] Compared with the prior art, the technical effects of the present invention are:

[0062] The present invention provides a perception fusion method for cameras and lidar sensors. It does not directly correspond and fuse image / point cloud features, but establishes a virtual intermediate space for feature fusion outside the bird's-eye view space and the pixel space, which decouples the computational complexity of the feature fusion process from the image pixel space / point cloud bird's-eye view space size, significantly reduces the computational complexity required for fusion, and improves computational efficiency; at the same time, it takes into account the characteristics of the gradually popularized high-performance lidars, and better utilizes the dense point clouds collected by the high-performance lidars commonly used in intelligent driving vehicles; the intermediate space includes sparse space, dense space, and small-size dense space, but ultimately the intermediate space can be directly fused with the bird's-eye view space where the point cloud is located, giving a dense feature map containing sufficient point cloud information, thereby improving algorithm performance and feature versatility, and providing unified feature support for subsequent multiple prediction algorithms. BRIEF DESCRIPTION OF THE DRAWINGS

[0063] Figure 1 It is a schematic diagram of the overall flow of the perception fusion method of the present invention;

[0064] Figure 2 The intermediate space of the present invention is a fusion schematic diagram of a sparse space;

[0065] Figure 3 It is a schematic diagram of the fusion of the intermediate space of the present invention as a dense space;

[0066] Figure 4 It is a schematic diagram of the fusion of the intermediate space of the present invention as a small-sized dense space. DETAILED DESCRIPTION

[0067] The present invention is described in detail below in conjunction with the accompanying drawings:

[0068] The existing perception fusion methods include a bird's-eye view space for mapping point cloud features and a pixel space for mapping image features. The present invention discloses a perception fusion method for cameras and lidar sensors, which sets an intermediate space for image feature fusion outside the bird's-eye view space and the pixel space. Figure 1 As shown, the perception fusion method comprises the following steps:

[0069] S1: Obtain the three-dimensional point cloud data collected by the laser radar and the color image data collected by the camera, perform point cloud feature extraction and image feature extraction respectively, and map the point cloud features to the bird's-eye view space to form the point cloud bird's-eye view space, and map the image features to the pixel space to form the image pixel space;

[0070] S2: Using camera extrinsic parameters, intrinsic parameters and lidar extrinsic parameters, the mapping relationship between the position of the 3D point cloud coordinate system and the position of the image pixel space is calculated;

[0071] S3: According to the three-dimensional point cloud coordinate system, an intermediate space for image feature fusion is established to obtain image features filled in the intermediate space;

[0072] S4: using the correspondence between the three-dimensional point cloud coordinate system and the point cloud bird's-eye view space, the image features corresponding to the k sparse coordinates in the intermediate space are converted into image bird's-eye view space features;

[0073] S5: Fuse the features of the point cloud bird's-eye view space in step S1 with the features of the image bird's-eye view space obtained in step S4 to obtain fused bird's-eye view space features for downstream algorithm processing.

[0074] In this embodiment, in the above step S1, the feature extraction of the three-dimensional point cloud data and the color image data is specifically performed in the following manner:

[0075] For three-dimensional point cloud data: first, divide it into multiple point cloud columns according to the coordinates, and perform statistics on the highest point, lowest point, centroid point, and number of point clouds in each point cloud column. The statistics are used as the initial features of each point cloud column. Afterwards, the initial features of the point cloud column are input into the backbone network of deep learning for processing, and the output is the point cloud features of the three-dimensional point cloud data.

[0076] For color image data: the regularized color image is input into the deep learning backbone network for processing, and its output obtains the image features of the color image data.

[0077] In this embodiment, in the above step S2, the mapping of the position of the three-dimensional point cloud coordinate system to the position of the pixel space includes the following steps:

[0078] S21: Multiply the image external parameter matrix and the internal parameter matrix, and combine the laser radar external parameters to obtain the mapping matrix from the pixel space position to the three-dimensional point cloud coordinate system position. The expression is: P = K c [R c ∣t c ]·P l ;

[0079] The total camera matrix P c For: P c =K c [R c ∣t c ]; among them, K c is the internal parameter matrix, [R c ∣t c ] is the external parameter matrix, R c is the rotation matrix, t c is the translation vector; the external parameter P of the laser radar l For: P l =[Rl ∣t l ]; where R l is the rotation matrix of the lidar, t l is the translation vector of the lidar;

[0080] S22: Invert the total camera matrix to obtain the mapping matrix from the 3D point cloud coordinate system to the pixel space position, which is expressed as: P -1 =(K[R|t]) -1 ;

[0081] S23: Use the augmented coordinates of the three-dimensional point cloud coordinate system to multiply the matrix obtained in step S22 to obtain a four-dimensional vector, where the point X in the three-dimensional point cloud coordinate system is 3d =[X,Y,Z,1] T , and P -1 Multiply to get the coordinate X in pixel space pixel =P -1 X 3d ;

[0082] S24: Determine whether the third dimension of the four-dimensional vector in step S23 can be mapped; if X pixel The third dimension z pixel <0, the point cannot be mapped; that is, if it is a negative value, the point is actually behind the camera imaging plane and cannot be mapped;

[0083] S25: For the points that can be mapped, divide the first two dimensions by the third dimension to obtain the image pixel space coordinates;

[0084] The coordinate vector of the point cloud is X pixel =[u,v,w,1] T , then the image pixel space coordinates (u pixel ,v pixel ), the expression is: pixel =u / w,v pixel =v / w;

[0085] S26: Divide the image pixel space coordinates obtained in step S25 by the image resolution to obtain the normalized image pixel space coordinates (u normalized ,v normalized ), the expression is: normalized =u pixel / W,v normalized =v pixel / H; where (W,H) is the resolution of the image;

[0086] S27: For points whose coordinate values ​​exceed the range of (0,1), it is considered that their mapping exceeds the image range and the point is removed. The normalized image pixel space coordinates are: 0≤unormalized ≤1,0≤v normalized ≤1;

[0087] If the condition is not met, the point is removed.

[0088] In this embodiment, the intermediate space includes a sparse space, a dense space, and a small-sized dense space; Figure 2 As shown in , sparse space is a space composed of a small number of points; Figure 3 As shown in , the dense space is a densely packed space with the same size as the point cloud bird's-eye view space; Figure 4 As shown, a small-sized dense space is a smaller-sized dense space, and the amount of calculation is proportional to the size ratio of the fusion space and the point cloud bird's-eye view space. Specifically, it is explained through the following implementation method.

[0089] (1) The intermediate space in the above step S3 includes a sparse space, and the process of establishing the sparse space is as follows:

[0090] S31: normalize the horizontal and vertical coordinates of the BEV space to between 0 and 1, and divide it into extremely small grid units (an extremely small grid unit is a grid unit smaller than 0.02×0.02, for example, a length and width of 0.001×0.001);

[0091] S32: enumerate the grid units divided in step S31 one by one, and obtain a grid list that can be mapped to the image according to the external parameters and calculation process calculated in step S2;

[0092] S33: randomly initialize k coordinate pairs whose values ​​are between 0 and 1. If they are in the grid list obtained in step S32, they are retained, otherwise they continue to be reinitialized;

[0093] S34: The obtained k coordinates are used as anchor points of the initialized intermediate space. The corresponding image features in the image pixel space are obtained based on the above k coordinates and the position mapping relationship calculated in step S2. The image features constitute a sparse space.

[0094] (2) The intermediate space in the above step S3 also includes a dense space, and the establishment process of the dense space is as follows:

[0095] S35: normalizing the horizontal and vertical coordinates of the BEV space to between 0 and 1, and dividing it into extremely small grid units (an extremely small grid unit is a grid unit smaller than 0.02×0.02, for example, a length and width of 0.001×0.001);

[0096] S36: enumerate the grid units divided in step S35 one by one, and obtain a grid list that can be mapped to the image according to the external parameters and calculation process calculated in step S2;

[0097] S37: Initialize coordinate pairs between 0 and 1 in one-to-one correspondence with the grid units in step S35 (if the grid unit in step S35 is 0.001×0.001 in length and width, it corresponds to 1000×1000 coordinates). If it is in the grid list obtained in step S35, keep it, otherwise continue to reinitialize;

[0098] S38: Obtain the one-to-one coordinates of the grid units as the anchor points of the initialized intermediate space, obtain the corresponding image features in the image pixel space through the one-to-one coordinates of the grid units and the position mapping relationship calculated in step S2, and the image features constitute a dense space.

[0099] (3) The intermediate space in the above step S3 also includes a small-sized dense space, and the process of establishing the small-sized dense space is as follows:

[0100] Before step S35, the divided grid units are downsampled to obtain a small-size dense space divided grid, and steps S36-S38 are executed; if the length and width of the grid unit in step S35 is 0.001×0.001, it can be upsampled to 0.002×0.002 grid units, which corresponds to 500×500 coordinates;

[0101] Through the above steps S36-S38, corresponding image features in the image pixel space are obtained, and the image features constitute a small-sized dense space.

[0102] In this embodiment, in the above step S4, the image features are converted into the image bird's-eye view spatial features, including the following steps:

[0103] S41: Calculate the distance weights of k sparse coordinates;

[0104] Since the coordinates are continuous coordinates (floating point numbers) and the bird's-eye view space is discrete coordinates (integers), in this embodiment, the distance weights of four discrete coordinates near the continuous coordinates (rounding up and rounding down the horizontal and vertical coordinates) are calculated using a bilinear interpolation method; specifically, for each continuous coordinate (xi, yi), the four nearby discrete coordinates are found to be:

[0105] For (x i ,y i ) and (x ij ,y ij ), bilinear interpolation weights w ij The expression is:

[0106]

[0107]

[0108]

[0109]

[0110] Among them, kk sparse coordinates {(x i ,y i ,z i )} k i＝1 and its corresponding image features {f i} k i＝1 ;

[0111] S42: For the continuous coordinates (x i ,y i ) The corresponding image feature f i , assign the weights calculated in step S41 to the four discrete coordinates (i.e., multiply the features corresponding to the coordinates by the weights), and the expression is:

[0112] F(x i1 ,y i1 )+=w i1 ·f i ,

[0113] F(x i2 ,y i2 )+=w i2 ·f i ,

[0114] F(x i3 ,y i3 )+=w i3 ·f i ,

[0115] F(x i4 ,y i4 )+=w i4 ·f i ;

[0116] After the above processing, the bird's-eye view spatial features of the image are obtained.

[0117] In this embodiment, in the above step S5, the features of the point cloud bird's-eye view space are fused with the features of the image bird's-eye view space, including the following steps:

[0118] S51: concatenate the features and then reduce the dimension through the convolutional neural network layer;

[0119] Among them, the point cloud bird's-eye view spatial feature in the original step S1 is F point, the image bird's-eye view spatial feature obtained in step S4 is F image , F point and F image Concatenate on the feature dimension to obtain the concatenated feature F concat , F concat =concat(F point ,F image ); then a layer of convolutional neural network (CNN) is used for dimensionality reduction;

[0120] The convolution kernel of the convolutional neural network layer is W conv , the bias is b, then the dimension reduction output feature is F fused , F fused =Conv(F concat ,W conv ,b);

[0121] Among them, Conv represents the convolution operation;

[0122] S52: Point cloud bird's eye view spatial feature F point and the image bird's-eye view spatial feature F image Add together to get the fused feature F fused , F fused =F point +F image .

[0123] The fused bird's-eye view spatial features are handed over to the downstream algorithm for processing.

[0124] Through the above embodiments, the perception fusion method of the present invention is disclosed. Compared with the prior art, the present invention has the following advantages:

[0125] 1. The perception fusion method of the present invention does not directly correspond and fuse the image / point cloud features, but establishes a virtual intermediate space for feature fusion, which decouples the computational complexity of the feature fusion process from the association between the image pixel space / point cloud bird's-eye view space size, thereby significantly reducing the computational complexity required for fusion and significantly improving computational efficiency;

[0126] 2. The computational complexity of the fusion process of the present invention is related to the size of the image pixel space / point cloud bird's-eye view space, taking into account the characteristics of the increasingly popular high-performance laser radars, and making better use of the dense point clouds collected by the high-performance laser radars commonly used in intelligent driving vehicles;

[0127] 3. The intermediate space of the present invention can be a sparse space, a dense space, or a small-size dense space, but ultimately the intermediate space can be directly merged with the bird's-eye view space where the point cloud is located to provide a dense feature map containing sufficient point cloud information for use by downstream algorithm modules, thereby improving algorithm performance and feature versatility.

[0128] The above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Other modifications or equivalent substitutions made to the technical solution of the present invention by ordinary technicians in this field should be included in the scope of the claims of the present invention as long as they do not depart from the spirit and scope of the technical solution of the present invention.

Claims

1. A perception fusion method for a camera and a lidar sensor, the fusion method comprising a bird's-eye view space for mapping point cloud features and a pixel space for mapping image features, characterized in that: An intermediate space for image feature fusion is set outside the bird's-eye view space and the pixel space; The perception fusion method comprises the following steps: S1: Obtain the three-dimensional point cloud data collected by the laser radar and the color image data collected by the camera, perform point cloud feature extraction and image feature extraction respectively, and map the point cloud features to the bird's-eye view space to form the point cloud bird's-eye view space, and map the image features to the pixel space to form the image pixel space; S2: Using camera extrinsic parameters, intrinsic parameters and lidar extrinsic parameters, the mapping relationship between the position of the 3D point cloud coordinate system and the position of the image pixel space is calculated; S3: According to the three-dimensional point cloud coordinate system, an intermediate space for image feature fusion is established to obtain image features filled in the intermediate space; S4: using the correspondence between the three-dimensional point cloud coordinate system and the point cloud bird's-eye view space, the image features corresponding to the k sparse coordinates in the intermediate space are converted into image bird's-eye view space features; S5: Fuse the features of the point cloud bird's-eye view space in step S1 with the features of the image bird's-eye view space obtained in step S4 to obtain fused bird's-eye view space features for downstream algorithm processing.

2. The perception fusion method for a camera and a lidar sensor according to claim 1, characterized in that: In the step S1, the three-dimensional point cloud data is divided into a plurality of point cloud columns according to the coordinates, and the highest point, the lowest point, the centroid point, and the number of point clouds in each point cloud column are statistically counted, and the statistics are used as the initial features of each point cloud column. The point cloud features of the three-dimensional point cloud data are obtained by inputting the initial features of the point cloud column into the backbone network of deep learning; For color image data, the image features of the color image data are obtained by inputting the regularized color image into the deep learning backbone network.

3. The perception fusion method for a camera and a laser radar sensor according to claim 1, characterized in that: In step S2, the mapping of the position of the three-dimensional point cloud coordinate system to the position of the pixel space includes the following steps: S21: Multiply the image external parameter matrix and the internal parameter matrix, and combine the laser radar external parameters to obtain the mapping matrix from the pixel space position to the three-dimensional point cloud coordinate system position. The expression is: P = K c [R c ∣t c ]·P l ; The total camera matrix P c For: P c =K c [R c ∣t c ]; among them, K c is the internal parameter matrix, [R c ∣t c ] is the external parameter matrix, R c is the rotation matrix, t c is the translation vector; The external parameter P of the laser radar l For: P l =[R l ∣t l ]; where R l is the rotation matrix of the lidar, t l is the translation vector of the lidar; S22: Invert the total camera matrix to obtain the mapping matrix from the 3D point cloud coordinate system to the pixel space position, which is expressed as: P -1 =(K c [R c ∣t c ]·P l ) -1 ; S23: Use the augmented coordinates of the three-dimensional point cloud coordinate system to multiply the matrix obtained in step S22 to obtain a four-dimensional vector, where the point X in the three-dimensional point cloud coordinate system is 3d =[X,Y,Z,1] T , and P -1 Multiply to get the coordinate X in pixel space pixel =P -1 X 3d ; S24: Determine whether the third dimension of the four-dimensional vector in step S23 can be mapped; if X pixel The third dimension z pixel <0, then this point cannot be mapped; S25: For the points that can be mapped, divide the first two dimensions by the third dimension to obtain the image pixel space coordinates; The coordinate vector of the point cloud is X pixel =[u,v,w,1] T , then the image pixel space coordinates (u pixel ,v pixel ), the expression is: pixel =u / w,v pixel =v / w; S26: Divide the image pixel space coordinates obtained in step S25 by the image resolution to obtain the normalized image pixel space coordinates (u normalized ,v normalized ), the expression is: normalized =u pixel / W, v normalized =v pixel / H; where (W,H) is the resolution of the image; S27: For points whose coordinate values ​​exceed the range of (0,1), it is considered that their mapping exceeds the image range and the point is removed. The normalized image pixel space coordinates are: 0≤u normalized ≤1,0≤v normalized ≤1; If the condition is not met, the point is removed.

4. The perception fusion method for a camera and a laser radar sensor according to claim 3, characterized in that: The intermediate space in step S3 includes a sparse space, and the establishment process of the sparse space is as follows: S31: normalize the horizontal and vertical coordinates of the BEV space to between 0 and 1, and divide it into grid units smaller than 0.02×0.02; S32: Calculate the grids divided in step S31 according to the process of step S2 to obtain a list of grids that can be mapped into the image; S33: randomly initialize k coordinate pairs whose values ​​are between 0 and 1. If they are in the grid list obtained in step S32, they are retained, otherwise they continue to be reinitialized; S34: The obtained k coordinates are used as anchor points of the initialized intermediate space, and corresponding image features in the image pixel space are obtained for the k coordinates and the position mapping relationship calculated in step S2, and the image features constitute a sparse space.

5. The perception fusion method for a camera and a laser radar sensor according to claim 3, characterized in that: The intermediate space in step S3 also includes a dense space, and the establishment process of the dense space is as follows: S35: normalize the horizontal and vertical coordinates of the BEV space to between 0 and 1, and divide it into grid units smaller than 0.02×0.02; S36: Calculate the grids divided in step S35 according to the process of step S2 to obtain a list of grids that can be mapped into the image; S37: Initialize coordinate pairs between 0 and 1 in one-to-one correspondence with the grid units in step S35. If they are in the grid list obtained in step S35, keep them, otherwise continue to reinitialize; S38: Obtain the one-to-one coordinates of the grid units as the anchor points of the initialized intermediate space, obtain the corresponding image features in the image pixel space through the one-to-one coordinates of the grid units and the position mapping relationship calculated in step S2, and the image features constitute a dense space.

6. The perception fusion method for a camera and a laser radar sensor according to claim 5, characterized in that: The intermediate space in step S3 also includes a small-sized dense space. The small-sized dense space is established by: before step S35, downsampling the divided grid units to obtain grids divided by the small-sized dense space, and executing steps S36-S38; Through the above steps S36-S38, corresponding image features in the image pixel space are obtained, and the image features constitute a small-sized dense space.

7. The perception fusion method for a camera and a laser radar sensor according to claim 1, characterized in that: In step S4, the image features are converted into image bird's-eye view spatial features, including the following steps: S41: Calculate the distance weights of k sparse coordinates; For each continuous coordinate (xi,yi), its four nearby discrete coordinates are: For (x i ,y i ) and (x ij ,y ij ), whose weight w ij The expression is: Among them, kk sparse coordinates {(x i ,y i ,z i )} k i＝1 and its corresponding image features {f i } k i＝1 ; S42: For the continuous coordinates (x i ,y i ) The corresponding image feature f i , assign the weights calculated in step S41 to the four discrete coordinates, and the expression is: F(x i1 ,y i1 )+=w i1 ·f i , F(x i2 ,y i2 )+=w i2 ·f i , F(x i3 ,y i3 )+=w i3 ·f i , F(x i4 ,y i4 )+=w i4 ·f i ; After the above processing, the bird's-eye view spatial features of the image are obtained.

8. The perception fusion method for a camera and a laser radar sensor according to claim 1, characterized in that: In step S5, the features of the point cloud bird's-eye view space are fused with the features of the image bird's-eye view space, including the following steps: S51: concatenate the features and then reduce the dimension through the convolutional neural network layer; Among them, the point cloud bird’s-eye view spatial feature is F point , the image bird's-eye view spatial feature is F image , the concatenated feature F concat , F concat =concat(F point ,F image ); The convolution kernel of the convolutional neural network layer is W conv , the bias is b, then the dimension reduction output feature is F fused , F fused =Conv(F concat ,W conv ,b); Among them, Conv represents the convolution operation; S52: The point cloud bird's eye view spatial feature F point and the image bird's-eye view spatial feature F image Add together to get the fused feature F fused : F fused =F point +F image .

Images