Vehicle target detection method based on binocular camera and laser radar fusion

Abstract

The invention discloses a vehicle target detection method based on binocular camera and laser radar fusion, and the method is characterized in that the method comprises the following steps: obtaining a left eye image and a right eye image of a vehicle target at a preset moment of the binocular camera, inputting the left eye image and the right eye image into a stereo matching network, and outputting a disparity map; inputting the false point cloud image into an image semantic segmentation network, segmenting a vehicle through the image semantic segmentation network, then generating a false point cloud only containing vehicle information, setting epsilon reliability and reducing the influence of noise by introducing an adaptive fusion factor delta and combining parallax prediction, and then fusing the true and false point clouds in a data set; a vehicle target is detected by rasterizing a vehicle target detection network, and the position, category and orientation of a three-dimensional target frame of a vehicle are obtained. According to the method, the precision of depth estimation, the effectiveness of the false point cloud and the precision of vehicle target detection are improved.

Description

Technical Field

[0001] The present disclosure belongs to the field of transportation technology, and in particular relates to a vehicle target detection method based on the fusion of a binocular camera and a laser radar. Background Art

[0002] In recent years, driverless technology has made great progress, and detecting and identifying objects on the road ahead is one of the important tasks of driverless vehicles. LiDAR and camera are the two major sensors that support target detection tasks. LiDAR can provide accurate three-dimensional distance information, so it is widely used. However, the cost of LiDAR increases significantly with the increase in the number of laser beams equipped. Therefore, the number of laser beams of LiDAR equipped on the vehicle is usually small, but the small number of laser beams makes the point cloud scanned by it relatively sparse, which is not conducive to target detection. Cameras, as low-priced sensors, are also widely used for object detection, but because the two-dimensional image data lacks distance information, cameras usually cannot complete the task of object detection in three-dimensional space well.

[0003] In order to make better use of images, the PSMNet network uses binocular RGB images to estimate disparity maps and generate depth images, which gives two-dimensional images distance information, called RGB-D images. However, due to the large error of image-based depth estimation, the performance of 3D target detection based on RGB-D is poor. The large gap between cameras and lidar in 3D target detection accuracy is not entirely due to the difference in the quality of the depth data of the two sensors, but a large part of the reason is due to the different representation of the data. The RGB-D image is used to generate a pseudo-lidar point cloud, and the existing 3D point cloud target detection method is used for detection, which greatly improves the accuracy of the image in 3D target detection. The pseudo point cloud is corrected using sparse lidar point clouds to further improve the accuracy of the pseudo point cloud. The above studies show that representing image data in 3D space for target detection is a better choice than directly performing target detection in 2D space. However, although the use of pseudo point clouds has improved the accuracy of previous image-based 3D target detection, the low coordinate accuracy of pseudo point clouds still leads to a large gap between it and the 3D target detection accuracy based on lidar point clouds. Summary of the invention

[0004] The present disclosure is proposed based on the above-mentioned needs of the prior art. The technical problem to be solved by the present disclosure is to provide a vehicle target detection method based on the fusion of binocular camera and lidar, which improves the accuracy of depth estimation, the effectiveness of pseudo point cloud and the accuracy of vehicle target detection.

[0005] In order to solve the above problems, the technical solutions provided by the present disclosure include:

[0006] A vehicle target detection method based on the fusion of a binocular camera and a laser radar is provided, which is characterized by comprising the following steps: a first step, obtaining a left-eye image and a right-eye image of a vehicle target at a predetermined time from a binocular camera, inputting the left-eye image and the right-eye image into a stereo matching network and then outputting a disparity map, wherein the calculation process of the stereo matching network includes:

[0007] S101, input the left eye image and the right eye image captured by the binocular camera into the MobileNetV2 network, extract the features of the left eye image and the right eye image, and output the left eye feature map and the right eye feature map. The feature extraction formula is: Conv input =H×W×C, where Conv represents a convolution operation; input represents an input left-eye image and a right-eye image; H×W×C represents a left-eye feature map or a right-eye feature map; S102, the left-eye feature map and the right-eye feature map H×W×C are input into a cross-image aggregation module, and for a pixel in the left-eye feature map, D pixels are selected to the left of the pixel at the same coordinates as the pixel on the right-eye feature map, and the similarity between the pixel in the left-eye feature map and the D pixels is calculated, and the similarity calculation formula is: Cost(p,d)=∑M l (x,y,C l ′,p)M r (x+d,y,C r ′, p), where Cost(p, d) represents the similarity or matching cost, that is, the cost value of the disparity d at the p-th pixel in the left eye feature map; M l represents the left eye feature map, M r represents the right eye feature map, d = 1, 2, 3, ..., D, according to experience, D is 192; x, y represent the coordinates of the pth pixel; S103, the matching cost obtained by the cross-graph aggregation module in S102 is input into the cost aggregation network to obtain the updated matching cost; S104, the two values ​​with the highest similarity of each pixel in the D dimension in the final matching cost are retained, and then the two pixels are normalized by the Softmax function, and the two dimensions are softly aggregated to obtain the initial disparity of each pixel, and then the full-resolution disparity map is obtained by disparity subdivision; S105, in the binocular camera, according to the geometric imaging principle of stereo vision, the depth calculation process is as follows:

[0008]

[0009] Among them, f is the focal length; d is the parallax, that is, point p l and p r In X aThe difference in the axis direction; Z is the depth information. After obtaining the depth information, the image coordinate system can be converted to the world coordinate system according to the camera's internal parameters, and finally converted to a pseudo point cloud. The calculation process of the conversion is

[0010]

[0011] Among them, x, y, z are the coordinates in the point cloud coordinate system; x′, y′ are the coordinates in the image coordinate system; Z is the depth; is the internal parameter of the camera; in the second step, the pseudo point cloud is input into the image semantic segmentation network, the vehicle is first segmented by the image semantic segmentation network, and then a pseudo point cloud containing only vehicle information is generated. By introducing an adaptive fusion factor δ and combining the parallax prediction confidence ε, the influence of noise is reduced, and then the true and false point clouds are fused in the data set. The fusion calculation process is:

[0012]

[0013] Among them, P is the coordinate of the fused point cloud; is the real point cloud coordinate, r is the reflection intensity;

[0014] is the pseudo point cloud coordinate; ε is the parallax prediction confidence; δ is the adaptive fusion factor, δ=1 when ε>0.9, δ=0 when ε≤0.9; the third step is to detect the vehicle target through the gridded vehicle target detection network to obtain the position, category and orientation of the three-dimensional target frame of the vehicle; S301, the grid encoder receives the fused point cloud and divides the fused point cloud into grid columns of uniform size, each grid column has a cross-sectional size of l×l and extends infinitely in the z direction; the grid encoder encodes the points in each grid column and finally outputs a tensor of dimension [D,P,N], where D=9, indicating the dimension of each point being encoded, P=12000, indicating the number of grid columns, and N=100, indicating the number of points in each grid column, wherein each point in the grid column is encoded as a vector [r,x,y,z,x c ,y c ,z c ,x p ,y p ], r is the reflection intensity, x, y, z are the coordinates of a predetermined point in the grid column, x c ,y c ,z c is the distance from this point to the arithmetic geometric center of all points in the grid column, x p ,y pis the distance from the point to the center point of the grid column; S302, the backbone ResNet-18 receives the tensor of dimension [D, P, N] output by the grid encoder, performs further feature extraction based on the preliminary features extracted by the grid encoder, and then uses the neck to aggregate the features of the backbone; S303, the detection head receives the features output by the backbone and the neck, performs vehicle target detection, and then outputs the position, category and orientation of the vehicle target frame. The position of the vehicle target frame is defined by 7 parameters: (x, y, z, w, l, h, θ), where x, y, z represent the coordinates of the center point of the vehicle target frame, w, l, h represent the width, length and height of the vehicle target frame, and θ represents the deflection angle of the vehicle target frame.

[0015] Preferably, the left-eye feature map and the right-eye feature map are multi-scale, and their resolutions are 1 / 4, 1 / 8, 1 / 16, and 1 / 32 of the resolutions of the left-eye image and the right-eye image, respectively.

[0016] Preferably, when the left-eye feature map and the right-eye feature map H×W×C are input into the cross-graph aggregation module, the left-eye feature map and the right-eye feature map are converted into multi-head dimensions, that is, the original C dimension is converted into N×C' dimension.

[0017] Preferably, the cost aggregation network adopts an hourglass network, which is composed of multiple 3D convolutional layers. The intermediate network layer will output matching costs of different scales, which are 1 / 4, 1 / 8, 1 / 16, and 1 / 32 resolutions respectively. The calculation process of the hourglass network includes: first inputting the matching cost obtained by the cross-graph aggregation module into the 3D convolutional layer, and then adding it to the matching cost output by the intermediate network layer to obtain a new matching cost. The new matching cost and the previously obtained updated left-eye feature map of the corresponding scale are input into the adaptive guidance network, and the final output matching cost is then passed through the 3D convolutional layer, wherein the adaptive guidance network includes The next step is: converting the C dimension of the updated left-eye feature map into an N×C' dimension, and initializing a C-dimensional query vector, mapping the query vector to an N×C' dimension, i.e., 1×N×C'; calculating the similarity between the query vector and the updated left-eye feature map, and obtaining N similarities corresponding to each pixel in the updated left-eye feature map, i.e., N weights; then using the N similarities to update the query vector, so that the N groups of features of the query vector can represent a feature representation of an average grouping of all pixels in a specific image; and calculating the N similarities between the updated query vector and the left-eye feature map again to obtain an updated matching cost.

[0018] Compared with the prior art, the present invention improves the accuracy of depth estimation, the effectiveness of pseudo point clouds and the accuracy of vehicle target detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the embodiments of this specification or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the embodiments of this specification. For ordinary technicians in this field, other drawings can also be obtained based on these drawings.

[0020] Figure 1 It is a binocular camera imaging principle diagram provided by a specific embodiment of the present invention. DETAILED DESCRIPTION

[0021] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0022] To facilitate understanding of the embodiments of the present invention, further explanation will be given below with reference to specific embodiments in conjunction with the accompanying drawings. The embodiments do not constitute a limitation on the protection scope of the present invention.

[0023] The present application discloses a vehicle target detection method based on the fusion of a binocular camera and a laser radar, and the vehicle target detection method comprises the following steps:

[0024] The first step is to obtain the left eye image and the right eye image of the vehicle target at a predetermined time from the binocular camera, input the left eye image and the right eye image into the stereo matching network, and then output a disparity map.

[0025] The calculation process of the stereo matching network includes:

[0026] S101, input the left eye image and the right eye image captured by the binocular camera into the MobileNetV2 network, perform feature extraction on the left eye image and the right eye image, and output the left eye feature map and the right eye feature map. The feature extraction formula is:

[0027] Conv input =H×W×C

[0028] Wherein, Conv represents the convolution operation; input represents the input left-eye image and right-eye image; H×W×C represents the left-eye feature map or the right-eye feature map.

[0029] The left-eye feature map and the right-eye feature map are multi-scale, and their resolutions are 1 / 4, 1 / 8, 1 / 16, and 1 / 32 of the resolutions of the left-eye image and the right-eye image, respectively.

[0030] S102, input the left-eye feature map and the right-eye feature map H×W×C into a cross-graph aggregation module, and convert the left-eye feature map and the right-eye feature map into multi-head dimensions, that is, convert the original C dimension into N×C' dimension.

[0031] For a pixel in the left feature map, find the matching point of the pixel on the right feature map, that is, select D pixels on the left side of the same coordinates as the pixel in the right feature map, and calculate the similarity between the pixel in the left feature map and the D pixels. The similarity calculation formula is:

[0032] Cost(p,d)=∑M l (x,y,C l ′,p)M r (x+d,y,C r ′,p)

[0033] Wherein, Cost(p, d) represents the similarity, i.e., the matching cost, that is, the cost value of the disparity d at the p-th pixel in the left eye feature map; M l represents the left eye feature map, M r represents the right eye feature map, d = 1, 2, 3, ..., D. According to experience, D is 192; x, y represent the coordinates of the pth pixel.

[0034] S103, input the matching cost obtained by the cross-graph aggregation module in S102 into the cost aggregation network. This specific implementation preferably adopts an hourglass network, which is composed of multiple 3D convolutional layers. The intermediate network layer will output matching costs of different scales, which are 1 / 4, 1 / 8, 1 / 16, and 1 / 32 resolutions respectively.

[0035] The calculation process of the hourglass network includes: first inputting the matching cost obtained by the cross-graph aggregation module into the 3D convolution layer, then adding it to the matching cost output by the intermediate network layer to obtain a new matching cost, and inputting the new matching cost and the previously obtained updated left-eye feature map of the corresponding scale into the adaptive guidance network, and the final output matching cost passes through the 3D convolution layer.

[0036] The adaptive guidance network includes the following steps:

[0037] Convert the C dimension of the updated left-eye feature map into an N×C' dimension, initialize a C-dimensional query vector, and map the query vector into an N×C' dimension, i.e., 1×N×C';

[0038] Calculate the similarity between the query vector and the updated left-eye feature map to obtain N similarities, i.e., N weights, corresponding to each pixel in the updated left-eye feature map. Then use the N similarities to update the query vector so that the N groups of features in the query vector can represent an average grouping feature representation of all pixels in a specific image;

[0039] The N similarities between the updated query vector and the left-eye feature map are calculated again to obtain an updated matching cost.

[0040] S104, retaining the two values ​​with the highest similarity of each pixel in the D dimension in the final matching cost, and then normalizing the two pixels through the Softmax function, and soft-aggregating the two dimensions to obtain the initial disparity of each pixel, and then obtaining the full-resolution disparity map through disparity subdivision.

[0041] S105, in the binocular camera, the left imaging center O l and right imaging center O r The line connecting the left camera and the right camera is the baseline b; the left camera optical axis Z l and the right camera optical axis Z r Parallel, and straight line Z l Z r and X in the spatial coordinate system a Axis perpendicular. A point P in three-dimensional space and the two imaging center points O l , O r And the mapping point of point P on the imaging plane of the left and right cameras, that is, point p l and p r In the same plane, the intersection of the plane and the two imaging planes is the epipolar line, such as Figure 1 As shown. According to the geometric imaging principle of stereo vision, the depth calculation process is:

[0042]

[0043] Among them, f is the focal length; d is the parallax, that is, point p l and p r In X a The difference in the axis direction; Z is the depth information.

[0044] After obtaining the depth information, the image coordinate system can be converted to the world coordinate system according to the camera's internal parameters, and finally converted to a pseudo point cloud. The calculation process of the conversion is:

[0045]

[0046] Among them, x, y, z are the coordinates in the point cloud coordinate system; x′, y′ are the coordinates in the image coordinate system; Z is the depth; is the internal parameter of the camera.

[0047] In the second step, the pseudo point cloud is input into the image semantic segmentation network. The vehicle is first segmented through the image semantic segmentation network, and then a pseudo point cloud containing only vehicle information is generated, and unnecessary point clouds are eliminated to reduce the amount of calculation. In order to solve the problem of pseudo point cloud data tailing and noise, the adaptive fusion factor δ is introduced, combined with the disparity prediction confidence ε, to reduce the impact of noise. Points with a disparity prediction confidence greater than 0.9 are retained, and points with a disparity prediction confidence less than 0.9 are eliminated, so as to correct the pseudo point cloud and reduce the impact of noise. The true and false point clouds are then fused in the data set. The fusion calculation process is as follows:

[0048]

[0049] Among them, P is the coordinate of the fused point cloud; is the real point cloud coordinate, r is the reflection intensity; is the pseudo point cloud coordinate; ε is the disparity prediction confidence; δ is the adaptive fusion factor, δ=1 when ε>0.9, and δ=0 when ε≤0.9.

[0050] The third step is to detect the vehicle target through the rasterized vehicle target detection network to obtain the position, category and orientation of the vehicle's three-dimensional target box.

[0051] The gridded vehicle target detection network includes:

[0052] S301, the grid encoder receives the fused point cloud and divides the fused point cloud into grid columns of uniform size, each grid column has a cross-sectional size of l×l and extends infinitely in the z direction.

[0053] The grid encoder encodes the points in each grid column and finally outputs a tensor of dimension [D, P, N], where D = 9, indicating the dimension of each point being encoded, P = 12000, indicating the number of grid columns, and N = 100, indicating the number of points in each grid column. Each point in the grid column is encoded as a vector with 9 dimensions

[0054] [r,x,y,z,x c ,y c ,z c ,x p ,y p ], r is the reflection intensity, x, y, z are the coordinates of a predetermined point in the grid column, x c ,y c ,z c is the distance from this point to the arithmetic geometric center of all points in the grid column, x p ,y pThe distance from this point to the center of the grid column.

[0055] S302, the backbone part ResNet-18 receives the tensor of dimension [D, P, N] output by the grid encoder, and performs further feature extraction based on the preliminary features extracted by the grid encoder.

[0056] Then, the features of the backbone are aggregated using the neck to enlarge the receptive field and fuse multi-scale contexts.

[0057] S303, the detection head receives the features output by the trunk and the neck, performs vehicle target detection, and outputs the position, category, and orientation of the vehicle target frame. The position of the vehicle target frame is defined by 7 parameters (x, y, z, w, l, h, θ), where x, y, z represent the coordinates of the center point of the vehicle target frame, w, l, h represent the width, length, and height of the vehicle target frame, and θ represents the deflection angle of the vehicle target frame.

[0058] The specific implementation methods described above further illustrate the objectives, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above description is only a specific implementation method of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A vehicle target detection method based on the fusion of binocular camera and laser radar, characterized in that: The following steps are involved: The first step is to obtain the left eye image and the right eye image of the vehicle target at a predetermined time from the binocular camera, input the left eye image and the right eye image into the stereo matching network, and then output the disparity map. The calculation process of the stereo matching network includes: S101, input the left eye image and the right eye image captured by the binocular camera into the MobileNetV2 network, perform feature extraction on the left eye image and the right eye image, and output the left eye feature map and the right eye feature map. The feature extraction formula is: Conv input =H×W×C Among them, Conv represents the convolution operation; input represents the input left-eye image and right-eye image; H×W×C represents the left-eye feature map or the right-eye feature map; S102, input the left eye feature map and the right eye feature map H×W×C into the cross-image aggregation module, for a pixel in the left eye feature map, find the pixel on the right eye feature map at the same coordinates as the pixel on the left side of the pixel, select D pixels, and calculate the similarity between the pixel in the left eye feature map and the D pixels. The similarity calculation formula is: Cost(p,d)=∑M l (x,y,C l ′,p)M r (x+d,y,C r ′,p) Wherein, Cost(p, d) represents the similarity, i.e., the matching cost, that is, the cost value of the disparity d at the p-th pixel in the left eye feature map; M l represents the left eye feature map, M r represents the right eye feature map, d = 1, 2, 3, ..., D, according to experience, D is 192; x, y represent the coordinates of the pth pixel; S103, inputting the matching cost obtained by the cross-graph aggregation module in S102 into the cost aggregation network to obtain an updated matching cost; S104, retaining the two values ​​with the highest similarity of each pixel in the D dimension in the final matching cost, and then normalizing the two pixels through the Softmax function, and soft-aggregating the two dimensions to obtain the initial disparity of each pixel, and then obtaining a full-resolution disparity map through disparity subdivision; S105, in a binocular camera, according to the geometric imaging principle of stereo vision, the depth calculation process is: Among them, f is the focal length; d is the parallax, that is, point p l and p r In X a The difference in the axis direction; Z is the depth information, After obtaining the depth information, the image coordinate system can be converted to the world coordinate system according to the camera's internal parameters, and finally converted to a pseudo point cloud. The calculation process of the conversion is: Among them, x, y, z are the coordinates in the point cloud coordinate system; x′, y′ are the coordinates in the image coordinate system; Z is the depth; is the internal parameter of the camera; In the second step, the pseudo point cloud is input into the image semantic segmentation network. The vehicle is first segmented by the image semantic segmentation network, and then a pseudo point cloud containing only vehicle information is generated. The influence of noise is reduced by introducing an adaptive fusion factor δ and combining the parallax prediction confidence ε. Then, the true and false point clouds are fused in the data set. The fusion calculation process is as follows: Among them, P is the coordinate of the fused point cloud; is the real point cloud coordinate, r is the reflection intensity; is the pseudo point cloud coordinate; ε is the disparity prediction confidence; δ is the adaptive fusion factor, δ=1 when ε>0.9, δ=0 when ε≤0.9; The third step is to detect the vehicle target through the rasterized vehicle target detection network to obtain the position, category and orientation of the vehicle's three-dimensional target box; S301, a grid encoder receives a fused point cloud and divides the fused point cloud into grid columns of uniform size, each grid column having a cross-sectional size of l×l and extending infinitely in the z direction; The grid encoder encodes the points in each grid column and finally outputs a tensor of dimension [D, P, N], where D = 9, indicating the dimension of each point being encoded, P = 12000, indicating the number of grid columns, and N = 100, indicating the number of points in each grid column, wherein each point in the grid column is encoded as a vector with 9 dimensions [r,x,y,z,x c ,y c ,z c ,x p ,y p ], r is the reflection intensity, x, y, z are the coordinates of a predetermined point in the grid column, x c ,y c ,z c is the distance from this point to the arithmetic geometric center of all points in the grid column, x p ,y p is the distance from this point to the center of the grid column; S302, the backbone ResNet-18 receives the tensor of dimension [D, P, N] output by the grid encoder, performs further feature extraction based on the preliminary features extracted by the grid encoder, and then aggregates the features of the backbone using the neck; S303, the detection head receives the features output by the trunk and the neck, performs vehicle target detection, and outputs the position, category and orientation of the vehicle target frame. The position of the vehicle target frame is defined by 7 parameters (x, y, z, w, l, h, θ), where x, y, z represent the coordinates of the center point of the vehicle target frame, w, l, h represent the width, length and height of the vehicle target frame, and θ represents the deflection angle of the vehicle target frame.

2. The vehicle target detection method based on the fusion of binocular camera and laser radar according to claim 1 is characterized in that: The left-eye feature map and the right-eye feature map are multi-scale, and their resolutions are 1 / 4, 1 / 8, 1 / 16, and 1 / 32 of the resolutions of the left-eye image and the right-eye image, respectively.

3. The vehicle target detection method based on the fusion of binocular camera and laser radar according to claim 1 is characterized in that: When the left-eye feature map and the right-eye feature map H×W×C are input into the cross-graph aggregation module, the left-eye feature map and the right-eye feature map are converted into multi-head dimensions, that is, the original C dimension is converted into N×C' dimension.

4. The vehicle target detection method based on the fusion of binocular camera and laser radar according to claim 1 is characterized in that: The cost aggregation network adopts an hourglass network, which is composed of multiple 3D convolutional layers. The middle network layer will output matching costs of different scales, which are 1 / 4, 1 / 8, 1 / 16, and 1 / 32 resolutions respectively. The calculation process of the hourglass network includes: first inputting the matching cost obtained by the cross-graph aggregation module into the 3D convolution layer, then adding it to the matching cost output by the intermediate network layer to obtain a new matching cost, and inputting the new matching cost and the previously obtained updated left-eye feature map of the corresponding scale into the adaptive guidance network, and finally outputting the matching cost through the 3D convolution layer. The adaptive guidance network includes the following steps: converting the C dimension of the updated left-eye feature map into an N×C' dimension, and initializing a C-dimensional query vector, mapping the query vector into an N×C' dimension, i.e., 1×N×C'; calculating the similarity between the query vector and the updated left-eye feature map, and obtaining N similarities corresponding to each pixel in the updated left-eye feature map, i.e., N weights; then updating the query vector using the N similarities, so that the N groups of features of the query vector can represent a feature representation of an average grouping of all pixels of a specific image; and calculating the N similarities between the updated query vector and the left-eye feature map again to obtain an updated matching cost.

Images