A vehicle target detection method based on binocular camera and laser radar fusion

Abstract

The application discloses a vehicle target detection method based on binocular camera and laser radar fusion, and has the characteristics that the left eye image and the right eye image of a vehicle target at a predetermined moment of a binocular camera are acquired, and a disparity map is output after the left eye image and the right eye image are input into a stereo matching network; the pseudo point cloud image is input into an image semantic segmentation network, a vehicle is segmented out through the image semantic segmentation network first, then a pseudo point cloud containing only vehicle information is generated, an adaptive fusion factor delta is introduced, a parallax prediction epsilon credibility is combined, the influence of noise is reduced, and then the true and pseudo point clouds are fused in a data set; a grid vehicle target detection network is used to detect a vehicle target, and the position, the category and the orientation of a three-dimensional target frame of the vehicle are obtained. The method improves the accuracy of depth estimation, the effectiveness of the pseudo point cloud and the accuracy of vehicle target detection.

Description

Technical Field

[0001] This disclosure belongs to the field of transportation technology, and specifically relates to a vehicle target detection method based on the fusion of binocular camera and lidar. Background Technology

[0002] In recent years, autonomous driving technology has made significant progress, and detecting and identifying objects on the road ahead is one of the key tasks of autonomous vehicles. LiDAR and cameras are the two main sensors supporting target detection. LiDAR can provide accurate three-dimensional distance information, hence its widespread use. However, the cost of LiDAR increases significantly with the number of laser beams, so the number of laser beams in vehicle-mounted LiDAR is usually relatively small. This smaller number of laser beams results in a sparser point cloud, which is detrimental to target detection. Cameras, as lower-cost sensors, are also widely used for object detection, but because two-dimensional image data lacks distance information, cameras typically cannot perform well in three-dimensional object detection tasks.

[0003] To better utilize images, the PSMNet network estimates disparity maps from binocular RGB images to generate depth images, thus imbuing 2D images with distance information, known as RGB-D images. However, due to the large errors in image-based depth estimation, RGB-D-based 3D object detection performance is poor. The significant gap in 3D object detection accuracy between cameras and LiDAR is not entirely due to differences in the quality of depth data from the two sensors, but largely due to the different data representation formats. Generating pseudo-LiDAR point clouds from RGB-D images and using existing 3D point cloud object detection methods significantly improves the accuracy of image-based 3D object detection. Using sparse LiDAR point clouds to correct the pseudo-point clouds further enhances their accuracy. These studies demonstrate that representing image data in 3D space for object detection is a better choice than directly performing object detection in 2D space. However, although using pseudo-point clouds improves the accuracy of previous image-based 3D object detection, the lower coordinate accuracy of pseudo-point clouds still results in a significant gap compared to LiDAR-based 3D object detection. Summary of the Invention

[0004] This disclosure is made in response to the aforementioned needs of the prior art. The technical problem to be solved by this disclosure is to provide a vehicle target detection method based on the fusion of binocular camera and lidar, which improves the accuracy of depth estimation, the effectiveness of pseudo point clouds, and the accuracy of vehicle target detection.

[0005] To address the aforementioned problems, the technical solutions provided in this disclosure include:

[0006] A vehicle target detection method based on the fusion of binocular camera and LiDAR is provided, characterized by the following steps: First, acquire the left and right eye images of the vehicle target at a predetermined time using the binocular camera; input the left and right eye images into a stereo matching network to output a disparity map; the calculation process of the stereo matching network includes:

[0007] S101, the left and right eye images captured by the binocular camera are input into the MobileNetV2 network. After feature extraction of the left and right eye images, left eye feature maps and right eye feature maps are output. The feature extraction formula is: Conv input =H×W×C, where Conv represents the convolution operation; input represents the input left and right eye images; H×W×C represents the left or right eye feature map; S102, input the left and right eye feature maps H×W×C into the cross-image aggregation module. For a pixel in the left eye feature map, find D pixels to the left of the pixel at the same coordinates as the pixel in the right eye feature map, and calculate the similarity between the pixel in the left eye feature map and these D pixels. The similarity calculation formula is: Cost(p,d)=∑M l (x,y,C l ′,p)M r (x+d,y,C r ′,p), where Cost(p,d) represents the similarity cost, i.e., the matching cost, which is the cost of the disparity d at the p-th pixel in the left eye feature map; M l M represents the feature map of the left eye. r Let S103 represent the right eye feature map, d = 1, 2, 3, ..., D, where D is empirically set to 192; x and y represent the coordinates of the p-th pixel; S104 inputs the matching cost obtained from the cross-image aggregation module in S102 into the cost aggregation network to obtain the updated matching cost; S105 retains the two values ​​with the highest similarity in dimension D for each pixel in the final matching cost, then normalizes these two pixels using the Softmax function, and performs soft aggregation on these two dimensions to obtain the initial disparity for each pixel, and then obtains the full-resolution disparity map through disparity subdivision; S106, in a binocular camera, according to the geometric imaging principle of stereo vision, the depth calculation process is as follows:

[0008]

[0009] Where f is the focal length; d is the parallax, i.e., the distance from point p. l and p r In X aThe difference along the axis; Z represents depth information. After obtaining the depth information, based on the camera's intrinsic parameters, the image coordinate system can be transformed into the world coordinate system, ultimately converting it into a pseudo-point cloud map. The transformation calculation process is as follows:

[0010]

[0011] Where x, y, z are the coordinates in the point cloud coordinate system; x′, y′ are the coordinates in the image coordinate system; and Z is the depth. The first step involves inputting the pseudo-point cloud image into an image semantic segmentation network. Vehicles are first segmented using this network, and then a pseudo-point cloud containing only vehicle information is generated. An adaptive fusion factor δ is introduced, combined with disparity prediction confidence ε, to reduce the impact of noise. Finally, the real and pseudo point clouds are fused in the dataset. The fusion calculation process is as follows:

[0012]

[0013] Where P represents the coordinates of the fused point cloud; Here are the actual point cloud coordinates, and r is the reflection intensity;

[0014] ε represents the pseudo-point cloud coordinates; ε represents the disparity prediction confidence; δ represents the adaptive fusion factor, δ = 1 when ε > 0.9, and δ = 0 when ε ≤ 0.9; The third step is to detect vehicle targets through a rasterized vehicle target detection network to obtain the position, category, and orientation of the vehicle's 3D target bounding box; S301, the grid encoder receives the fused point cloud and divides the fused point cloud into grid pillars of uniform size, each grid pillar having a cross-sectional size of l×l and extending infinitely in the z direction; the grid encoder encodes the points in each grid pillar and finally outputs a tensor of dimension [D, P, N], where D = 9, representing the dimension of each point being encoded, P = 12000, representing the number of grid pillars, and N = 100, representing the number of points in each grid pillar, wherein each point in the grid pillar is encoded as a vector with 9 dimensions [r, x, y, z, x c ,y c ,z c ,x p ,y p ], r is the reflection intensity, x, y, z are the coordinates of a predetermined point in the grid pillar, x c ,y c ,z c x is the distance from the point to the arithmetic geometric center of all points in the grid column. p ,y pS302, the ResNet-18 backbone receives the [D,P,N] tensor output by the grid encoder, performs further feature extraction based on the preliminary features extracted by the grid encoder, and then aggregates the features of the backbone using the neck; S303, the detection head receives the features output by the backbone and the neck, performs vehicle target detection, and outputs the position, category, and orientation of the vehicle target box. The position of the vehicle target box is defined by 7 parameters (x,y,z,w,l,h,θ), where x,y,z represent the coordinates of the center point of the vehicle target box, w,l,h represent the width, length, and height of the vehicle target box, and θ represents the deflection angle of the vehicle target box.

[0015] Preferably, the left-eye feature map and the right-eye feature map are multi-scale, with resolutions of 1 / 4, 1 / 8, 1 / 16, and 1 / 32 of the resolutions of the left-eye image and the right-eye image, respectively.

[0016] Preferably, when the left-eye feature map and the right-eye feature map H×W×C are input into the cross-image aggregation module, the left-eye feature map and the right-eye feature map are converted into multiple dimensions, that is, the original C dimension is converted into N×C' dimension.

[0017] Preferably, the cost aggregation network employs an hourglass network, which consists of multiple 3D convolutional layers. The intermediate network layers output matching costs at different scales, specifically 1 / 4, 1 / 8, 1 / 16, and 1 / 32 resolutions. The calculation process of the hourglass network includes: inputting the matching cost obtained from the cross-graph aggregation module into the 3D convolutional layer, then adding it to the matching cost output by the intermediate network layers to obtain a new matching cost. This new matching cost, along with the previously obtained updated left-eye feature map at the corresponding scale, is input into an adaptive guidance network. The final output matching cost is then processed through a 3D convolutional layer. The adaptive guidance network includes... The next step is to transform the C-dimensional dimension of the updated left-eye feature map into an N×C'-dimensional dimension, and initialize a C-dimensional query vector, mapping the query vector to an N×C'-dimensional dimension, i.e., 1×N×C'; calculate the similarity between the query vector and the updated left-eye feature map to obtain N similarities, i.e., N weights, for each pixel in the updated left-eye feature map; then update the query vector using the N similarities, so that the N sets of features of the query vector can represent an average grouped feature representation of all pixels in a specific image; calculate the N similarities between the updated query vector and the left-eye feature map again to obtain the updated matching cost.

[0018] Compared with existing technologies, this invention improves the accuracy of depth estimation, the effectiveness of pseudo-point clouds, and the accuracy of vehicle target detection. Attached Figure Description

[0019] To more clearly illustrate the technical solutions in the embodiments of this specification or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the embodiments of this specification. For those skilled in the art, other drawings can be obtained based on these drawings.

[0020] Figure 1 This is a schematic diagram of a binocular camera imaging principle provided by a specific embodiment of the present invention. Detailed Implementation

[0021] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0022] To facilitate understanding of the embodiments of the present invention, further explanations and descriptions will be provided below with reference to the accompanying drawings and specific embodiments. These embodiments do not constitute a limitation on the scope of protection of the present invention.

[0023] This application discloses a vehicle target detection method based on the fusion of binocular camera and lidar, the vehicle target method including the following steps:

[0024] The first step is to acquire the left and right eye images of the vehicle target from the binocular camera at a predetermined time. The left and right eye images are then input into a stereo matching network to output a disparity map.

[0025] The calculation process of the stereo matching network includes:

[0026] S101, the left and right eye images captured by the binocular camera are input into the MobileNetV2 network. After feature extraction from the left and right eye images, left eye feature maps and right eye feature maps are output. The formula for feature extraction is:

[0027] Conv input =H×W×C

[0028] Where Conv represents the convolution operation; input represents the input left and right eye images; H×W×C represents the left eye feature map or the right eye feature map.

[0029] The left and right eye feature maps are multi-scale, with resolutions of 1 / 4, 1 / 8, 1 / 16, and 1 / 32 of the resolutions of the left and right eye images, respectively.

[0030] S102, input the left eye feature map and the right eye feature map H×W×C into the cross-image aggregation module, and convert the left eye feature map and the right eye feature map into a multi-head dimension, that is, convert the original C dimension into N×C' dimension.

[0031] For a pixel in the left-eye feature map, find the matching point of the pixel in the right-eye feature map. That is, select D pixels to the left of the same coordinates as the pixel in the right-eye feature map, and calculate the similarity between the pixel in the left-eye feature map and these D pixels. The formula for calculating the similarity is:

[0032] Cost(p,d)=∑M l (x,y,C l ′,p)M r (x+d,y,C r ′,p)

[0033] Where Cost(p,d) represents the similarity cost, i.e., the matching cost, which is the cost of the disparity d at the p-th pixel in the left eye feature map; M l M represents the feature map of the left eye. r Let d represent the feature map of the right eye, where d = 1, 2, 3, ..., D. Based on experience, D is taken as 192; x and y represent the coordinates of the p-th pixel.

[0034] S103, the matching cost obtained by the cross-graph aggregation module in S102 is input into the cost aggregation network. In this specific embodiment, an hourglass network is preferably used. The hourglass network consists of multiple 3D convolutional layers. The intermediate network layers will output matching costs at different scales, namely 1 / 4, 1 / 8, 1 / 16 and 1 / 32 resolutions.

[0035] The calculation process of the hourglass network includes: inputting the matching cost obtained by the cross-graph aggregation module into the 3D convolutional layer, and then adding it to the matching cost output by the intermediate network layer to obtain a new matching cost. The new matching cost and the previously obtained updated left eye feature map of the corresponding scale are input into the adaptive guidance network, and the final output matching cost is then passed through the 3D convolutional layer.

[0036] The adaptive guidance network includes the following steps:

[0037] The updated left eye feature map is transformed from dimension C to dimension N×C', and a C-dimensional query vector is initialized. The query vector is then mapped to dimension N×C', i.e., 1×N×C'.

[0038] The similarity between the query vector and the updated left-eye feature map is calculated to obtain N similarities, i.e., N weights, for each pixel in the updated left-eye feature map. Then, the query vector is updated using the N similarities, so that the N sets of features of the query vector can represent an average grouped feature representation of all pixels in a specific image.

[0039] The updated matching cost is obtained by recalculating the N similarities between the updated query vector and the left eye feature map.

[0040] S104 retains the two values ​​with the highest similarity in the D dimension for each pixel in the final matching cost, then normalizes these two pixels using the Softmax function, and performs soft aggregation on these two dimensions to obtain the initial disparity for each pixel. Finally, disparity subdivision is used to obtain the full-resolution disparity map.

[0041] S105, in a binocular camera, the left imaging center O l and right imaging center O r The line connecting the two points is the baseline b; the optical axis of the left camera is Z. l And the right camera optical axis Z r Parallel, and the straight line Z l Z r X in the spatial coordinate system a The axes are perpendicular. A point P in three-dimensional space and the two imaging center points O. l O r And the mapping points of point P on the imaging planes of the left and right cameras, i.e., point p l and p r Located in the same plane, the intersection of the plane and the two imaging planes is the epipolar line, such as... Figure 1 As shown. Based on the geometric imaging principle of stereo vision, the depth calculation process is as follows:

[0042]

[0043] Where f is the focal length; d is the parallax, i.e., the distance from point p. l and p r In X a The difference along the axis; Z represents depth information.

[0044] After obtaining depth information, the image coordinate system can be transformed into the world coordinate system based on the camera's intrinsic parameters, ultimately resulting in a pseudo-point cloud map. The transformation calculation process is as follows:

[0045]

[0046] Where x, y, z are the coordinates in the point cloud coordinate system; x′, y′ are the coordinates in the image coordinate system; and Z is the depth. This refers to the camera's internal parameters.

[0047] The second step involves inputting the pseudo-point cloud image into an image semantic segmentation network. The network first segments the vehicles, then generates a pseudo-point cloud containing only vehicle information, eliminating unnecessary points and reducing computational load. To address the issues of tailing and noise in the pseudo-point cloud data, an adaptive fusion factor δ is introduced, combined with the disparity prediction confidence ε, to reduce the impact of noise. Points with a disparity prediction confidence greater than 0.9 are retained, while those with a disparity prediction confidence less than 0.9 are discarded, thus correcting the pseudo-point cloud and reducing noise. Finally, the true and pseudo point clouds are fused in the dataset. The fusion calculation process is as follows:

[0048]

[0049] Where P represents the coordinates of the fused point cloud; Here are the actual point cloud coordinates, and r is the reflection intensity; ε represents the pseudo-point cloud coordinates; ε represents the disparity prediction confidence; δ represents the adaptive fusion factor, where δ = 1 when ε > 0.9 and δ = 0 when ε ≤ 0.9.

[0050] The third step involves using a rasterized vehicle target detection network to detect vehicle targets and obtain the position, category, and orientation of the vehicle's 3D target bounding box.

[0051] The gridded vehicle target detection network includes,

[0052] S301, the grid encoder receives the fused point cloud and divides the fused point cloud into grid pillars of uniform size. Each grid pillar has a cross-sectional size of l×l and extends infinitely in the z direction.

[0053] The grid encoder encodes the points in each grid post, ultimately outputting a tensor of dimensions [D, P, N], where D = 9, representing the dimension of each encoded point, P = 12000, representing the number of grid posts, and N = 100, representing the number of points in each grid post. Each point in the grid post is encoded as a vector with 9 dimensions.

[0054] [r,x,y,z,x c ,y c ,z c ,x p ,y p ], r is the reflection intensity, x, y, z are the coordinates of a predetermined point in the grid pillar, x c ,y c ,z c x is the distance from the point to the arithmetic geometric center of all points in the grid column. p ,y pThis is the distance from the point to the center of the grid column.

[0055] S302, the backbone ResNet-18 receives the tensor of dimension [D,P,N] output by the mesh encoder, and performs further feature extraction based on the preliminary features extracted by the mesh encoder.

[0056] Then, features of the main trunk portion are aggregated using the neck region to expand the receptive field and fuse multi-scale context.

[0057] S303, the detection head receives the features output from the main body and neck, performs vehicle target detection, and outputs the position, category, and orientation of the vehicle target box. The position of the vehicle target box is defined by seven parameters (x, y, z, w, l, h, θ), where x, y, z represent the coordinates of the center point of the vehicle target box, w, l, h represent the width, length, and height of the vehicle target box, and θ represents the deflection angle of the vehicle target box.

[0058] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A vehicle target detection method based on the fusion of binocular camera and lidar, characterized in that, Includes the following steps: The first step is to acquire the left and right eye images of the vehicle target from the binocular cameras at a predetermined time. These images are then input into a stereo matching network, which outputs a disparity map. The calculation process of the stereo matching network includes: S101, the left and right eye images captured by the binocular camera are input into the MobileNetV2 network. After feature extraction of the left and right eye images, left eye feature maps and right eye feature maps are output. The formula for feature extraction is: Where Conv represents the convolution operation; input represents the input left and right eye images; H×W×C represents the left eye feature map or the right eye feature map; S102, input the left-eye feature map and the right-eye feature map H×W×C into the cross-image aggregation module. For a pixel in the left-eye feature map, find D pixels to the left of the same coordinates in the right-eye feature map as the pixel, and calculate the similarity between the pixel in the left-eye feature map and these D pixels. The similarity calculation formula is as follows: Where Cost(p,d) represents the similarity cost, i.e., the matching cost, which is the cost of the disparity d at the p-th pixel in the left eye feature map; This represents the feature map of the left eye. Let d represent the feature map of the right eye, where d = 1, 2, 3, ..., D, and D is 192; x and y represent the coordinates of the p-th pixel. This represents the number of channels in the left eye feature map. This represents the number of channels in the right eye feature map. S103, input the matching cost obtained from the cross-graph aggregation module in S102 into the cost aggregation network to obtain the updated matching cost; S104, retain each pixel in the final matching cost. The two values ​​with the highest similarity in each dimension, then through The function normalizes the two pixels and performs soft aggregation on the two dimensions to obtain the initial disparity of each pixel, and then obtains the full-resolution disparity map through disparity subdivision. S105, in a binocular camera, based on the geometric imaging principle of stereo vision, the depth calculation process is as follows: in, Focal length; Parallax, i.e., point and exist The difference along the axis; Z represents depth information. After obtaining depth information, the image coordinate system is transformed into the world coordinate system based on the camera's intrinsic parameters, and finally converted into a pseudo-point cloud map. The transformation calculation process is as follows: in, , , The coordinates are in the point cloud coordinate system; , These are coordinates in the image coordinate system; For depth; , For the camera's internal parameters; The second step involves inputting the pseudo-point cloud image into an image semantic segmentation network. First, the vehicle is segmented using this network. Then, a pseudo-point cloud containing only vehicle information is generated, and an adaptive fusion factor is introduced. Combined with disparity prediction confidence To reduce the impact of noise, the real and fake point clouds are then fused in the dataset. The fusion calculation process is as follows: in, The coordinates of the merged point cloud; For the actual point cloud coordinates, Reflection intensity; These are pseudo-point cloud coordinates; For disparity prediction confidence; For adaptive fusion factor, hour , hour ; The third step is to detect vehicle targets using a rasterized vehicle target detection network to obtain the position, category, and orientation of the vehicle's 3D target bounding box. S301, the grid encoder receives the fused point cloud and divides the fused point cloud into grid pillars of uniform size, each grid pillar having a cross-sectional size of [missing information]. And in Extending infinitely in direction; The grid encoder encodes the points in each grid pillar, ultimately outputting a dimension of... The tensor, in which , representing the dimension encoded for each point. , indicating the number of grid bars, , representing the number of points in each grid post, wherein each point in the grid post is encoded as a vector with 9 dimensions. , Reflection intensity, This represents the coordinates of a predetermined point within a grid column. This is the distance from the point to the arithmetic geometric center of all points in the grid column. This is the distance from the point to the center point of the grid column; S302, main section The dimension of the output of the grid encoder is received. The tensor is used to perform further feature extraction based on the preliminary features extracted by the mesh encoder, and then the features of the trunk are aggregated using the neck. S303, the detection head receives the features output from the main body and neck, performs vehicle target detection, and outputs the position, category, and orientation of the vehicle target bounding box. The position of the vehicle target bounding box is defined by seven parameters. ,in, This represents the coordinates of the center point of the vehicle's target bounding box. This represents the width, length, and height of the vehicle's target bounding box. This indicates the deflection angle of the vehicle's target box.

2. The vehicle target detection method based on the fusion of binocular camera and lidar as described in claim 1, characterized in that, The left and right eye feature maps are multi-scale, with resolutions of 1 / 4, 1 / 8, 1 / 16, and 1 / 32 of the resolutions of the left and right eye images, respectively.

3. The vehicle target detection method based on the fusion of binocular camera and lidar according to claim 1, characterized in that, When the left and right feature maps H×W×C are input into the cross-image aggregation module, the left and right feature maps are converted into multiple dimensions, that is, the original C dimension is converted into N×C' dimension.

4. The vehicle target detection method based on the fusion of binocular camera and lidar as described in claim 1, characterized in that, The cost aggregation network employs an hourglass network, which consists of multiple 3D convolutional layers. The intermediate network layers output matching costs at different scales: 1 / 4, 1 / 8, 1 / 16, and 1 / 32 resolution. The calculation process of the hourglass network includes: first, inputting the matching cost obtained by the cross-graph aggregation module into a 3D convolutional layer, and then adding it to the matching cost output by the intermediate network layer to obtain a new matching cost. The new matching cost, along with the previously obtained updated left eye feature map of the corresponding scale, is input into the adaptive guidance network. Finally, the output matching cost is then processed by a 3D convolutional layer. The adaptive guidance network includes the following steps: transforming the C-dimensional dimension of the updated left-eye feature map into an N×C'-dimensional dimension, and initializing a C-dimensional query vector, mapping the query vector to an N×C'-dimensional dimension, i.e., 1×N×C'; calculating the similarity between the query vector and the updated left-eye feature map, obtaining N similarities, i.e., N weights, corresponding to each pixel in the updated left-eye feature map; and then using the... The query vector is updated with a similarity, such that the query vector's similarity is... Group features can represent the feature representation of an average group of all pixels in a specific image; the updated query vector and the left eye feature map are then recalculated. Given a similarity, we obtain the updated matching cost.

Images