A method and system for target detection by fusing lidar and visual camera

Abstract

The application discloses a kind of laser radar and visual camera fusion target detection method and system, the target detection method includes: obtaining laser radar point cloud data expressed in space spherical coordinate system;Point cloud data under space spherical coordinate system is carried out view cone gridding processing;From the point cloud view cone grid obtained, the feature extraction of point cloud data is carried out, and image view cone grid feature is obtained;Image view cone grid feature is fused with the feature of visual image obtained by visual camera, and fusion image is obtained, and target detection is carried out by fusion image.The application can effectively solve the problem that the depth information of visual image is easy to be missing in the existing fusion detection method, the probability depth obtained by fitting visual image pixel points is low efficient and unreliable, and can effectively solve the problem that the spatial information in Z-axis direction is not fully utilized in the existing fusion detection method, resulting in feature loss, and is especially suitable for the field of autonomous driving.

Description

Technical Field

[0001] This invention relates to the technical field of target detection methods, and particularly to the technical field of target detection methods that fuse data from lidar and visual cameras. Background Technology

[0002] In the field of autonomous driving, 3D target detection based on different sensors, such as visual cameras or LiDAR, is crucial. Different sensors excel at acquiring different types of features. For example, visual cameras contain rich semantic information, but they struggle to accurately identify 3D information within space or perform 3D target detection by object category. Conversely, LiDAR can obtain point cloud data containing rich spatial location and geometric information, but it lacks semantic information, making it difficult to obtain accurate imaging results. Therefore, fusing information from these two complementary sensors is a more ideal target detection solution.

[0003] In the current field of LiDAR and visual camera fusion perception, the main technical approach is to utilize images obtained by the visual camera to provide rich semantic features, and then use point cloud data obtained by the LiDAR to provide localization and geometric information, such as the Bevfusion method. In this method, the visual end can use a deep learning network such as LSS (Lift, Splat, Shoot) to achieve discrete depth fitting of visual image information, scattering each feature pixel into a spatial discrete pseudo-point cloud with probability distribution attributes. Then, spatial BEV pooling is used to aggregate features within a fixed-size horizontal BEV grid, thus converting the visual image information into horizontal BEV grid features. Similarly, the point cloud can be converted into spatial grid features using a voxel gridding algorithm, and then the spatial grid features are reduced to horizontal BEV grid features through Z-axis flattening, thus also converting the point cloud information into horizontal BEV grid features. Stacking these two BEV grid features yields the fused BEV grid features. This fused feature is then input into a BEV feature network for learning, enabling 3D fusion detection of the target and obtaining the target's 3D bounding box and corresponding motion information.

[0004] However, the Bevfusion fusion method described above still has the following drawbacks:

[0005] The deep learning network used does not perform well in fitting the depth of pixels in visual images and cannot fully utilize the data characteristics of the visual camera.

[0006] Both BEV mesh feature transformation on visual images and BEV mesh feature transformation on LiDAR point clouds have poor utilization of spatial information in the Z-axis direction. They do not fully utilize the spatial geometric and semantic features of LiDAR and visual camera data, resulting in feature loss. Summary of the Invention

[0007] To address the shortcomings of existing technologies, the present invention aims to provide a novel target detection method and system that fuses point cloud data obtained by LiDAR with image data obtained by a visual camera. This detection method can effectively solve the problems of easy loss of depth information in visual images and inefficient and unreliable probabilistic depth obtained by fitting visual image pixels in existing fusion detection methods. It can also effectively solve the problem of feature loss caused by the failure to fully utilize spatial information in the Z-axis direction in existing fusion detection methods.

[0008] The technical solution of the present invention is as follows:

[0009] A target detection method that fuses lidar and a visual camera, comprising:

[0010] S2 obtains lidar point cloud data expressed in a spatial spherical coordinate system, i.e., point cloud data in a spatial spherical coordinate system;

[0011] S3 performs frustum meshing on the point cloud data in the spatial spherical coordinate system to obtain several point cloud frustum meshes and the position data of the point cloud data in the point cloud frustum meshes;

[0012] S4 extracts features from the point cloud data within each point cloud frustum grid based on the point cloud frustum grid and the position data of the point cloud data within the point cloud frustum grid, thereby obtaining its feature data, i.e., image frustum grid features.

[0013] S5 fuses the image frustum grid features with the features of the visual image obtained by the visual camera to obtain a fused image that integrates the lidar point cloud and visual image information.

[0014] S6 inputs the fused image into the detection model for target detection.

[0015] According to some preferred embodiments of the present invention, the view frustum meshing process includes:

[0016] In the spatial spherical coordinate system, S31 sets the azimuth angle as... Polar angle is The smallest cone unit

[0017] S32 according to the minimum cone unit The point cloud data in the spatial spherical coordinate system is divided into a grid to obtain a point cloud view frustum grid, and the position data of the point cloud within it is obtained by the following formula:

[0018]

[0019]

[0020] αindex-n =Round(α) index′-n )

[0021] ω index-n =Round(ω) index-n′ (3)

[0022] Where, α cn ω represents the azimuth angle of the nth point cloud data in the spatial spherical coordinate system. cn α represents the polar angle of the nth point cloud data in the spatial spherical coordinate system. index′-n ω index-n′ These represent the azimuth and polar angle transition variables of the point cloud data in the nth spatial spherical coordinate system during meshing processing, respectively.

[0023] α index-n ω index-n These represent the azimuth position ordinal number and polar position ordinal number of the point cloud data in the nth spatial spherical coordinate system after meshing, respectively; Round() represents the rounding function.

[0024] According to some preferred embodiments of the present invention, the target detection method further includes:

[0025] S1 converts the point cloud data obtained by the lidar in the three-dimensional Cartesian coordinate system to the three-dimensional Cartesian coordinate system of the vision camera, and obtains the point cloud data after the three-dimensional Cartesian coordinate system conversion.

[0026] S2 converts the point cloud data after the three-dimensional rectangular coordinate system transformation to the spatial spherical coordinate system to obtain the point cloud data in the spatial spherical coordinate system.

[0027] According to some preferred embodiments of the present invention, the transformation in S1 is achieved through the following transformation model:

[0028]

[0029] Among them, X L1 ... X Ln Y represents the x-coordinate of the point cloud data in the lidar coordinate system. L1 ..., Y Ln Z represents the ordinate of the point cloud data in the lidar coordinate system. L1 ..., Z Ln The vertical coordinate of the point cloud data in the lidar coordinate system; X C1 ... X Cn This represents the x-coordinate and y-coordinate of the point cloud data in the visual camera coordinate system after coordinate system transformation. C1 ..., Y Cn Z represents the ordinate of the point cloud data in the visual camera coordinate system after transformation. C1..., Z Cn This represents the vertical coordinate of the point cloud data in the visual camera coordinate system after transformation, where n represents the ordinal number of the point cloud data; R 3×3 T represents the rotation matrix determined by the extrinsic parameters of the visual camera and the LiDAR; 3×1 This represents the translation matrix determined by the extrinsic parameters of the visual camera and the lidar.

[0030] According to some preferred embodiments of the present invention, the transformation in S2 adopts the following transformation model:

[0031]

[0032] Where, r cn ω represents the radius of the nth point cloud data after transformation. cn α represents the polar angle after the nth point cloud data is transformed. cn X represents the azimuth angle after the nth point cloud data is transformed. Cn Y represents the x-coordinate of the nth point cloud data in the visual camera coordinate system. Cn Z represents the ordinate of the nth point cloud data in the visual camera coordinate system. Cn This represents the vertical coordinate of the nth point cloud data in the visual camera coordinate system.

[0033] According to some preferred embodiments of the present invention, the image frustum grid feature includes: the number n of point cloud data within the frustum grid, and the average distance of the point cloud data within the frustum grid. and the average reflection intensity of point cloud data within the frustum grid

[0034] According to some preferred embodiments of the present invention, the features of the visual image include the R, G, and B values ​​of the visual image.

[0035] According to some preferred embodiments of the present invention, obtaining the fused image includes:

[0036] Obtain the number n of point cloud data within the view frustum grid and the average distance of the point cloud data within the view frustum grid. and the average reflection intensity of point cloud data within the frustum grid The three-channel image view frustum grid features;

[0037] Obtain the features of a three-channel visual image, including the R, G, and B values ​​of the image;

[0038] The image frustum grid features of the three channels are superimposed with the features of the visual image of the three channels to obtain a fused image in the form of a six-channel pseudo-image.

[0039] According to some preferred embodiments of the present invention, the detection model includes a backbone network based on Voxnet and a detection head connected to the backbone network, wherein the detection head uses a single-lens detector.

[0040] The present invention further provides a detection system applying the above target detection method, which includes: a preprocessing module that completes steps S1-S5, and a target detection model module that stores the detection model and performs target detection based on the fused image obtained by the preprocessing module.

[0041] The present invention has the following beneficial effects:

[0042] (1) The present invention can determine the image pixel frustum where the point cloud is located in space by using the point cloud with accurate three-dimensional spatial coordinates. The AOV algorithm is used to realize the mapping between spatial points and image pixel frustum. On the basis of fully preserving the semantic information of the image, the point cloud information is used to supplement the depth, fill degree and reflectivity in the pixel frustum to obtain more comprehensive image information, which is particularly suitable for autonomous driving.

[0043] (2) The method of the present invention can accurately achieve pixel-level fusion of images and point clouds. Compared with the Bevfusion algorithm, which has a fusion accuracy of Bev mesh map within the entire detection space, the present invention has higher accuracy, more accurate fusion, and more full utilization of spatial information.

[0044] (3) This invention innovatively processes point cloud data into AOV (Angle of View) data, which can convert point cloud data represented by three-dimensional rectangular coordinates (x, y, z) into data represented by spatial spherical coordinates (α, ω, r). Then, through downsampling, the complex three-dimensional features are converted into three-channel two-dimensional features. While making full use of visual images and LiDAR point cloud data, the pixel-level fusion accuracy of the two is achieved.

[0045] (4) In addition to being applicable to point cloud data represented by spatial rectangular coordinates, the present invention can also directly process raw point cloud data information represented by horizontal rotation angle, vertical elevation angle, and distance. When raw point cloud data information is used directly, the coordinate system transformation time can be saved, that is, the data processing time inside the lidar.

[0046] (3) The core network architecture used in this invention for target detection is two-dimensional convolution, which is lightweight. Therefore, it has stronger real-time performance compared to other fusion target detection methods. Attached Figure Description

[0047] Figure 1 This is a schematic diagram illustrating the conversion of Cartesian coordinate system point cloud data into spatial spherical coordinate system data in a specific implementation method.

[0048] Figure 2 This is a schematic diagram illustrating the view frustum meshing process in a specific implementation.

[0049] Figure 3 This is a schematic diagram of the fused image obtained in a specific implementation method.

[0050] Figure 4 This is a schematic diagram of the structure of Voxnet used in a specific implementation. Detailed Implementation

[0051] The present invention will now be described in detail with reference to embodiments and accompanying drawings. However, it should be understood that the embodiments and drawings are for illustrative purposes only and do not constitute any limitation on the scope of protection of the present invention. All reasonable modifications and combinations included within the inventive spirit of the present invention fall within the scope of protection of the present invention.

[0052] In some specific embodiments, the target detection method fusion of lidar and visual camera of the present invention includes the following steps:

[0053] S1 converts the point cloud data obtained by the LiDAR in the three-dimensional Cartesian coordinate system to the three-dimensional Cartesian coordinate system of the vision camera, thus obtaining the point cloud data after the three-dimensional Cartesian coordinate system conversion.

[0054] In some specific implementations, the transformation described in S1 is achieved through the following transformation model:

[0055]

[0056] Among them, X L1 ... X Ln Y represents the x-coordinate of the point cloud data in the lidar coordinate system. L1 ..., Y Ln Z represents the ordinate of the point cloud data in the lidar coordinate system. L1 ..., Z Ln The vertical coordinate of the point cloud data in the lidar coordinate system; X C1 ... X Cn This represents the x-coordinate and y-coordinate of the point cloud data in the visual camera coordinate system after coordinate system transformation. C1 ..., Y Cn Z represents the ordinate of the point cloud data in the visual camera coordinate system after transformation. C1 ..., Z Cn This represents the vertical coordinate of the point cloud data in the visual camera coordinate system after transformation, where n represents the ordinal number of the point cloud data; R 3×3 T represents the rotation matrix determined by the extrinsic parameters of the visual camera and the LiDAR; 3×1 This represents the translation matrix determined by the extrinsic parameters of the visual camera and the lidar.

[0057] The above process can effectively convert the point cloud data collected by LiDAR to the coordinate system of the visual camera, providing a foundation for multi-sensor fusion.

[0058] S2 converts the point cloud data obtained from the three-dimensional rectangular coordinate system to the spatial spherical coordinate system, thus obtaining the point cloud data in the spatial spherical coordinate system.

[0059] In some specific implementations, refer to the appendix. Figure 1 The transformation described in S2 adopts the following transformation model:

[0060]

[0061] Where, r cn ω represents the radius of the nth point cloud data after transformation (i.e., R in the diagram). cn α represents the polar angle (i.e., ω in the diagram) after the nth point cloud data has been transformed. cn This represents the azimuth angle after the nth point cloud data is transformed (i.e., α in the diagram).

[0062] In the above process, both Cartesian coordinates and spherical coordinates are common spatial representations, each with its own advantages in different fields. Whether it's a visual camera or a LiDAR, the essence of their operation is to utilize light rays hitting a target and analyze the scattering and reflection phenomena on the target surface. Therefore, the correspondence along the ray direction is an excellent approach for high-precision fusion of visual cameras and LiDAR. The two angles of the spatial spherical coordinate system precisely correspond to the accurate parameters of its ray direction. Therefore, performing the transformation between these two coordinate systems to achieve further data analysis and visualization is a crucial step in this invention.

[0063] S3 performs Angle of View (AOV) meshing on the obtained point cloud data in the spatial spherical coordinate system to obtain several point cloud AOV meshes and the position data of the point cloud data in the point cloud AOV meshes.

[0064] Among them, refer to the appendix Figure 2 The view frustum meshing process includes:

[0065] In the spatial spherical coordinate system, S31 sets the azimuth angle as... Polar angle is The smallest cone unit

[0066] S32 is based on the smallest cone unit The point cloud data in the spatial spherical coordinate system is divided into a grid to obtain a point cloud view frustum grid, and the position data of the point cloud in the grid is obtained by the following formula (3):

[0067]

[0068] α index-n =Round(α) index′-n )

[0069] ω index-n =Round(ω) index-n′ (3)

[0070] Where, α index′-n ω index-n′ α represents the azimuth and polar angle transition variables of the point cloud data in the nth spatial spherical coordinate system during meshing. index-n ω index-n These represent the azimuth position ordinal number and polar position ordinal number of the point cloud data in the nth spatial spherical coordinate system after meshing, respectively; Round() represents the rounding function.

[0071] The above process considers the pixels within the image as having horizontal and vertical viewing angles, respectively. The view frustum grid, with The smallest view frustum unit is meshed, and then the point cloud, converted to a spatial spherical coordinate system (α, ω, r), is used to find the view frustum mesh in which it is located based on the values ​​of its azimuth angle α and polar angle ω.

[0072] S4 extracts features from the point cloud data within each point cloud frustum grid to obtain its feature data, namely the image frustum grid features.

[0073] In some preferred embodiments, considering that the point cloud data of the lidar contains not only coordinate values ​​but also reflection intensity, and that due to the low penetration of the lidar, the distance difference between the point cloud within the frustum grid and the lidar itself is not large (i.e., the distance between points within the grid is relatively stable), the present invention further obtains the number n of point cloud data in each frustum grid by sorting and counting the point cloud data in each frustum grid, which is used as the fill degree of the frustum grid. The average distance of the point cloud data in each frustum grid is obtained by averaging the distances of the point cloud data within the frustum grid. The average reflection intensity of the point cloud data within each frustum grid is obtained by averaging the reflection intensity of the point cloud data within that frustum grid. (n, The pseudo-pixels of the LiDAR stream are formed in the form of (3*H1*W1) to form the three-channel feature data of the point cloud within the point cloud frustum grid, which is the image frustum grid feature. Here, H1 represents the minimum vertical resolution of the pseudo-pixels of the LiDAR stream, and W1 represents the minimum horizontal resolution of the pseudo-pixels of the LiDAR stream.

[0074] S5 fuses the image frustum grid features with the features of the visual image obtained through the visual camera to obtain a fused image that combines the information from the LiDAR point cloud and the visual image.

[0075] In some preferred embodiments, the obtained image frustum grid features are three-channel feature data in the form of (3*H1*W1), which are directly superimposed with the three-channel features of the visual image. The features of the visual image include its R, G, and B values, forming three-channel feature data in the form of (3*H*W). After superposition, the fused image is obtained, as shown in the attached figure. Figure 3 As shown, H represents the minimum vertical resolution of a pixel, and W represents the minimum horizontal resolution of a pixel.

[0076] The fused image obtained by the above implementation method is a six-channel pseudo-image, which can be further fed into a graph convolutional neural network for learning, fitting, and testing to achieve object detection.

[0077] S6 inputs the fused image into the detection model for target detection.

[0078] In some preferred embodiments, the detection model used in this invention includes:

[0079] A backbone network based on Voxnet and a detection head connected to the backbone network, wherein the detection head uses a Single Shot Detector (SSD).

[0080] The Voxnet described herein refers to the literature “Voxelnet: End-to-end learning for point cloud-based 3D object detection” (In CVPR, 2018. Y. Zhou and O. Tuzel.) or “Pointpillars: Fast encoders for object detection from point clouds” (In Proceedings of the IEEE / CVF conference on computer vision and pattern recognition, pages 12697–12705, 2019. Alex H Lang, Sourabh Vora, Holger Caesar, Lubing Zhou, Jiong Yang, and Oscar Beijbom.), and its specific structure is shown in the appendix. Figure 4As shown, the network has three blocks of fully convolutional layers. The first layer of each block downsamples the feature map by half through a 2D convolution with a stride of 2, followed by a series of 2D convolutions with a stride of 1. After each convolutional layer, normalization (BN) and pooling (ReLU) operations are applied. Through these operations, the present invention can upsample the output of each block to a fixed size and stitch them together to construct a high-resolution feature map.

[0081] Referring to the literature "SSD: Single shot multibox detector" (In ECCV, 2016, 2, 4. W. Liu, D. Anguelov, D. Erhan, C. Szegedy, S. Reed, C.-Y. Fu, and ACBerg.), this invention uses a single-lens detector to detect three-dimensional objects. It can randomly generate two-dimensional target boxes according to the channel size, refine them within the boxes, and map the feature map to the desired learning target, including the target's category, the target's three-dimensional bounding box information, and the target's motion attributes, thereby achieving the function of three-dimensional target detection.

[0082] The above embodiments are merely preferred embodiments of the present invention, and the scope of protection of the present invention is not limited to the above embodiments. All technical solutions falling within the scope of the present invention's concept are within the scope of protection of the present invention. It should be noted that for those skilled in the art, improvements and modifications made without departing from the principles of the present invention should also be considered within the scope of protection of the present invention.

Claims

1. A target detection method that fuses lidar and a visual camera, characterized in that, It includes: S2 obtains lidar point cloud data expressed in a spatial spherical coordinate system, i.e., point cloud data in a spatial spherical coordinate system; S3 performs frustum meshing on the point cloud data in the spatial spherical coordinate system to obtain several point cloud frustum meshes and the position data of the point cloud data in the point cloud frustum meshes; S4, based on the point cloud view frustum grid and the position data of the point cloud data within the point cloud view frustum grid, performs feature extraction on the point cloud data within each point cloud view frustum grid to obtain its feature data, i.e., image view frustum grid features; the image view frustum grid features include: the number n of point cloud data within the view frustum grid, and the average distance of the point cloud data within the view frustum grid. and the average reflection intensity of point cloud data within the frustum grid ; S5. The image frustum grid features are fused with the features of the visual image obtained by the visual camera to obtain a fused image that combines the lidar point cloud and visual image information. S6 Input the fused image into the detection model for target detection; The view frustum meshing process includes: S31 In the spatial spherical coordinate system, the azimuth angle is set as... Polar angle is The smallest cone unit ; S32 According to the minimum cone unit The point cloud data in the spatial spherical coordinate system is divided into a grid to obtain a point cloud view frustum grid, and the position data of the point cloud within it is obtained by the following formula: （3） in, This represents the azimuth angle of the nth point cloud data in the spatial spherical coordinate system. This represents the polar angle of the nth point cloud data in the spherical coordinate system. These represent the azimuth and polar transition variables of the point cloud data in the nth spatial spherical coordinate system during meshing processing, respectively. These represent the azimuth position ordinal number and polar position ordinal number of the point cloud data in the nth spatial spherical coordinate system after meshing, respectively; This represents the rounding function.

2. The target detection method according to claim 1, characterized in that, It also includes: S1 converts the point cloud data obtained by the lidar in the three-dimensional Cartesian coordinate system to the three-dimensional Cartesian coordinate system of the vision camera, and obtains the point cloud data after the three-dimensional Cartesian coordinate system conversion. S2 converts the point cloud data after the three-dimensional rectangular coordinate system transformation to the spatial spherical coordinate system to obtain the point cloud data in the spatial spherical coordinate system.

3. The target detection method according to claim 2, characterized in that, The transformation described in S1 is achieved through the following transformation model: （1） Among them, X L1 ... X Ln Y represents the x-coordinate of the point cloud data in the lidar coordinate system. L1 ..., Y Ln Z represents the ordinate of the point cloud data in the lidar coordinate system. L1 ..., Z Ln The vertical coordinate of the point cloud data in the lidar coordinate system; X C1 ... X Cn This represents the x-coordinate and y-coordinate of the point cloud data in the visual camera coordinate system after coordinate system transformation. C1 ..., Y Cn Z represents the ordinate of the point cloud data in the visual camera coordinate system after transformation. C1 ..., Z Cn This represents the vertical coordinate of the point cloud data in the visual camera coordinate system after transformation, where n represents the ordinal number of the point cloud data; This represents the rotation matrix determined by the extrinsic parameters of the visual camera and the LiDAR. This represents the translation matrix determined by the extrinsic parameters of the visual camera and the lidar.

4. The target detection method according to claim 2, characterized in that, The transformation described in S2 adopts the following transformation model: （2） in, This represents the radius of the nth point cloud data after transformation. This represents the polar angle after the nth point cloud data has been transformed. X represents the azimuth angle after the nth point cloud data is transformed. Cn Y represents the x-coordinate of the nth point cloud data in the visual camera coordinate system. Cn Z represents the ordinate of the nth point cloud data in the visual camera coordinate system. Cn This represents the vertical coordinate of the nth point cloud data in the visual camera coordinate system.

5. The target detection method according to claim 1, characterized in that, The features of the visual image include the R, G, and B values ​​of the visual image.

6. The target detection method according to claim 1, characterized in that, The fused image is obtained by: Obtain the number n of point cloud data within the view frustum grid and the average distance of the point cloud data within the view frustum grid. and the average reflection intensity of point cloud data within the frustum grid The three-channel image frustum grid features; Obtain the features of a three-channel visual image, including the R, G, and B values ​​of the image; The image frustum grid features of the three channels are superimposed with the features of the visual image of the three channels to obtain a fused image in the form of a six-channel pseudo-image.

7. The target detection method according to claim 1, characterized in that, The detection model includes a backbone network based on Voxnet and a detection head connected to the backbone network. The detection head uses a single-lens detector.

8. A detection system employing the target detection method according to any one of claims 1-7, comprising: The preprocessing module completes steps S1-S5, and the target detection model module stores the detection model and performs target detection based on the fused image obtained from the preprocessing module.

Images