Vision-laser radar fusion method and system based on depth canonical correlation analysis

Abstract

The invention relates to the technical field of automatic driving, in particular to a vision-laser radar fusion method and system based on deep canonical correlation analysis. The method comprises the following steps: synchronously acquiring an RGB image and point cloud data of a road surface; performing feature extraction on the RGB image to obtain RGB features; sequentially performing coordinate system conversion and rasterization processing on the point cloud data, and then performing feature extraction to obtain point cloud features; inputting the point cloud features and the RGB features into a pre-established and trained fusion model at the same time, outputting feature-enhanced fusion point cloud features, and fusing, by the fusion model, the RGB features into the point cloud features by using correlation analysis in combination with a deep neural network; and inputting the fused point cloud features into a pre-established target detection network to realize target detection. The depth canonical correlation analysis method is creatively used, and the similarity calculation matrix is utilized to fuse two different modal features of the point cloud and the image, so that the precision is improved under the condition of necessary speed compromise.

Description

technical field [0001] The invention relates to the technical field of automatic driving, in particular to a vision-lidar fusion method and system based on deep canonical correlation analysis. Background technique [0002] In recent years, the field of autonomous driving technology has developed rapidly, in which environmental perception technology is an indispensable part. Self-driving vehicles rely on sensors such as lidar, camera, and millimeter-wave radar to perceive the surrounding environment, and collect and process environmental information and in-vehicle information, mainly involving technologies such as road boundary monitoring, vehicle detection, and pedestrian detection. Environmental perception can feed back real-time road information to the car, so as to provide more precise control of the driving behavior of the vehicle, thereby improving driving safety and comfort. [0003] As one of the basic functions of the automatic driving system, 3D object detection ca...

AI Technical Summary

Problems solved by technology

However, both sensors have certain drawbacks

For cameras, RGB images, as a dense 2D representation, express spatial information through pixel positions and pixel values, and compress the depth information of the Z axis, which makes it difficult to predict the absolute position of objects in space in 3D target detection tasks. ; In addition, the camera lens is also very sensitive to changes in light, often overexposed or too dark, the data collected in this scene is difficult to identify, missing effective information

For lidar, although the precise spatial position information of the target can be obtained, as an active sensor, lidar can only accept the laser signal emitted by the sensor, and cannot accept the signal reflected by the object through visible light, so the color of the object cannot be obtained. Texture and other feature information; moreover, the wavelength of the laser is very short, which makes the lidar data very sensitive to special weather. For example, the lidar point cloud obtained in rainy and snowy weather usually has a lot of noise

[0006] To sum up, if images are used as the main input information, the detection system will suffer serious failures when image information is missing due to changes in lighting conditions. In addition, the lack of spatial distance information in images also causes shortcomings in detection.

Although the lidar point cloud can overcome the problem of illumination changes, the sparseness and disorder of the point cloud also limit its detection ability. It is likely that small objects beyond 100 meters cannot be detected, which is not conducive to the speed of the car at high speed. perception

Therefore, in order to solve the problem of insufficient detection ability when using only on-board cameras or lidar, a novel idea is to consider combining more sensors and perform data fusion to improve the ability to perceive the environment

However, a large number of experiments at this stage have shown that if the pixel-level fusion of the point cloud information obtained by the lidar and the RGB image obtained by the camera is simply performed, not only will it not improve the accuracy of the perception algorithm, but it will have a negative effect. Reduces perception accuracy to a certain extent

Images