Night scene fusion different-source image-oriented depth estimation method and device

Abstract

The invention relates to a nighttime scene-oriented depth estimation method and a nighttime scene-oriented depth estimation device for fusing different-source images. The nighttime scene-oriented depth estimation device comprises a different-source image acquisition module, a depth estimation module for fusing the different-source images and a posture estimation module. And the different-source image acquisition module is used for synchronously acquiring different-source images in real time. In the depth estimation module for fusing the different-source images, an RGB image feature extraction unit and an infrared image feature extraction unit respectively realize extraction of depth-related features in an RGB image and an infrared image; and the different-source feature fusion unit realizes real-time depth estimation of the scene by fusing the two features. And the camera attitude obtained by the attitude estimation module is used for constraining a depth estimation result in the depth estimation module, so that the depth estimation result meets distortion transformation in epipolar geometry. According to the method, real-time depth estimation for the night scene is established, real-time perception of the surrounding environment in the night scene is realized by synchronously acquiring the RGB image and the infrared image and fusing the corresponding depth features, and the method has relatively high flexibility and practicability.

Description

Technical Field

[0001] The present invention belongs to the field of image processing, and specifically relates to a real-time depth estimation method and device for nighttime scenes, which is applied to real-time depth estimation of targets including obstacles, pedestrians, etc. in nighttime low-light scenes of autonomous driving, to assist in intelligent decision-making of subsequent autonomous driving. Background Art

[0002] Real-time perception is an important direction in the field of autonomous driving, especially the real-time depth estimation of various targets on the road under low-light conditions at night, which can effectively ensure the safety of autonomous driving. At the same time, real-time depth estimation of night scenes can also optimize night routes in real time and improve the intelligent decision-making of the autonomous driving system under low light. Therefore, depth estimation for night scenes is of great significance to the safety of the entire autonomous driving system.

[0003] Common methods for depth estimation of night scenes mainly include: the first is to directly use RGB images under low light as input and use the self-supervised monocular depth estimation method to perform depth estimation. The disadvantage of this method is that the RGB images in night scenes lack a lot of detail information, and there are some image blurs, resulting in low accuracy of the estimated depth value; another method is to directly use the image collected by the infrared camera as input and use the self-supervised monocular depth estimation model to perform depth estimation. Although the infrared camera can avoid the loss of details caused by low light, the image quality itself is poor, mainly manifested in low contrast, blurred visual effects, low signal-to-noise ratio, and unclear local features. These shortcomings make the depth estimation accuracy of this method insufficient; the third method is to use laser radar and other equipment for depth estimation. The disadvantages of this type of method are firstly that the cost is too high, and secondly that the depth information obtained is sparse, that is, not pixel by pixel. In addition, this type of method has strict restrictions on the measurement distance. In addition, there is a depth estimation method based on binocular stereo vision. This method not only has poor real-time performance due to high computational complexity, but also has the disadvantages of the method, that is, it is limited by the loss of scene information caused by low light. Summary of the invention

[0004] In order to solve the above-mentioned deficiencies in the prior art, the present invention proposes a depth estimation method and device for nighttime scenes by fusing heterogeneous images, which is used to solve the problem of insufficient accuracy of depth maps obtained by only using monocular depth estimation based on RGB images and only using monocular depth estimation based on infrared images in the existing depth estimation methods for nighttime scenes. At the same time, it also solves the problems of high cost and insufficient density of depth information of sensors such as lidar, and overcomes the shortcomings of poor real-time performance and insufficient accuracy of existing depth estimation of nighttime scenes based on binocular stereo vision.

[0005] The technical means adopted by the present invention are as follows:

[0006] A depth estimation device for night scene fusion heterogeneous images is provided, and the following modules are set to establish a network model offline and iteratively train to obtain an optimized device, so that it can be used to perform depth estimation on heterogeneous images of night scenes. The device includes:

[0007] The heterogeneous image acquisition module controls heterogeneous cameras to synchronously acquire heterogeneous night scene images and establish training set data;

[0008] The depth estimation module for fusing heterogeneous images includes an RGB image feature extraction unit, an infrared image feature extraction unit, and a heterogeneous feature fusion unit, which are used to extract and fuse heterogeneous image features for depth estimation; at the same time, the relative posture estimation T output by the posture estimation module is combined t→t' Reconstruct the night target scene and train it through the corresponding photometric loss to achieve night scene depth estimation based on heterogeneous images;

[0009] The posture estimation module includes an RGB camera posture estimation unit, an infrared camera posture estimation unit, and a posture consistency constraint unit, which is used to estimate the camera posture of heterogeneous cameras and strengthen the convergence through consistency constraints; finally, the relative posture estimation result T is combined with the epipolar geometry t→t' Combined with the estimation results of the depth estimation module, the simultaneous training of the pose network and the depth estimation network can be achieved.

[0010] The heterogeneous camera adopts an RGB camera and an infrared camera to collect real-time monocular images in night scenes; the training set data is a heterogeneous image pair of a current frame and previous and next adjacent frames.

[0011] The heterogeneous image acquisition module also includes an acquisition controller for synchronously triggering heterogeneous cameras to synchronously acquire images; the RGB camera and the infrared camera are on the same horizontal line and are close to each other, so as to shoot images from the same viewing angle.

[0012] The depth estimated by the depth estimation module for fusing heterogeneous images is based on the scene in the RGB camera to obtain the pixel-by-pixel depth value in the RGB image.

[0013] The RGB image feature extraction unit and the infrared image feature extraction unit both adopt a multi-scale deep feature extraction network based on the U-Net structure, and their internal encoders both adopt a network structure based on MobileNetV3; the network input is the corresponding synchronized image, and the network output is the corresponding depth-related feature map.

[0014] The extraction of infrared image features also includes mask processing, which is used to mask the identification area that only appears in the infrared image, and retain the identification area that simultaneously appears in the heterogeneous image as a mask template area.

[0015] The mask shield sets the corresponding position on the feature map obtained by the infrared image feature extraction unit to 0; the mask template area is obtained through neural network training.

[0016] The RGB camera attitude estimation unit and the infrared camera attitude estimation unit both adopt CNN networks to estimate the camera attitude between adjacent RGB image frames and the camera attitude between adjacent infrared image frames, respectively, and constrain them through the attitude consistency constraint unit so that the camera attitude estimation results of the two heterogeneous cameras should be consistent.

[0017] The camera pose and depth information estimated by combining the epipolar geometry constraint satisfy the distortion transformation in the epipolar geometry, and the relative pose estimate T t→t' Includes camera offsets in three directions and three angle changes.

[0018] The method of reconstructing a night scene and finally outputting a dense depth map includes:

[0019] Combined with locally differentiable bilinear interpolation, we get the source image I t' Reconstructed target reconstructed image I t'→t ;

[0020] The reconstructed pixel-by-pixel photometric loss is established for iterative training to optimize the corresponding network model parameters of the depth estimation module and the posture estimation module that fused heterogeneous images.

[0021] A depth estimation method for night scene fusion heterogeneous images is used to perform the following steps in night driving scenes to perform night scene depth estimation in real time:

[0022] 1) The heterogeneous image acquisition module controls two heterogeneous cameras in real time to perform synchronous image acquisition;

[0023] 2) In the depth estimation module that fuses heterogeneous images, the RGB image feature extraction unit and the infrared image feature extraction unit receive the RGB real-time image and the infrared real-time image respectively and perform corresponding feature extraction. The heterogeneous feature fusion unit will fuse the above two features and finally output an estimated dense depth map.

[0024] The beneficial effects of the present invention are:

[0025] 1. The present invention establishes the depth perception of the autonomous driving system in night scenes, which can solve the problem of insufficient accuracy when using a single source image for depth estimation under low light to a certain extent. At the same time, it overcomes the shortcomings of using sensors such as lasers, such as high cost and sparse estimation. It can improve the intelligent decision-making of the autonomous driving system in low light and ensure the safety of autonomous driving in night scenes, and has strong practicality.

[0026] 2. The present invention provides a method and device for depth estimation based on fusion of heterogeneous images for night scenes. Through a heterogeneous image acquisition module, a depth estimation module for fusion of heterogeneous images, and a posture estimation module, a set of depth estimation methods and devices based on fusion of heterogeneous features are constructed, which can estimate the depth information of night scenes in real time. At the same time, combined with the corresponding scene depth information, it assists the intelligent decision-making of the subsequent autonomous driving system. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 It is an overall schematic diagram of a depth estimation device for fusing heterogeneous images for nighttime scenes according to the present invention;

[0028] Figure 2 It is a schematic diagram of the flow of the depth estimation module for fusing heterogeneous images of the present invention;

[0029] Figure 3 It is a flow chart of the posture estimation module of the present invention. DETAILED DESCRIPTION

[0030] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the specific implementation method of the present invention is described in detail below in conjunction with the accompanying drawings. In the following description, many specific details are set forth to facilitate a full understanding of the present invention. However, the present invention can be implemented in many other ways different from those described herein, and those skilled in the art can make similar improvements without violating the connotation of the invention, so the present invention is not limited by the specific implementation disclosed below.

[0031] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art of the present invention. The terms used in the specification of the invention herein are only for the purpose of describing specific embodiments and are not intended to limit the present invention.

[0032] The present invention establishes a real-time depth estimation method for night scene fusion of heterogeneous images, which is used for the automatic driving system to perceive the real-time depth information of the surrounding environment in low-light night scenes, including obstacles, pedestrians and other targets in the night low-light scenes of automatic driving. The present invention can effectively improve the accuracy of real-time depth estimation, thereby better assisting the subsequent corresponding decision-making tasks and improving the security of the automatic driving system. Therefore, the present invention has strong practicality.

[0033] A depth estimation device for fusing heterogeneous images for night scenes, such as Figure 1As shown, it includes: a heterogeneous image acquisition module, a depth estimation module for fusing heterogeneous images, and a posture estimation module; the heterogeneous image acquisition module controls two heterogeneous cameras to perform synchronous acquisition through an acquisition controller, and inputs the acquired heterogeneous images into the depth estimation module for fusing heterogeneous images and the posture estimation module, the depth estimation module will output real-time depth information, and the posture estimation module will output corresponding camera posture information according to the input heterogeneous image sequence during the training process. In addition, during the training process, the posture estimation module and the depth estimation module are mutually constrained by the distortion transformation in the epipolar geometry to achieve simultaneous training.

[0034] like Figure 2 The figure shows the process flow of the depth estimation module for fusion of heterogeneous images. The collected real-time heterogeneous images will be input into the corresponding feature extraction units respectively. Since the dense depth map outputted in the end corresponds to the RGB real-time image pixel by pixel, and there is a certain offset between the RGB real-time image and the infrared real-time image, it is necessary to extract the features obtained by the infrared image feature extraction unit. Figure 2 The features are obtained by performing dot product with the binary mask. Figure 2 Further processing is performed to reduce the impact of noise areas such as occlusion. The binary mask here is also obtained through training. Features Figure 2 After processing and features Figure 1 Channel cascading is performed, and then feature fusion is achieved through a series of convolutions, and finally real-time depth information is output.

[0035] like Figure 3 The figure shows a flow chart of the attitude estimation module, which is only used in the training process. During the training process, the module inputs the corresponding heterogeneous image sequence and simultaneously realizes the attitude estimation of two heterogeneous cameras through the corresponding camera attitude estimation unit. Since the two cameras are fixed on the same rigid body bracket and are close to each other, they can be constrained by the attitude consistency unit, that is, the attitude results of the two cameras should be consistent. The module finally outputs the camera attitude estimation result. During the training process, the camera attitude estimation result and the depth estimation result will be constrained by the distortion transformation in the epipolar geometry, so as to realize the mutual assistance of the attitude estimation module and the depth estimation module in the training process.

[0036] When performing real-time depth estimation for the depth estimation device for fusing heterogeneous images for nighttime scenes of the present invention, the specific steps are as follows:

[0037] S1 When the depth estimation device receives the instruction to collect training images, the acquisition controller will control the two heterogeneous cameras to collect synchronously and save the obtained heterogeneous image sequence;

[0038] S2 preprocesses the collected image sequence to produce the corresponding training set. Each subsequence in the training set contains an RGB image sequence and a synchronized infrared image sequence, and both sequences contain three frames of images. Here, the middle frame is named the target image, and the previous and next frames are named the source image. Since the training process adopts a self-supervised method, no real depth information is required for annotation;

[0039] During the S3 training process, the heterogeneous target image in the subsequence is input into the depth estimation module that fuses the heterogeneous images to obtain the depth map corresponding to the RGB target image; the complete subsequence is input into the posture estimation module, and after passing through two heterogeneous camera posture estimation units and the posture consistency constraint unit, the corresponding camera posture estimation result is obtained.

[0040] During S4 training, it is necessary to combine the distortion transformation to establish the relationship between the original RGB target image and the source image. t and p t' Represent the original target image I t and source image I t' The homogeneous coordinates of the pixels in . Therefore, the following formula can be established based on epipolar geometry:

[0041] p t' ~KT t→t' D t (p t )K -1 p t

[0042] Among them, K represents the camera internal parameter, D t represents the depth estimation result obtained by the depth estimation module, T t→t' is the relative posture estimation result output by the posture estimation module. According to the above formula, combined with the local differentiable bilinear interpolation, we can get t' Reconstructed target image I t'→t , training is performed by establishing pixel-by-pixel photometric loss after reconstruction. The training is terminated after the loss function value converges, and finally the corresponding network model parameters of the depth estimation module and the posture estimation module of the fusion heterogeneous image are obtained;

[0043] When S5 is running in the test phase, when receiving the perception signal, the acquisition controller will synchronously collect the real-time RGB image and real-time infrared image of the night scene, and transmit them to the trained depth estimation module that fuses heterogeneous images;

[0044] S6 In the depth estimation module, the RGB image feature extraction unit and the infrared image feature extraction unit receive the RGB real-time image and the infrared real-time image respectively, and perform corresponding feature extraction to obtain feature Figure 1 and Features Figure 2 ;

[0045] After S7 completes feature extraction, the feature Figure 2 First, do a dot product operation with the trained binary mask, and then Figure 1 After cascading in the channel direction and going through multiple layers of convolution operations, heterogeneous feature fusion is finally achieved, and the corresponding real-time depth information is output.

[0046] The present invention describes a depth estimation method and device for fusing heterogeneous images for nighttime scenes. Through a heterogeneous image acquisition module, a depth estimation module for fusing heterogeneous images, and a posture estimation module, a set of depth estimation methods and devices based on heterogeneous monocular vision are constructed. The method and device can estimate the dense depth information in nighttime scenes in real time, thereby better assisting the corresponding decision-making tasks of the subsequent autonomous driving system and improving the safety of the autonomous driving system.

[0047] The above is a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present invention. These improvements and modifications should be regarded as within the scope of protection of the present invention.

Claims

1. A depth estimation device for fusing heterogeneous images for nighttime scenes, characterized in that: The following modules are set up to establish a network model offline and iteratively train it, and obtain an optimized device for depth estimation of heterogeneous images of night scenes. The device includes: The heterogeneous image acquisition module controls heterogeneous cameras to synchronously acquire heterogeneous night scene images and establish training set data; The depth estimation module for fusing heterogeneous images includes an RGB image feature extraction unit, an infrared image feature extraction unit, and a heterogeneous feature fusion unit, which are used to extract and fuse heterogeneous image features for depth estimation; at the same time, the relative posture estimation T output by the posture estimation module is combined t→t' Reconstruct the night target scene and train it through the corresponding photometric loss to achieve night scene depth estimation based on heterogeneous images; The posture estimation module includes an RGB camera posture estimation unit, an infrared camera posture estimation unit, and a posture consistency constraint unit, which is used to estimate the camera posture of heterogeneous cameras and strengthen the convergence through consistency constraints; finally, the relative posture estimation result T is combined with the epipolar geometry t→t' Combined with the estimation results of the depth estimation module, the simultaneous training of the pose network and the depth estimation network can be achieved.

2. The depth estimation device for fusing heterogeneous images for nighttime scenes according to claim 1, characterized in that: The heterogeneous camera adopts an RGB camera and an infrared camera to collect real-time monocular images in night scenes; the training set data is a heterogeneous image pair of a current frame and previous and next adjacent frames. The heterogeneous image acquisition module also includes an acquisition controller for synchronously triggering heterogeneous cameras to synchronously acquire images; the RGB camera and the infrared camera are on the same horizontal line and are close to each other, so as to shoot images from the same viewing angle.

3. The depth estimation device for fusing heterogeneous images for nighttime scenes according to claim 1, characterized in that: The depth estimated by the depth estimation module for fusing heterogeneous images is based on the scene in the RGB camera to obtain the pixel-by-pixel depth value in the RGB image.

4. The depth estimation device for fusing heterogeneous images for nighttime scenes according to claim 3, characterized in that: The RGB image feature extraction unit and the infrared image feature extraction unit both adopt a multi-scale deep feature extraction network based on the U-Net structure, and their internal encoders both adopt a network structure based on MobileNetV3; the network input is the corresponding synchronized image, and the network output is the corresponding depth-related feature map.

5. The depth estimation device for fusing heterogeneous images for nighttime scenes according to claim 1, characterized in that: The extraction of infrared image features also includes mask processing, which is used to mask the identification area that only appears in the infrared image, and retain the identification area that simultaneously appears in the heterogeneous image as a mask template area.

6. The depth estimation device for fusing heterogeneous images for nighttime scenes according to claim 5, characterized in that: The mask shield sets the corresponding position on the feature map obtained by the infrared image feature extraction unit to 0; the mask template area is obtained through neural network training.

7. The depth estimation device for fusing heterogeneous images for nighttime scenes according to claim 1, characterized in that: The RGB camera attitude estimation unit and the infrared camera attitude estimation unit both adopt CNN networks to estimate the camera attitude between adjacent RGB image frames and the camera attitude between adjacent infrared image frames, respectively, and constrain them through the attitude consistency constraint unit so that the camera attitude estimation results of the two heterogeneous cameras should be consistent.

8. The depth estimation device for fusing heterogeneous images for nighttime scenes according to claim 1, characterized in that: The camera pose and depth information estimated by combining the epipolar geometry constraint satisfy the distortion transformation in the epipolar geometry, and the relative pose estimate T t→t' Includes camera offsets in three directions and three angle changes.

9. The depth estimation device for fusing heterogeneous images for nighttime scenes according to claim 1, characterized in that: The method of reconstructing a night scene and finally outputting a dense depth map includes: Combined with locally differentiable bilinear interpolation, we get the source image I t' Reconstructed target reconstructed image I t'→t ; The reconstructed pixel-by-pixel photometric loss is established for iterative training to optimize the corresponding network model parameters of the depth estimation module and the posture estimation module that fused heterogeneous images.

10. A depth estimation method for night scene fusion of heterogeneous images, characterized in that: It is used to perform the following steps in night driving scenarios to perform real-time depth estimation of night scenes: 1) The heterogeneous image acquisition module controls two heterogeneous cameras in real time to perform synchronous image acquisition; 2) In the depth estimation module that fuses heterogeneous images, the RGB image feature extraction unit and the infrared image feature extraction unit receive the RGB real-time image and the infrared real-time image respectively and perform corresponding feature extraction. The heterogeneous feature fusion unit will fuse the above two features and finally output an estimated dense depth map.

Images