A self-supervised depth estimation method in three-dimensional reconstruction of mine safety hidden danger scene

Abstract

The application discloses a self-supervised depth estimation method in mine safety hidden danger scene three-dimensional reconstruction, first, the depth estimation network and the attitude estimation network model of normal illumination image and low illumination image are constructed respectively, the position perception module of self-attention mechanism is used in the middle of the encoder decoder, to obtain the context information of the scene structure and better feature representation; in the process of network training, normal illumination image and low illumination image obtained through CycleGAN are used for training, and then the image output by CycleGAN is processed by using a mapping image enhancement (MIE) algorithm, so that the need of keeping the brightness consistency is met, and the influence caused by low illumination and uneven illumination is solved. The feature representation at the detail is enhanced, and the depth estimation effect on the complex background is strengthened. The added mapping image enhancement module makes the brightness and contrast of the low illumination image obviously improved, so that higher visibility is brought to the low illumination image, and more details are reserved.

Description

Technical Field

[0001] This invention relates to the field of 3D reconstruction technology, specifically a self-supervised depth estimation method for 3D reconstruction of mine safety hazard scenarios. This method performs depth measurement and 3D scene reconstruction of mine safety hazard scenarios, enabling simulation and deduction, and achieving the function of predicting mine safety hazard scenarios through the obtained virtual 3D scene. Background Technology

[0002] With the emergence of concepts such as smart mines, how to perform 3D reconstruction of mine safety hazard scenarios and realize simulation and deduction to provide strong technical support for safe production in mines is an urgent problem to be solved. Depth estimation of mine safety hazard scenarios is an important component of 3D reconstruction. Depth estimation has wide applications in augmented reality, autonomous driving, and robotics. Early methods used depth sensors (LiDAR and DOF) for depth estimation, but equipping such sensors in mines requires high costs and continuous operation, limiting their use. Self-supervised monocular depth estimation can predict the depth of pixels in a single image without such equipment. Furthermore, since the available ground truth depth data for mine hazard scenarios is limited, unsupervised learning methods that do not require precise ground truth data are more suitable. Therefore, unsupervised monocular depth estimation for mine hazard scenarios has attracted widespread attention from researchers.

[0003] Existing self-supervised monocular depth estimation methods typically use geometric constraints on stereo image pairs or monocular sequences as supervision, and have achieved significant progress. Eigen,D's paper, "Depth map prediction from a single image using a multi-scale deep network," introduces related techniques. However, most current self-supervised monocular depth estimation methods primarily address depth estimation for daytime images. Mine images are often acquired under low-light and complex environmental conditions, and depth estimation for mine images is extremely unstable due to low visibility and uneven lighting. CycleGAN converts low-light information into daytime information under good lighting conditions at both the image and feature levels, thus enabling depth estimation for low-light images. However, CycleGAN networks using low-light conditions as input struggle to obtain natural daytime images or features, resulting in limited performance. Monodepth2 is an effective self-supervised monocular depth estimation method, but directly inputting images processed by CycleGAN into Monodepth2 fails to estimate depth details due to the complex environment and uneven lighting.

[0004] Existing technologies have the following drawbacks. First, the depth estimation network of monodepth2 is based on the U-Net framework, and the decoder only uses cascading and a basic convolution to fuse high-level and low-level features. These operations cannot preserve enough detail or accurately recover spatial information, resulting in the ineffective representation of depth features in complex environments.

[0005] Second: During training, images with good lighting conditions are still used, while during testing, low-light images are directly processed by CycleGAN and used as input, making it difficult to obtain image features under natural, good lighting conditions. Summary of the Invention

[0006] To address the shortcomings of the existing technology, the corresponding solutions are as follows:

[0007] First, a position-aware module with a self-attention mechanism is used in the encoder and decoder to obtain contextual information about the scene structure and better feature representation.

[0008] Second, during network training, normal lighting images and low-light images processed by CycleGAN are used for training. Then, the Mapping Image Enhancement (MIE) algorithm is applied to process the images output by CycleGAN to maintain brightness consistency and address the effects of low light and uneven lighting.

[0009] A self-supervised depth estimation method for 3D reconstruction of mine safety hazard scenes is proposed. The method constructs models consisting of depth estimation networks and pose estimation networks for both normally lit and low-light images.

[0010] The specific steps are as follows:

[0011] S1: Normal lighting image Converted into low-light nighttime images using CycleGAN

[0012] S2: The generated low-light image is input into the MIE module for processing;

[0013] S3: Normal lighting image The input is fed into the encoder of the depth estimation network, and the output is the feature map. Low-light images processed by S2 The input is fed into the encoder of the depth estimation network, and the output is the feature map.

[0014] S4: Feature map with the lowest resolution Input is sent to the location sensing module;

[0015] S5: After passing through the position-aware module, the input is sent to the decoder, which will output feature maps respectively. The corresponding depth map

[0016] S6: Combine adjacent frames of normal lighting images and low lighting images. and Inputting the data into a pose estimation network, the six-DOF relative pose information T is calculated. t→s The depth map obtained by combining the depth estimation network Construct a reconstructed view from the original view

[0017] Both the depth estimation network and the pose estimation network employ an encoder-decoder structure.

[0018] The MIE module processing is implemented using the luminance mapping function b' = γ(b):

[0019]

[0020] Here, γ is a monomial mapping function that maps the input brightness to a single specific output, assuming the frequency distribution of the input image. First, a frequency parameter σ is preset. Frequencies exceeding this parameter are clipped to prevent noise signal amplification. Then, the clipped frequencies are evenly distributed across each brightness level. Finally, the result is obtained using the following formula.

[0021]

[0022] cdf min and CDF max These represent the minimum and maximum values ​​of the CDF, respectively, and L represents the number of brightness levels;

[0023] The encoder part in S3 uses ResNet18 as the backbone network, and removes the last average pooling layer and fully connected layer of ResNet18, resulting in the max pooling layer, layer2, layer3, layer4, and layer5, respectively.

[0024] The location-aware module enhances the features of the queried location by aggregating location features from other locations. Assuming the input and output features of this module are A and Z, respectively, the module can be represented as:

[0025]

[0026]

[0027] Among them, Sij The influence of position j on position i is measured; N represents the total number of pixels; W k W q W v These represent Key, Query, and Value, respectively, obtained through a linear transformation of A.

[0028] In step S5, the depth of the normally lit image is used as a pseudo-label to constrain the depth of the low-lit image; the similarity loss is defined as:

[0029]

[0030] Where x∈[1,N], N is and The number of pixels, where x refers to the x-th pixel.

[0031] In the S5 reconstructed view

[0032] p'=KT t→s I z K -1 p

[0033] I' t =W t (I t-1 ,p')

[0034] Where K is the camera intrinsic parameter matrix, p is the homogeneous coordinate of the pixel; p' is the coordinate of p through T. t→t-1 The transformed coordinates, W(·), are differentiable bilinear samplers used to obtain I. t-1 The pixel at p' and in I t The linear interpolation pixel is specified at point p.

[0035] The reconstructed view utilizes a structural similarity index combined with L1 loss as the photometric loss L. ph To constrain the reconstruction of the view, the loss function can be expressed as:

[0036]

[0037] Where α is taken as 0.75.

[0038] The present invention has the following effects: First, the existing monocular depth estimation network decoder only uses a cascaded approach to connect high-level and low-level features, which not only causes the loss of many details, but also cannot make good use of contextual information. By adding a position awareness module using an attention mechanism between the encoder and decoder, the contextual information of the scene is obtained, which enhances the feature representation of details and improves the depth estimation effect in complex backgrounds.

[0039] Second, normal lighting images and low lighting images are input into the network for training, since their depth maps are theoretically identical. A similarity loss is used to constrain the depth estimation of the low lighting image. Simultaneously, the added image enhancement module significantly improves the brightness and contrast of the low lighting image, thereby enhancing its visibility and preserving more details. Attached Figure Description

[0040] Figure 1 This is a schematic diagram of the overall network framework;

[0041] Figure 2 This is a schematic diagram of the location sensing module;

[0042] Figure 3 It is the depth estimation effect of mine safety hazard scenarios. Detailed Implementation

[0043] This invention describes an unsupervised monocular depth estimation method for low-light mine images, with the main objective of improving the accuracy of unsupervised monocular depth estimation in low-light environments. The method mainly includes the following parts: 1. Employing a co-training mode using normal-light and low-light images, the depth map of the normal-light image is used as a pseudo-label to train the low-light image, and similarity loss is used to constrain depth estimation; 2. Incorporating a position-aware module utilizing an attention mechanism into the applied depth estimation network. The overall network model diagram is shown below. Figure 1 As shown.

[0044] The entire model can be viewed as consisting of four networks: depth estimation networks for normal lighting images and pose estimation networks for low lighting images. The depth estimation and pose estimation networks share the same network architecture. The network input will be a normal lighting image. CycleGAN processes normal lighting images into low-light images. Both the depth estimation network and the pose estimation network employ an encoder-decoder structure, where the depth estimation network is used to acquire the desired pose. Depth map Theoretically, the depth maps of the same scene image under different lighting conditions should be the same. Therefore, we use the depth map of the normally lit image as a pseudo-label to constrain the depth of the low-light image. Thus, a similarity loss is added here to constrain the depth map of the low-light image. The pose estimation network is used to estimate the relative pose change between consecutive frames. After obtaining the camera's intrinsic parameter matrix, the relative pose is calculated using the image reconstruction formula, and combined with the depth map obtained from the depth network to construct a reconstructed view of the original view. Ideally, a sufficiently good network should result in a reconstructed view that is consistent with the original view; therefore, a reconstruction loss is added here to constrain the reconstructed view.

[0045] Part 1: Obtaining Feature Maps

[0046] First, CycleGAN transforms normal lighting images Processed low-light image Then use the MIE module to process low-light images The processing, where the MIE module is implemented using the luminance mapping function b' = γ(b):

[0047]

[0048] Here, γ is a monomial mapping function that maps the input brightness to a single specific output, assuming the frequency distribution of the input image. We first preset the frequency parameter σ. We then clip frequencies greater than the preset parameter to avoid amplifying noise signals. Next, we evenly distribute the clipped frequencies across each brightness level. Finally, we obtain the result using the following formula.

[0049]

[0050] cdf min and CDF max These represent the minimum and maximum values ​​of the CDF, respectively, and L represents the number of brightness levels (usually 256 in color images).

[0051] Will The inputs are fed into the encoder of the depth estimation network, which uses ResNet18 as the backbone network. However, the final average pooling layer and fully connected layer of ResNet18 are removed. Feature maps are obtained from the max pooling layer, layer 2, layer 3, layer 4, and layer 5, respectively. Part Two: Obtaining the Depth Map

[0052] First, the feature map with the lowest resolution output from the encoder is... Input to the location-aware module, such as Figure 2 As shown, the location-aware module aims to enhance the features of the query location by aggregating location features from other locations. Assuming the input and output features of this module are A and Z respectively, the module can be represented as:

[0053]

[0054]

[0055] Among them, S ij Measure the influence of position j on position i.

[0056] Then, the depth features obtained from the location-aware module are input into the decoder. To reduce the computational cost of the network, skip connections are used at higher-level features. The image outputs a depth map after passing through the entire network. The network framework proposed in this invention takes a normally lit image and a correspondingly generated low-light image as input, so the estimated depth maps of the two images should be consistent. Due to the inherent advantage of normally lit images in depth estimation, the estimated depth of the low-light image should be as close as possible to that of the normally lit image; that is, the depth of the normally lit image is used as a pseudo-label to constrain the depth of the low-light image. Therefore, the similarity loss is defined as:

[0057]

[0058] Where x∈[1,N], N is and The number of pixels, where x refers to the x-th pixel.

[0059] Part Three: Reconstructing the View

[0060] Adjacent frames of normal lighting and low lighting images and Inputting the data into a pose estimation network, the six-DOF relative pose information T is calculated. t→s Combined with the depth map obtained from the depth estimation network in Part 2. Construct a reconstructed view from the original view

[0061] p'=KT t→s I z K -1 p

[0062] I' t =W t (I t-1 ,p')

[0063] Where K is the camera intrinsic parameter matrix, p is the homogeneous coordinate of the pixel; p' is the coordinate of p through T. t→t-1 The transformed coordinates, W(·), are a differentiable bilinear sampler used to obtain I. t-1 The pixel at p' and in I t The linear interpolation pixel is specified at point p.

[0064] Ideally, a sufficiently good network should reconstruct a view that is consistent with the original view. This is achieved by using a structural similarity index (SSIM) combined with L1 loss as the photometric loss. ph To constrain the reconstruction of the view, the loss function can be expressed as:

[0065]

[0066] Where α is taken as 0.75.

[0067] In the experimental testing phase, low-light nighttime images were used, and the encoder backbone network was ResNet18. Higher values ​​for the first three indicators indicate better prediction performance, while lower values ​​for the last three indicators indicate better performance. The final results show that the method of this invention has good performance in depth estimation of low-light mine images, and all indicators are improved.

[0068] Experimental results:

[0069]

[0070] like Figure 3 As shown, the left side is the input image, and the right side outputs the depth image after processing by our network method. The result demonstrates how our method can accurately construct a depth map of a mine safety hazard scene, providing effective depth information for 3D reconstruction.

Claims

1. A self-supervised depth estimation method for 3D reconstruction of mine safety hazard scenarios, characterized in that, First, depth estimation and pose estimation network models are constructed for normal lighting images and low lighting images, respectively. Both the depth estimation network and the pose estimation network adopt an encoder-decoder structure. The processing steps are as follows: S1: Normal lighting image Converted into low-light nighttime images using CycleGAN ; S2: The generated low-light image is input to the MIE module for processing; the MIE module processes the image using a brightness mapping function. Achieved: ； in, It is a monad mapping function that maps input brightness to a single specific output, assuming the frequency distribution of the input image. First, preset the frequency parameters. Frequency values ​​exceeding a preset parameter are clipped to avoid amplifying noise signals. The clipped frequencies are then evenly distributed across each brightness level. Finally, the result is obtained using the following formula. : ； in and These represent the minimum and maximum values ​​of the CDF, respectively, and L represents the number of brightness levels; S3: Normal lighting image The input is fed into the encoder of the depth estimation network, and the output is the feature map. i = 1, 2, 3, 4, 5, representing the low-light images processed by S2. The input is fed into the encoder of the depth estimation network, and the output is the feature map. , i = 1, 2, 3, 4, 5; S4: Lowest resolution feature map , Input to the location-aware module; S5: After passing through the position-aware module, the data is input into the decoder, which will output feature maps respectively. , The corresponding depth map , ; In step S5, the depth of the normally lit image is used as a pseudo-label to constrain the depth of the low-lit image; the similarity loss is defined as: in N is and The number of pixels, where x refers to the x-th pixel; S6: Combine adjacent frames of normal lighting images and low lighting images. , and , Input the data into the pose estimation network to calculate the six-DOF relative pose information. The depth map obtained by combining the depth estimation network , Construct a reconstructed view from the original view , .

2. The self-supervised depth estimation method for three-dimensional reconstruction of mine safety hazard scenarios according to claim 1, characterized in that, The encoder part in S3 uses ResNet18 as the backbone network, and removes the last average pooling layer and fully connected layer of ResNet18, resulting in the max pooling layer, layer2, layer3, layer4, and layer5, respectively.

3. The self-supervised depth estimation method for three-dimensional reconstruction of mine safety hazard scenarios according to claim 1, characterized in that, The location-aware module enhances the features of the queried location by aggregating location features from other locations. Assuming the input and output features of this module are A and Z respectively, the module is represented as follows: in, Measure the influence of position j on position i; N represents the total number of pixels; These represent Key, Query, and Value, respectively, obtained through a linear transformation of A.

4. The self-supervised depth estimation method for three-dimensional reconstruction of mine safety hazard scenarios according to claim 1, characterized in that, In the S6 reconstructed view Where K is the camera intrinsic parameter matrix, p is the homogeneous coordinate of the pixel; It is p through Transformed coordinates It is a differentiable bilinear sampler used to acquire middle Pixels at the location and The linear interpolation pixel is specified at point p.

5. The self-supervised depth estimation method for three-dimensional reconstruction of mine safety hazard scenarios according to claim 4, characterized in that, The reconstructed view utilizes a structural similarity index combined with L1 loss as the photometric loss L. ph To constrain the reconstruction of the view, the loss function is expressed as: in Take 0.75.

Images