Mine enhanced reality unmanned mining working face depth estimation method

Abstract

The application discloses a kind of mine augmented reality unmanned mining working face depth estimation method, belong to image processing technical field, construct the deep encoder of convolution layer, pooling layer, residual layer, down-sampling module composition, shallow high-resolution feature information of image is extracted in shallow layer of encoder network, through the stack of convolution operation in deep layer of encoder network, image low rate feature information is extracted, shallow high-resolution feature map is fused into deep low rate feature map by hierarchical feature adjustment module;Deep decoder network is constructed by deconvolution layer, up-sampling module and jump connection, and the depth of image is estimated;Camera pose prediction network is constructed, and the photometric loss of re-projection to target image by the estimation of camera adjacent frame motion pose is used as self-supervised signal constraint;Improved color-geometry difference information is constructed to predict coplanar depth map module, and vector gradient gradual change is used as constraint signal, and the initial depth map and predicted coplanar depth map are constrained.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of image processing, and specifically relates to a mine enhanced reality unmanned mining working face depth estimation method, which can be applied to mine unmanned mining working face depth estimation. BACKGROUND

[0002] Intelligent coal mine construction has experienced a development process from mechanization, automation, intelligence to unmanned, among which intelligent unmanned working face mining is an important link of intelligent coal mine construction, but it is still in the initial stage and a series of technical problems need to be solved. At present, the main technical difficulties of intelligent unmanned mining working face construction include the construction exploration, inclination adjustment, straightening and leveling of the working face, the on-site supervision of the equipment, automatic monitoring of hidden dangers, anti-interference and transmission efficiency in remote control of the system and other problems, which seriously affect the progress space and popularization coverage of intelligent mine construction. How to improve the intelligent degree of the working face to achieve the purpose of cost reduction and efficiency improvement, safe production and reduction of enterprise labor cost is an important basis for realizing intelligent mining. Therefore, in the exploration of intelligent unmanned working face and the three-dimensional reconstruction, augmented reality and mathematical modeling in the real-time monitoring and hidden danger maintenance stage, how to improve the reconstruction and modeling capability puts forward higher requirements for depth estimation.

[0003] Depth estimation as a computer vision task aims to estimate the depth of objects in a two-dimensional image. The task requires inputting an RGB image and outputting a depth image. The depth image includes information about the distance from the viewpoint to the objects in the image, and the viewpoint is usually the camera that takes the image. Depth estimation as a low-level task needs to complete high-level tasks such as three-dimensional reconstruction, autonomous driving and three-dimensional target detection. Current depth estimation can be divided into supervised depth estimation, semi-supervised depth estimation and self-supervised depth estimation according to the supervision mode. Supervised depth estimation needs to use real depth label data for training, semi-supervised depth estimation combines other information and real depth label data for training, thereby reducing the dependence on real depth label data, and self-supervised depth estimation does not rely on real depth label data for training. Due to the difficulty in obtaining real depth label data, high equipment requirements and high price, in real life, especially for depth estimation of mine unmanned mining working face, self-supervised depth estimation is undoubtedly the best depth estimation method. Most current depth estimation networks use the U-Net network framework.

[0004] The quality of the image on the mine unmanned mining face is obviously decreased when the existing self-supervised depth estimation method is used for training. The main reason is that the situation of the unmanned working face is very different from the mine image. The unmanned mining face is full of a large number of low light and low texture areas, and the image lacks many texture features, such as smooth surfaces of the working face roadway, walls and transportation belts. The paper published in the Journal of Mining and Rock Control Engineering by Zhang Kexue et al. Intelligent Unmanned Mining Face Applicability Multilevel Fuzzy Comprehensive Evaluation Research has made a preliminary discussion, and still needs to overcome the following technical problems.

[0005] 1. The existing depth estimation method is only applicable to outdoor monocular depth estimation, and the effect of predicting the depth map of the unmanned mining face with complex structure, low texture and low light is poor, and the structural rules of the image are not fully utilized for depth estimation.

[0006] 2. In the neural network of the depth estimation method, the down-sampling of the encoder and the up-sampling of the decoder are irreversible. With the increase of the down-sampling level of the encoder, the features of the image are inevitably missing. In the up-sampling of the decoder, the feature information of the middle layer of the image is not fully utilized, resulting in inaccurate estimated depth. SUMMARY

[0007] The present application proposes the following two ways to solve the technical problem of inaccurate depth estimation of the current unmanned mining face based on the StructDepth structure framework.

[0008] First, in view of the problem of low texture area in the unmanned working face affecting the depth estimation effect, a new constraint signal based on the structure of the unmanned working face, vector gradient gradual loss, is proposed to constrain the difference between the initial estimated depth map and the predicted depth map.

[0009] Second, in view of the problem of feature loss and not making good use of other layer feature information in the up-sampling and down-sampling of the encoder-decoder in the current depth estimation network, causing inaccurate depth estimation, blurred and even lost boundaries, etc., the present application proposes a depth estimation network structure with an added hierarchical feature adjustment module based on attention mechanism.

[0010] The specific technical solutions are as follows:

[0011] 1. Construct a deep encoder based on convolutional layers, pooling layers, residual layers, down-sampling modules, and hierarchical feature adjustment modules based on attention mechanisms, for extracting image feature information. 2. Construct a deep decoder composed of convolutional layers, up-sampling modules, skip connections, and hierarchical feature adjustment modules based on attention mechanisms, for estimating the depth map of the image. 3. Construct a camera pose prediction network, and use the photometric loss of the re-projection of the estimated motion pose between adjacent frames of the camera to the target image as a self-supervised signal as a constraint. 4. Construct an improved color-geometry difference information to predict the planar depth map module, using the vector gradient as a self-supervised signal to constrain the predicted initial depth map and planar depth map.

[0012] Construct a depth estimation encoder structure: construct a deep encoder composed of convolutional layers, pooling layers, residual layers, and down-sampling modules, extract shallow high-resolution feature information in the image at the shallow layer of the encoder network, and extract deep low-resolution feature information in the image through the stacking of convolutional operations at the deep layer of the encoder network, and stack the shallow high-resolution feature map to the deep low-resolution feature map through the hierarchical feature fusion module;

[0013] Construct a depth estimation decoder structure: construct a deep decoder composed of convolutional layers, up-sampling modules, and skip connection methods to estimate the depth map of the image;

[0014] Construct a co-planar constraint prediction depth map module: construct a camera pose prediction network, and use the photometric loss of the re-projection of the estimated motion pose between adjacent frames of the camera to the target image as a self-supervised signal constraint;

[0015] Predict the co-planar depth map: construct an improved color-geometry difference information to predict the co-planar depth map module, and use the vector gradient as a constraint signal to constrain the initial depth map and the predicted depth map.

[0016] Further, the depth encoder of step (1) is ResNet18.

[0017] Further, the depth encoder first passes the original RGB image with a size of HxW and a channel number of 3 through a 7x7 convolutional layer with a step size of 2, a batch normalization layer, a linear rectifier function layer, and a maximum pooling layer, to obtain a high-resolution feature map F1 with a size of and a channel number of 64; then the feature map is processed by the BasicBlock module of Layer1 of the Resnet18 network to obtain a feature map F2 with an output size of and a channel number of 64, and then F2 is processed by the BasicBlock module of Layer2 of the Resnet18 network to obtain a feature map F3 with an output size of The feature map F3 with 128 channels is then down-sampled by the BasicBlock module of the Layer3 layer of the ResNet18 network to obtain an output with a size of The feature map F4 with 256 channels is finally down-sampled by the BasicBlock module of the Layer4 layer of the ResNet18 network to obtain an output with a size of The feature map F5 with 512 channels.

[0018] Further, the deep encoder adds a hierarchical feature adjustment module based on an attention mechanism, which is composed of a parallel adjustment module and an attention module. The size of the feature map F2 with 64 channels is first increased to 128 channels by the parallel adjustment module, i.e., a 1x1 convolutional layer, and then the size is changed to The feature map F2 with 64 channels is first increased to 128 channels by the parallel adjustment module, i.e., a 1x1 convolutional layer, and then the size is changed to

[0019] F′=σ(MLP(AvgPool(F))+MLP(MaxPool(F))) (5-1)

[0020] wherein σ is a sigmoid activation function mapping process, and the feature map F′ obtained from the channel attention channel module is input into the spatial attention module. The process is as follows: F′ and the maximum pooling and average pooling in the channel dimension are respectively obtained to obtain two new feature information blocks, the two feature information blocks are spliced according to the channel direction, the spliced result is processed by a 7x7 convolutional layer, and a sigmoid activation function is used for mapping to obtain a spatial attention value, which is multiplied by the feature map F′ to obtain the final feature map F″; the formula is shown in 5-2:

[0021] F″=σ(f 7×7 ([AvgPool(F);MaxPool(F)])) (5-2)

[0022] wherein σ is a sigmoid activation function mapping process, and f 7×7 ​is the result of the hierarchical feature adjustment module; the feature map F" is superimposed and fused with the feature map F3 to obtain a new feature map F3 which is input into the BasicBlock module of the Layer4 layer of the ResNet18 network; on the other hand, the new feature map F3 is input into the hierarchical feature adjustment module to obtain a new feature map which is superimposed and fused with the feature map F4 and then input into the BasicBlock module of the Layer5 layer of the ResNet18 network as input to obtain the final feature map F5.

[0023] Further, the feature map F5 has a size of The channel number is 512, and the depth estimation decoder is used to gradually reduce the channel number and improve the resolution to finally obtain a depth estimation image with a channel number of 3 and a size same as that of the original image; the specific expression is as follows:

[0024] The decoder expands the feature map F5 output by the encoder by one time through upsampling processing to obtain a feature map F'5 with a size of The channel number is changed to half of the original number, i.e. 256, through convolution operation to obtain a feature map F'4. The feature maps F4 and F'4 of the fourth layer of the encoder are spliced in the channel direction, and then the upsampling processing, deconvolution operation and jump connection operation are performed three times according to this method to obtain a feature map F'1 with a size same as that of the original image HxW and a channel number of 64, and then the final disparity map is obtained through a fully connected layer, and the sigmoid activation function is used to process to obtain the final depth estimation image; the hierarchical fusion part of the decoder is that the output F'3 of the third layer of the decoder passes through the same attention mechanism module as the encoder to obtain a new feature map S1 and the output F'2 of the second layer of the decoder passes through the attention mechanism module to obtain a new feature map S2, the feature maps S1 and S2 are added through the hyperparameter weight λ=0.2, and then added with the feature map F'1 output by the first layer of the decoder to perform feature fusion.

[0025] Further, the step (3) is specifically as follows:

[0026] (1) Manhattan region detection is performed on the objects in the image to obtain the aligned normal By calculating the total number of planes in the unmanned mining working face, N is selected as the limit value, when N is greater than 10, 50% of the planes are selected as the planes for estimating the normal vector, when N is less than or equal to 10, 70% of the planes are selected as the planes for estimating the normal vector, the normal of the estimated plane is predicted n, Manhattan normal detection is used to detect the plane normal belonging to the main plane, and then the cosine similarity S(n, n) is used to compare the difference between the normal vector n of the estimated plane and each possible main direction n p k ​​

[0027] n p ∈(n1,n2,,,n 70％d )

[0028] where d is the total number of planes and n1>n2>n3....n 70％d ,

[0029]

[0030] where is the alignment normal, the cosine similarity is defined as s(n p ,η k )=(n p ·η k ) / (||n p ||·||η k ||);

[0031] (2) Predicting co-planar depth map

[0032] Using the alignment normal calculated in Manhattan region detection to derive its dissimilarity, let Xp be the 3D coordinate of pixel p, assuming that the 3D point lies in the plane whose normal is the alignment normal, then the distance of the plane to the origin is calculated as:

[0033]

[0034] Assume q is the neighboring pixel of p, the normal dissimilarity between them is defined as the Euclidean distance between two vectors:

[0035]

[0036] By and representing the maximum and minimum dissimilarity between all neighboring pixels respectively,

[0037] Define the [·] operator,

[0038]

[0039] then the dissimilarity of the distance of the plane to the origin is defined as:

[0040] D d (p,q)=|d p -d q |

[0041] Then the geometric information dissimilarity combines both dissimilarities into a normalized one:

[0042] D g (p,q)=[D n (p,q)]+[D d (p,q)]

[0043] Color information difference is calculated as:

[0044] D c (p,q) = ||I P -I q |

[0045] where I P , I q are RGB colors; finally, color information difference and geometry information difference are assigned weights to combine:

[0046] D(p,q) = 0.4*D c (p,q) + 0.6*D g (p,q)

[0047] Based on the difference, graph-based segmentation and filtering out small areas to obtain the planar region, the plane fitting of the three-dimensional points in the planar region is obtained, and the plane parameter θ = -n / d ∈ R 3 is obtained by solving the least square problem,

[0048] X T θ = 1

[0049] where X ∈ R 3×N represents the three-dimensional points in the planar region; then the inverse depth ρ p of the pixel p is calculated by plane fitting:

[0050]

[0051] where K represents the camera intrinsic matrix, and then the inverse depth is converted into depth Using the depth obtained from plane fitting

[0052] (3) Vector normal gradual change constraint

[0053] A self-supervised constraint signal is established by using the normal vector to supervise the corner, the angle between n i-1 and n i+1 is calculated to represent the gradient g i of the normal angle, and then the L1 loss is applied:

[0054] g i = arcose(n i-1 · n i+1 )

[0055]

[0056] where, g i and are the gradients estimated from the predicted initial depth map and the predicted co-planar depth map, respectively.

[0057] Advantages

[0058] Firstly, the present application is a new method of encoder level feature fusion, which uses a level feature adjustment module to change the size and channel number of the feature information of the shallow layer of the encoder (the second and third layer feature outputs of the encoder described in the invention) through a parallel adjustment module, which has a smaller semantic gap than directly changing the size and channel number. In addition, the output of the parallel adjustment module is sequentially passed through the channel attention module and the spatial attention module, so that important features of the fusion are highlighted and unimportant features are suppressed. Thus, the level fused features can better express the characteristics of the image, reducing the error.

[0059] Secondly, a new decoder level feature fusion method. The present application uses a level feature adjustment module to pass the feature information of the third and second layers of the decoder through the attention module in the above-mentioned encoder, increases the weight of important features, suppresses the interference caused by unimportant features, and weights their output by a super parameter λ = 0.2 to be added to the output of the fifth layer of the decoder. This way, the features of other levels are fully utilized, making the estimated depth map boundary more detailed and the depth information recovery better. Moreover, with the reference of the super parameter, the interference to the backbone decoder network is reduced, thereby reducing the possible negative impact on precision.

[0060] Thirdly, a new loss function based on structural rules. The present application estimates and calculates the vector n i-1 , the vector n i+1 , and uses the angle between the two vectors to represent the gradient, and then uses L1 loss to represent the gradient difference between the two, so that when L1 is 0, it represents the corner of the vertical plane in the unmanned working face, which well monitors the corner of the vertical plane, thereby serving as a self-supervised signal to constrain the depth map. The proposed loss function well solves the problem of model deterioration caused by false planes in low-texture and low-illumination areas during estimation of the depth of the unmanned working face. BRIEF DESCRIPTION OF DRAWINGS

[0061] Figure 1 is the overall framework diagram of the present application;

[0062] Figure 2 is the structure diagram of the depth estimation network;

[0063] Figure 3 is the schematic diagram of the level feature adjustment module;

[0064] Figure 4 is the vector corner principle diagram;

[0065] Figure 5 is the estimated depth map of the network model of the application and the advanced network on the NYU Depth v2 dataset,

[0066] Figure 6 is the estimated depth map of the network model of the application and the advanced network on the NYU Depth v2 dataset,

[0067] Figure 7 is the estimated depth map of the application on the unmanned mining working face. DETAILED DESCRIPTION

[0068] The technical solutions of the application will be further explained below in combination with the drawings.

[0069] The application proposes a self-supervised depth estimation method for three-dimensional reconstruction and augmented reality in the intelligent unmanned mining working face scene, which innovatively proposes a new constraint supervision signal between the initial depth map and the predicted depth map, innovatively improves the existing depth estimation network, improves the depth estimation effect of the intelligent unmanned mining working face, and improves the accuracy of the depth estimation. The depth estimation method includes three parts: 1. A depth encoder based on a convolutional layer, a pooling layer, a residual layer, a down-sampling module and a hierarchical feature adjustment module based on an attention mechanism is constructed, which is used for extracting image feature information. 2. A depth decoder composed of a convolutional layer, an up-sampling module, a skip connection and a hierarchical feature adjustment module based on an attention mechanism is constructed, which is used for estimating the depth map of the image. 3. A camera pose prediction network is constructed, and the photometric loss of the camera inter-frame motion pose re-projection to the target image is used as a self-supervised signal as a constraint. 4. An improved color-geometry difference information is constructed to predict the coplanar depth map module, and the vector gradient gradient is used as a constraint signal to constrain the initial depth map and the predicted depth map. The overall framework of the application is shown in Fig. 1: the first part is to construct the depth estimation encoder structure Figure 1

[0070] ​In a general image depth estimation network, the features of the image are extracted by taking a residual network or a dense network as a feature extractor. The present application takes ResNet18 as a depth feature information encoder to extract feature information. In the shallow layer of the encoder network, simple information of the image such as color, texture, object boundary and the like is extracted. Further, in the deep layer of the encoder network, important feature information inside the image can be extracted through the stacking of convolution operations and the increase of the receptive field. At the same time, the process of the encoder extraction is a process of changing the extracted features from specific to abstract. The more abstract, the more the internal geometric spatial correlation of the scene in the original image will gradually weaken. In order to maintain this spatial correlation, a hierarchical feature adjustment module based on an attention mechanism is added to the structure of ResNet18 as the encoder. After the high-resolution feature map in the shallow layer is stacked and fused into the low-resolution feature map in the deep layer through the hierarchical feature adjustment module, the spatial correlation can be strengthened. The specific operation is as shown in Figure 2

[0071] As shown in Figure 2 , the left part of the figure is the ResNet18 feature extraction as the encoder. First, an RGB original image with a size of HxW and a channel number of 3 is passed through a 7x7 convolution with a step of 2, a batch normalization layer (BN layer) and a linear rectifier function layer (ReLu layer), a maximum pooling layer (Maxpool layer), to obtain a high-resolution feature map F1 with a size of and a channel number of 64. Then the feature map is passed through the BasicBlock module of Layer1 of ResNet18 network for down-sampling operation to obtain a feature map F2 with an output size of and a channel number of 64. Then F2 is passed through the BasicBlock module of Layer2 of ResNet18 network for down-sampling operation to obtain a feature map F3 with an output size of and a channel number of 128. Then F3 is passed through the BasicBlock module of Layer3 of ResNet18 network for down-sampling operation to obtain a feature map F4 with an output size of and a channel number of 256. Finally, F4 is passed through the BasicBlock module of Layer4 of ResNet18 network for down-sampling operation to obtain a feature map F5 with an output size of and a channel number of 512, thus the main framework of the encoder is completely constructed.

[0072] As shown in Figure 3 , the hierarchical feature adjustment module is composed of a parallel adjustment module and an attention module. The size of the input of the module is ​The feature map F2 with 64 channels is first improved to 128 channels by a parallel adjustment module, i.e., a 1x1 convolution layer, and then the size is changed to The feature map F2 is directly superimposed with new feature information obtained by a 3x3 convolution layer with a step of 2 to form a feature information map F. In order to enable the superimposed feature map F to allocate more weights to positions where important information channels and information aggregation are more in the channel direction, suppress unnecessary information, F is first passed through a channel attention module (CAM), i.e., F is subjected to maximum pooling and average pooling operations to obtain two feature blocks, and then is input into a shared multi-layer perceptron (MLP) to learn the features in the channel dimension and the importance degree of each channel. Finally, the two learned feature blocks are directly added, a sigmoid activation function is used for mapping processing to obtain the final channel attention value, and the new feature map F' is obtained by multiplying the channel attention value and the input feature map F. The formula is shown in 5-1:

[0073] F' = σ(MLP(AvgPool(F)) + MLP(MaxPool(F))) (5-1)

[0074] wherein σ is a sigmoid activation function mapping processing, the feature map F' obtained from the channel attention module is input into a spatial attention module, and the process is as follows: F' is subjected to maximum pooling and average pooling in the channel dimension to obtain two new feature information blocks, the two feature information blocks are spliced according to the channel, the spliced result is subjected to a 7x7 convolution layer processing, a sigmoid activation function is used for mapping processing to obtain a spatial attention value, and the final feature map F'' is obtained by multiplying the spatial attention value and the feature map F'; the formula is shown in 5-2:

[0075] F'' = σ(f 7×7 ([AvgPool(F); MaxPool(F)]) (5-2)

[0076] wherein σ is a sigmoid activation function mapping processing, f 7×7 is the above-mentioned 7x7 convolution processing; the feature map F'' obtained at this time is the result of the hierarchical feature adjustment module; the feature map F'' is superimposed and fused with the feature map F3 to obtain a new feature map F3, which is input into the BasicBlock module of the Layer4 layer of the ResNet18 network as input; on the other hand, the new feature map F3 is input into the hierarchical feature adjustment module to obtain a new feature map, which is superimposed and fused with the feature map F4 and then is input into the BasicBlock module of the Layer5 layer of the ResNet18 network as input to obtain the final feature map F5, thereby completing the construction of the encoder network.

[0077] Second part: constructing a depth estimation decoder structure

[0078] The depth estimation decoder main structure is to obtain a feature map F5 with a size of 512 channels from the encoder, gradually reduce the number of channels and increase the resolution to finally obtain a depth image with the same size as the original image and a channel number of 3. The specific expression is:

[0079] The decoder expands the feature map size by one time through the upsampling operation of the feature map F5 output by the encoder, that is, and then through a convolution operation, the number of channels is changed to half of the original, that is, 256, to obtain a feature map F'4. The fourth layer feature map F4 of the encoder is spliced with F'4 in the channel direction, and then the upsampling processing, convolution operation and jump splicing operation are performed 3 times according to this method to obtain a feature map F'1 with the same size as the original image HxW and a channel number of 64. Then, the final disparity map is obtained through the full connection layer, and the sigmoid activation function operation is performed to obtain the final depth estimation map. The hierarchical fusion part of the decoder is that the third layer output F'3 of the decoder obtains a new feature map S1 through the same attention mechanism module as the encoder, and the second layer output F'2 of the decoder obtains a new feature map S2 through the attention mechanism module. Then, the feature maps S1 and S2 are added through the hyperparameter weight λ=0.2, and then added with the first layer output feature map F'1 of the decoder to perform feature fusion.

[0080] Third part: constructing a co-planar constraint prediction depth map module

[0081] 3.1 Manhattan region detection

[0082] The present application first performs Manhattan region detection on the objects in the image to obtain the aligned normal The total number of planes in the unmanned mining working face is calculated, and then N is selected as the limit value. When N is greater than 10, the larger plane (taking 50%) is selected as the plane of the estimated normal vector, and when N is less than or equal to 10, the larger plane (taking 70%) is selected as the plane of the estimated normal vector. The estimated plane normal prediction n is detected by using the Manhattan normal vector detection to classify the plane normal belonging to the main plane, and then the cosine similarity S(n, n) is used to compare the difference between the estimated plane normal n p and each possible main direction η k , and select one with the best similarity, that is,

[0083] n p ∈(n1,n2,,,n 70％d )

[0084] where d is the total number of planes and n1>n2>n3....n 70％d .

[0085]

[0086] where is the aligned normal, and the cosine similarity is defined as s(n p ,η k ) = (n p ·η k ) / (||n p ||·||η k ||).

[0087] 3.2 Predicting the co-planar depth map

[0088] This self-supervised approach adopted by the present invention takes into account the color, the plane normal and the distance from the origin to the plane, as well as the gradient of the perpendicular face turning angle. The invention uses the aligned normal computed in the Manhattan region detection to derive its dissimilarity. Let the 3D coordinates of pixel p be X p . Assuming that this 3D point lies in a plane whose normal is the aligned normal, the distance from the plane to the origin is computed as:

[0089]

[0090] Assuming that q is a neighboring pixel of p, the normal dissimilarity between them is defined as the Euclidean distance between the two vectors:

[0091]

[0092] By and representing the maximum and minimum dissimilarity between all neighboring pixels, respectively, another [·] operator is defined,

[0093]

[0094] The distance dissimilarity from the plane to the origin is then defined as:

[0095] D d (p,q) = |d p -d q |

[0096] The geometric information dissimilarity combines both dissimilarities normalized:

[0097] D g (p,q) = [D n (p,q)] + [D d (p,q)]

[0098] The color information dissimilarity is computed as:

[0099] D c (p,q) = ||IP -I q ||

[0100] where I P , I q are RGB colors. Finally, according to the fact that the information of the unmanned working face depends on more geometric information, the weight values are assigned to the color information difference and the geometric information difference to combine them:

[0101] D(p,q) = 0.4*D c (p,q) + 0.6*D g (p,q)

[0102] Based on the difference, the application applies graph-based segmentation and filters out small areas to obtain a planar area. Compared with using only color information, our method fully considers the geometric information in the structure of the unmanned mining working face of the mine, avoids false planar areas that cannot be distinguished by color and excessive segmentation caused by different colors. After reasonably detecting the planar area, the coplanar constraint is called to flatten the three-dimensional points located in the planar area. The first step is the plane fitting of the three-dimensional points in the planar area. The application obtains the plane parameter θ = -n / d ∈ R 3 As the previous work, by solving the least squares problem,

[0103] X T θ = 1

[0104] where X ∈ R 3×N represents the three-dimensional points in the planar area. Then the inverse depth ρ p of the pixel p is calculated by the plane fitting as follows:

[0105]

[0106] where K represents the camera intrinsic matrix, and then the inverse depth is converted into depth using the depth

[0107] 3.3 Vector normal gradual change constraint

[0108] The application uses the characteristics of the normal vector at the corner of the vertical plane to establish a self-supervised constraint signal. The normal changes near the corner of the vertical plane, and the principle is shown in Figure 4 .

[0109] In order to supervise the corner, the application calculates the angle between n i-1 and n i+1 to represent the gradient g i of the normal angle, and then applies the L1 loss:

[0110] g i = arcos(ni-1 ·n i+1 )

[0111]

[0112] where g i and are the gradient vectors predicted from the predicted initial depth map and the predicted planar depth map, respectively.

[0113] To verify the effectiveness of the improved method in this paper, the indoor dataset NYU DepthV2 dataset is used for training and testing, and compared with several current advanced indoor self-supervised monocular depth estimation algorithms. The experimental results are shown in Table 1.

[0114] Table 1 Comparison of experimental results of network model in this paper and advanced network on NYU DepthV2 dataset

[0115]

[0116] From Table 1, compared with other indoor monocular depth estimation algorithms, this paper achieves the optimal effect in the six evaluation indexes. The most important index δ < 1.25 for measuring depth estimation effect is improved by 0.5% compared with the current advanced algorithm StructDepth, RMSE is reduced by 0.8%, Abs Rel is reduced by 0.4%, which shows that the monocular depth estimation network model proposed in this paper is the best in the current main self-supervised indoor monocular depth estimation network. The specific prediction depth map experimental effect is shown in Table 2. Figure 5 It can be seen that the network model in this paper and the existing indoor main self-supervised method p 2 Net, StructDepth improves the current indoor depth estimation of vertical plane corner blur, depth blur and detail loss, especially in the red box.

[0117] In addition, in order to more fully illustrate the effectiveness of our method in depth estimation on unmanned mining face, several mine scenes are used for depth estimation, as shown in Table 3. Figure 6 Good results are achieved.

[0118] Table 2 Ablation experiment

[0119]

[0120] The ablation experiment further verifies the effectiveness of each module on the depth estimation network. As shown in Table 2, the experiment finds that the experimental effect of the encoder level feature adjustment fusion innovation point is better than that of the decoder level feature adjustment fusion innovation point and the vector normal gradient gradual change loss function innovation point. Whether it is a combination of two or a combination of three, it can be better than method 0-StructDepth effect, and the combination of three has the best experimental effect.

Claims

1. A method of estimating the depth of a mine enhanced reality unmanned mining face, characterized in that, Comprise the following steps: (1) constructing a depth estimation encoder structure: constructing a deep encoder composed of convolutional layers, pooling layers, residual layers, and down-sampling modules; shallow high-resolution feature information in the image is extracted in the shallow layer of the encoder network, and low-resolution feature information in the deep layer of the image is extracted through the stacking of convolutional operations; the high-resolution feature map of the shallow layer is stacked into the low-resolution feature map of the deep layer through a hierarchical feature fusion module; (2) constructing a depth estimation decoder structure: constructing a depth estimation network composed of convolutional layers, up-sampling modules, and skip connection methods; (3) constructing a co-planar constraint predicted depth map module: constructing a camera pose prediction network, and projecting the camera inter-frame motion pose to the target image through the estimation of the photometric loss as a self-supervised signal; (4) predicting an initial depth map using the depth estimation network and the camera pose network; (5) predicting a co-planar depth map: constructing an improved color-geometry difference information prediction co-planar depth map module to estimate the predicted co-planar depth map; (6) using vector gradient gradual change as a constraint signal to constrain the predicted initial depth map and the predicted co-planar depth map; The deep encoder first processes the RGB original image with a size of and a channel number of 3 through a 7*7 convolution layer with a step of 2, a batch normalization layer, a linear rectifier function layer and a maximum pooling layer to obtain a high-resolution feature map with a size of and a channel number of 64 ; then the feature map is processed through the BasicBlock module of the Layer1 layer of the ResNet18 network for down-sampling to obtain a feature map with an output size of and a channel number of 64 ; then is processed through the BasicBlock module of the Layer2 layer of the ResNet18 network for down-sampling to obtain a feature map with an output size of and a channel number of 128 ; then is processed through the BasicBlock module of the Layer3 layer of the ResNet18 network for down-sampling to obtain a feature map with an output size of and a channel number of 256 ; finally, is processed through the BasicBlock module of the Layer4 layer of the ResNet18 network for down-sampling to obtain a feature map with an output size of and a channel number of 512 . The deep encoder adds an attention mechanism-based hierarchical feature adjustment module, which is composed of a parallel adjustment module and an attention module. , and the channel number is 64 First, the parallel adjustment module, i.e. , is used to increase the channel number of the feature information to 128, and then the maximum pooling layer is used to change the size to , and the feature map is directly superimposed with the new feature information image pixels obtained by the convolution layer with a step of 2 to obtain a feature information map , which is input into the channel attention module, i.e. , to obtain two feature information blocks through maximum pooling and average pooling operations, respectively, and then the two blocks are input into a shared multi-layer perceptron (MLP) to learn the feature in the channel dimension and the importance of each channel. Finally, the two learned feature blocks are added pixel by pixel, and the channel attention value is obtained through the sigmoid activation function. The new feature map is obtained by multiplying the channel attention value and the input feature map ; and the formula is shown in 5-1: （5-1） wherein, is a sigmoid activation function mapping process, and the feature map obtained from the channel attention channel module is input into the spatial attention module, and the process is as follows: the feature map is is input into the spatial attention module, and the process is as follows: the feature map is is input into the spatial attention module, and the process is as follows: the feature map is is multiplied by the feature map ; the formula is shown in 5-2: （5-2） wherein, is a sigmoid activation function mapping process, is the above convolution processing; at this time, the feature map is the result of the hierarchical feature adjustment module; the feature map is superimposed and fused with the feature map to obtain a new feature map is sent to the BasicBlock module of the Layer4 layer of the ResNet18 network as input; on the other hand, the new feature map is sent to the hierarchical feature adjustment module to obtain a new feature map and a feature map is superimposed and fused to be sent to the BasicBlock module of the Layer5 layer of the ResNet18 network as input, to obtain the final feature map ; The feature map , the size is , the number of channels is 512, the number of channels is gradually reduced and the resolution is gradually improved by using a depth estimation decoder, and finally a depth estimation image with the same size as the original image and 3 channels is obtained; The specific expression is: The decoder will output the feature map from the encoder. After upsampling, the feature map size is doubled. Then, a convolution operation is used to reduce the number of channels to half, which is 256, to obtain the feature map. ; the encoder's 4th layer feature map and The images are stitched together according to the channel direction, and then upsampling, deconvolution, and skip connection operations are performed three times in the same way to obtain the image with the same dimensions as the original image. Feature maps of uniform size with 64 channels The final disparity map is then obtained through a fully connected layer, and the final depth estimation map is obtained by processing it with a sigmoid activation function; the layer fusion part of the decoder is the output of the third layer of the decoder. A new feature map is obtained through a hierarchical feature adjustment module that uses the same attention mechanism as the encoder. and the second layer output from the decoder New feature maps are obtained through the attention mechanism module. , feature map and Through hyperparameter weights The addition operation is then performed, followed by the addition of the feature map output from the first layer of the decoder. Add them together to perform feature fusion.

2. The method of claim 1, wherein, The depth encoder of step (1) is ResNet18.

3. The mine enhanced reality unmanned mining face depth estimation method of claim 1, wherein, The specific steps of step (3) are: (1) The object in the image is first subjected to Manhattan region detection to obtain the alignment normal , by calculating the total number of planes in the unmanned mining face, selecting N as the limit value, N greater than 10, selecting 50% of the planes as the plane of the estimated normal vector, N less than or equal to 10, selecting 70% of the planes as the plane of the estimated normal vector, estimating the plane normal prediction n, using Manhattan normal detection to detect the plane normal belonging to the main plane classification, then using the cosine similarity S(, ) to compare the difference between the estimated plane normal and each possible main direction , and select one with the best similarity, that is ​ ， where d is the total number of planes and n1> n2> n3... , ， wherein is the aligned normal, the cosine similarity is defined as ; (2) predicting a co-planar depth map The alignment normal calculated in the Manhattan region detection is used to derive its dissimilarity, and the three-dimensional coordinates of pixel p are Xp, assuming that the three-dimensional point is located in the plane with the normal as the alignment normal, then the distance from the plane to the origin is calculated as: ， Assuming that q is the adjacent pixel of p, the normal dissimilarity between them is defined as the Euclidean distance between two vectors: ， By and respectively denote the maximum and minimum dissimilarity between all adjacent pixels, Define the [·] operator, ， Then the dissimilarity of the distance from the plane to the origin is defined as: ， Then the geometric information dissimilarity combines the two kinds of dissimilarity into: ， The color information dissimilarity is calculated as: ， wherein , is an RGB color; and finally the color information difference and the geometry information difference are weighted to combine: ， Based on the difference, based on the graph is divided and filter out small area, to get the planar region, the planar region in three-dimensional point plane fitting, get the plane parameters by solving the least squares problem, ， wherein represents a three-dimensional point within the planar region; the inverse depth of pixel p is then computed by a plane fitting is: ， where K represents the camera intrinsic matrix, and then convert inverse depth to depth , using the depth obtained from the plane fitting ; (3) vector normal gradual change constraint A self-supervised constraint signal is established using the normal vector to supervise the corner, computing and the angle between the gradients represents the normal angle and then applying the L1 loss: ， ， wherein, and are predicted gradients estimated from the initial depth map and the predicted co-planar depth map, respectively.

Images