A monocular indoor depth estimation algorithm based on deep network and motion information

Abstract

The application discloses a monocular indoor depth estimation algorithm based on a deep network and motion information. First, in order to solve the problem of inaccurate feature extraction in a dynamic scene, a coding and decoding depth estimation network architecture is designed, a monocular indoor depth estimation algorithm combining a depth network and motion information is proposed according to the constraint relationship between front and rear frames. The core idea is to fully exploit the context information between adjacent frames by using the inter-frame geometric constraint relationship, track the motion area information between adjacent frames by using an optical flow estimation method, infer the depth information by using a motion vector (MV), and combine a hierarchical feature extraction network, so as to finally realize indoor static and dynamic scene depth estimation by using a feature extraction method combining a traditional geometric constraint and a depth neural network, and further improve the adaptability and robustness of the monocular indoor depth estimation algorithm.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of monocular indoor depth estimation algorithm, and particularly relates to a monocular indoor depth estimation algorithm based on a depth network and motion information. BACKGROUND

[0002] Monocular depth estimation has important applications in target recognition, scene understanding, navigation positioning and other fields. Traditional monocular depth estimation methods based on stereo vision have certain limitations on the scene. At present, the method of using a coding and decoding network to restore the depth information of an image has been widely applied. However, due to the underdetermination of monocular depth estimation itself, the low-quality of true value data and other problems, in the actual application scene, there are also problems such as motion interference, scene occlusion caused by object motion, optical distortion and missing area of material, which have a great influence on the accuracy of depth estimation, resulting in the loss of texture, edge and other detailed information of the network output depth information, greatly reducing the robustness of the existing algorithm. SUMMARY

[0003] The purpose of the present application is to overcome the defects of the prior art and provide a monocular indoor depth estimation algorithm based on a depth network and motion information.

[0004] The present application is implemented by the following technical solutions:

[0005] A monocular indoor depth estimation algorithm based on a depth network and motion information, specifically comprising the following steps:

[0006] (2) performing data preprocessing on the obtained video data;

[0007] (2) constructing a DenseNet selection feature extractor to continuously improve the accuracy;

[0008] (3) designing a hierarchical compression excitation ASPP network, using multiple parallel atrous convolutions with different sampling steps, further processing the features extracted under different sampling steps in separate branches, and then fusing each atrous convolution to form multi-scale features.

[0009] (4) for moving objects appearing in dynamic scenes, combining a double-discriminative mode motion estimation block to calculate the MV information, using a mask mechanism to shield the non-motion part, performing Gaussian denoising on the motion part, taking the motion points as depth clues to solve their depth, and realizing effective depth estimation of dynamic areas;

[0010] (5) constructing an energy function;

[0011] (6) realizing the depth information fusion of the ASPP network and the double-discriminative mode motion estimation, considering both static and dynamic scenes, and effectively realizing depth estimation in various indoor scenes.

[0012] The data preprocessing of the obtained video data in step (1) is specifically as follows: first, motion analysis is performed on the original video. The video collected by the imaging device is analyzed frame by frame, and the bad motion source is detected and recognized. The camera motion parameters are estimated by analyzing the space-time correspondence between consecutive frames. Second, outlier deletion. In order to remove the error caused by camera jitter in the calculation process, the second step mainly performs motion outlier detection and removal. Most outlier detectors consider two adjacent frames, and mark all displacements that do not conform to the general observed motion as outliers, which can be achieved by calculating the fitting error between the estimated camera model and the motion of objects in the video. The present application adopts the most commonly used random sample consensus (RANSAC) algorithm for outlier processing in the data preprocessing stage.

[0013] The construction of the selected feature extractor in step (2) is specifically as follows: in the classic computer vision tasks such as scene classification, target recognition detection, etc., the DenseNet network can be expanded to hundreds of layers, and the increase of the number of parameters does not cause the network to appear performance degradation or overfitting signs, and can continuously improve the accuracy, at least does not reduce the accuracy. Therefore, the DenseNet-121 network is selected as the feature extractor of the network architecture encoding end.

[0014] The design of the hierarchical compression excitation ASPP network in step (3) is specifically as follows:

[0015] (a) Receptive field and dilated convolution: it is essential to have a large receptive field for each output pixel, so that important information is not missed when making predictions. ASPP generates multi-scale features through multiple parallel dilated convolutions. For an input two-dimensional feature image x, after the dilated convolution operation, any point i of the output is as shown in formula (1):

[0016]

[0017] Where r represents the dilated rate of the dilated convolution, i.e. the sampling step; w represents the convolution kernel; w(n) represents that there are n parameters in the convolution kernel, and N represents the size of the convolution kernel.

[0018] The calculation of the receptive field R of the lth layer is as shown in formula (2):

[0019]

[0020] Where f l represents the size of the convolution kernel of the lth layer, represents the accumulation of convolution steps from the 1st layer to the lth layer. The size of the receptive field can be adjusted by changing the sampling step r. When the value of the step r is always 1, formula (2) becomes the formula for calculating the size of the receptive field of the standard convolution.

[0021] (b) H-CE module: This step uses the global average pooling technique, focusing on the whole image, to convert the feature map to 1x1xC size, so that the network can fully utilize the information sharing mechanism between channels. The calculation method of the global average pooling technique is shown in formula (3):

[0022]

[0023] where f k represents the output value of the kth convolution kernel after convolution conversion.

[0024] Secondly, the output image after compression operation is stimulated to capture the dependence between channels, so as to achieve the purpose of automatic adjustment of feature weight. The flattening processing of the tensor enhances the interactivity between channels, and the result after flattening processing is input to the fully connected layer of the excitation operation to obtain the final feature vector. Finally, the feature map F and the excitation vector E are weighted and fused by the features between channels, and the calculation formula is shown in formula (4), and the output Y of the H-CE module is finally obtained.

[0025]

[0026] where, represents element-wise multiplication operation, E k represents the excitation vector obtained by excitation operation in the kth channel.

[0027] (c) Constructing a hierarchical compression excitation ASPP structure block

[0028] A difference matrix is constructed to enable the network to obtain a global optimal solution. The specific steps are as follows: first, the cross likelihood ratio (CLR) clustering algorithm is used to cluster similar classes of features. Then, the weight coefficient w between adjacent two classes is calculated, and according to the class correspondence relationship of the clustering result, the weight coefficient w is written into the correlation matrix W in turn; from the global of the image, the weight coefficient η between the current class of the image and all classes is calculated, and the global matrix G η is obtained, as shown in formula (5). Finally, the global matrix G η is subtracted from the correlation matrix W to construct a difference matrix S, as shown in formula (6):

[0029]

[0030] S=G η -W (6)

[0031] In the formula, diag(g) represents a diagonal matrix, η represents the number of classes after clustering, and m represents the current class number (1, 2, 3, L, n).

[0032] According to the difference matrix of formula (6), the eigenvalues and the eigenvectors corresponding to the eigenvalues are calculated, the smallest eigenvalue is λ0, and the eigenvector corresponding to λ0 is S0. According to the size of the receptive field, the threshold is set to divide the network into layers, and with the continuous training of the network, the hierarchical design structure of the network is continuously optimized. The hierarchical design is constrained by the number of non-zero rows of the difference matrix S, and the similarity matrix and clustering problem are learned at the same time.

[0033] At the low, medium and high levels of the network, the hierarchical optimization problem of the network can be converted into solving the minimum value of formula (7):

[0034]

[0035] In the formula, F T represents the correlation matrix trained by the network.

[0036] After multiple iterative training of the network under the constraint of the eigenvector S0 corresponding to the minimum eigenvalue λ0, F T is continuously updated in a loop, and when the solved F T and the correlation matrix W reach the minimum distance, the correlation matrix F T at this moment is recorded. T When the correlation matrix F * enters the H-CE module, the module can adaptively and purposefully process useful features and suppress useless feature parts.

[0037] (d) Fusion of multi-scale feature information

[0038] The decoding end fuses local and global multi-scale feature information for the network through layer-by-layer upsampling, jump connection and other technologies, and calculates the final depth estimation value, thereby improving the accuracy of the depth estimation result.

[0039] (e) Construction of loss function

[0040] Through a mask matrix, each pixel value in the image is recalculated, only the loss on the effective points is calculated, and the pixels of the invalid points are ignored. The scale-invariant error loss function proposed by Eigen et al. is adopted as the loss function of the network of the present application. As shown in formula (8):

[0041]

[0042] Wherein, y is the depth map predicted by the network; y *GT is data; the input and output image size of the network in this paper is 640*480; y i and y * i is the pixel value of each point in the image, the output pixel value constitutes a 640*480 matrix expression form; n is the number of pixel points of the whole image; lambda is a scale factor, which is set to 0.5.

[0043] The motion estimation under the double discrimination mode described in step (4) infers depth, and the specific process is as follows:

[0044] First, the classical diamond search method (Diamond Search, DS) is used to perform block matching on the front and rear frames of the video to find the mainstream motion vector (Mainstream Motion Vector, MMV) of the moving object in the image; then, the motion vector obtained by the DS is used to constrain the POFA iterative calculation, effectively excluding the brightness interference in the pixel block, and greatly reducing the time of the original POFA iteration to obtain the optical flow information. After extracting the MV under the double discrimination mode, the mask is used to extract the motion region to be processed.

[0045] Step (5) builds an energy function for subsequent processing of data.

[0046] Step (6) fuses depth information and optimizes the depth result.

[0047] The advantages of the present application are that the present application combines the feature extraction method of traditional geometric constraints and depth neural network to solve the problem of inaccurate feature extraction caused by the uncertainty of static and dynamic in indoor scenes;

[0048] When the indoor scene has motion conditions such as objects and people, the present application not only extracts the spatial dimension information of the image by using CNN, but also pays attention to the feature information in the time dimension, and more accurately extracts the motion features.

[0049] The present application fully excavates the context information between adjacent frames by using the inter-frame geometric constraint relationship, and tracks the motion region information between adjacent frames by using the optical flow estimation method, and infers the depth information by using the motion vector (Motion Vector, MV), and finally realizes the depth estimation of indoor static and dynamic scenes by combining the hierarchical feature extraction network and the feature extraction method of traditional geometric constraints and depth neural network, and further improves the adaptability and robustness of the monocular indoor depth estimation algorithm. BRIEF DESCRIPTION OF DRAWINGS

[0050] Figure 1 The present application is a method flowchart;

[0051] Figure 2 It is a main flowchart of image stabilization operation.

[0052] Figure 3 Execution diagram for the DS algorithm;

[0053] Figure 4 This describes the implementation process of a network under ASPP dense block for hierarchical compression excitation. Detailed Implementation

[0054] The present invention will be further described below with reference to the accompanying drawings and examples.

[0055] This example uses a handheld data acquisition device to demonstrate how a monocular indoor depth estimation algorithm based on deep networks and motion information is implemented, following the general process: Figure 1 As shown:

[0056] 1. Use a camera to acquire video streams of the indoor environment and perform data preprocessing.

[0057] 2. Construct a selection feature extractor.

[0058] 3. Design a hierarchical compression-excited ASPP network.

[0059] 4. Motion estimation inference depth under dual-discrimination mode.

[0060] 5. Construct the energy function.

[0061] 6. Integrate depth information to optimize depth results.

[0062] In summary, when estimating the depth of indoor objects, the data acquired by the camera is combined with the device's motion information through an ASPP codec network to estimate the depth information of objects in the environment, and the depth estimation results from both are then fused. The final reconstructed data more closely matches human perception of the actual scene, achieving relatively ideal results in restoring depth information in both static and dynamic scenes.

[0063] The specific process of this example is as follows:

[0064] Step 1: As Figure 2 As shown, motion analysis is performed on the original video. Using the video acquired by the imaging device, frame-by-frame analysis is performed to detect and identify unwanted motion sources. Camera motion parameters are estimated by analyzing the spatiotemporal correspondence between consecutive frames. Since the depth estimation task in this paper is performed in an indoor scene, and the scene has uncertainties in both static and dynamic attributes, the acquired data is represented as follows to account for all possible scenarios:

[0065] I(x,t+1)=I(x,t)+J(x,t)+M(x,t)

[0066] Where I(x, t+1) represents the (t+1)th frame signal, I(x, t) represents the tth frame signal, J(x, t) represents the camera shake signal, and M(x, t) represents the motion signal existing in the scene.

[0067] The pixels or pixel blocks in the current frame are matched with the pixels or pixel blocks in the next frame using a similarity measure or a distance measure. All the actions occurring in the video are mastered, thereby better completing the motion estimation task. Subsequently, motion outliers are mainly detected and removed. Most outlier detectors consider two adjacent frames and mark all displacements that do not conform to the general observed motion as outliers, which can be achieved by calculating the fitting error between the estimated camera model and the motion of objects in the video. Finally, the relationship between adjacent frames of the video stream is compared and constrained, the motion points are tracked quickly and accurately, and the motion compensated result is obtained as the input of the depth estimation network of the application.

[0068] Step 2: Obtain the RGB data of the image, and then select DenseNet as a good feature extractor for various convolution-based computer vision tasks to learn a more compact model.

[0069] Step 3: Since the value of the neuron receptive field is larger, it means that the neuron at the point has a larger mapping range on the input image, and at the same time, it means that the point may contain more global and higher semantic level features; on the contrary, the smaller the value is, the more the point contains features that tend to be local and detailed. Although the ordinary convolution kernel expands the receptive field when down-sampling the image, it causes the reduction of the spatial resolution. The dilated convolution can increase the receptive field while keeping the resolution of the feature map unchanged. Therefore, multiple dilated convolutions in the parallel structure of the ASPP network are used to generate multi-scale features. For an input two-dimensional feature image x, after the dilated convolution operation, any point i of the output is as shown in where r represents the dilated rate of the dilated convolution, i.e., the sampling step; w represents the convolution kernel; w(n) represents that there are n parameters in the convolution kernel, and N represents the size of the convolution kernel.

[0070] The calculation of the receptive field R of the lth layer is as shown in where f l represents the size of the convolution kernel of the lth layer, The cumulative convolution step from the first layer to the lth layer is represented. The size of the receptive field can be adjusted by changing the sampling step r. Then the network is layered according to the obtained receptive field, and the abstraction level of the current layer of CNN is determined. When the feature map extracted by the feature extractor is convolved by the convolution kernel with different sampling steps, the feature weight obtained after the convolution kernel is recalibrated. Using multiple parallel dilated convolutions with different sampling steps, the features extracted under different sampling steps are further processed in separate branches, and then fused to form multi-scale features. According to the setting of the threshold size and parameters of the receptive field, the compressed excitation ASPP structure block is divided into three layers: low level, middle level and high level. As shown in Figure 4 , the pixel point i feature information at the current position is extracted using the divided levels, respectively denoted as L(i), M(i) and H(i). The multi-level feature information extracted in the same pixel point is effectively fused. The bilinear feature X(i) of the low, middle and high three different levels is obtained. The sum of the bilinear feature X(i) obtained at each pixel point is obtained to obtain the bilinear feature information of the entire image.

[0071] Step 4: As shown in Figure 3 , the process quantity of the moving object is analyzed by using the double discrimination mode, that is, the classical diamond search method (Diamond Search, DS) is used to perform block matching on the front and rear frames of the video to find the mainstream motion vector (Mainstream Motion Vector, MMV) of the moving object in the image; then the motion vector after the POFA iterative calculation is constrained according to the MMV obtained by the DS, effectively excluding the brightness interference inside the pixel block, while greatly reducing the time of the original POFA iteration to obtain the optical flow information. After extracting the MV in the double discrimination mode, the mask is used to extract the motion region to be processed.

[0072] Step 5: Construct the corresponding energy function.

[0073] Step 6: The estimation of depth in step 3 does not fully consider the dynamic scene, so there is a problem of insufficient robustness in monocular indoor depth estimation of dynamic scenes. In step 4, the double discrimination mode motion estimation block calculates its MV information, uses the mask mechanism to shield the non-motion part, performs Gaussian denoising on the motion part, and takes the motion point as the depth clue to solve its depth, thereby realizing effective estimation of the depth of the dynamic region. Therefore, the depth information of the detected motion region is replaced with the depth of the region position estimated by the original ASPP algorithm, realizing depth information fusion. This depth information fusion method considers both static and dynamic scenes, effectively realizing depth estimation in various indoor scenes.

[0074] The application fully mines context information between adjacent frames by using inter-frame geometric constraint relationship, tracks motion region information between adjacent frames by using optical flow estimation method, infers depth information by using motion vector (MV), and finally realizes indoor static and dynamic scene depth estimation by combining hierarchical feature extraction network and the feature extraction method of joint traditional geometric constraint and depth neural network, thereby further improving the adaptability and robustness of monocular indoor depth estimation algorithm.

[0075] The application preprocesses acquired indoor scene video data by using the feature extraction method of joint traditional geometric constraint and depth neural network. Since the situation that handheld device instability affects data quality objectively exists, a camera deblurring module is constructed to eliminate the influence of external factors on network output. Secondly, a motion estimation module is constructed to identify and process dynamic objects and calculate depth according to motion information. The depth of the preprocessed data is estimated and sent to the motion estimation module, and geometric constraint between adjacent frames, double-discrimination mode motion analysis, mask mechanism, etc. are used to identify and process static and dynamic objects. Finally, the information output by two paths is fused to optimize the output result, so that the network adaptively processes the features extracted from static and dynamic scenes according to the input data to obtain the depth information finally output by the network.

Claims

1. A monocular indoor depth estimation algorithm based on deep network and motion information, characterized in that: Specifically comprising the following steps: (1) obtaining video data of an indoor environment, and performing data preprocessing on the obtained video data; (2) inputting the data after the data preprocessing into a feature extractor; (3) inputting the data after the processing of the feature extractor into a hierarchical compressed excitation ASPP network, using multiple parallel hole convolutions with different sampling steps to further process the features extracted under different sampling steps in separate branches, and then fusing the hole convolutions to form multi-scale features; (4) for a moving object appearing in a dynamic scene, combining a double-discriminative mode motion estimation block to calculate MV information of the moving object, shielding the non-motion part by using a mask mechanism, performing Gaussian denoising on the motion part, taking the motion point as a depth clue to solve the depth, and realizing effective depth estimation of the dynamic area; (5) constructing an energy function; (6) realizing depth information fusion of the ASPP network and the double-discriminative mode motion estimation, and realizing depth estimation in various indoor scenes; The data preprocessing on the obtained video data in step (1) is specifically as follows: Firstly, motion analysis is performed on the obtained video data: the video data collected by the imaging device is analyzed frame by frame, and the bad motion source is detected and identified, and the imaging device motion parameters are estimated by analyzing the space-time correspondence between the continuous frames; Secondly, outlier deletion: the outlier detector considers two adjacent frames, and marks all displacements that do not conform to the observed motion as outliers, and removes the outliers by using a random sampling consensus algorithm; The specific process of realizing effective depth estimation of the dynamic area in step (4) is as follows: Firstly, the diamond search method is used to perform block matching on the front and rear frames of the video to find the main motion vector of the moving object in the image; and then, the motion vector after the POFA iterative calculation is constrained by the MMV obtained by the DS, and the motion area to be processed is extracted by using a mask after the MV is extracted in the double-discriminative mode. 2.The monocular indoor depth estimation algorithm based on deep network and motion information according to claim 1, characterized in that: The feature extractor in step (2) selects a DenseNet-121 network as the feature extractor of the network architecture encoding end. 3.The monocular depth estimation algorithm based on deep network and motion information according to claim 2, characterized in that: The hierarchical compressed excitation ASPP network in step (3) uses multiple parallel hole convolutions with different sampling steps to further process the features extracted under different sampling steps in separate branches, and then fuses the hole convolutions to form multi-scale features, which is specifically as follows: (a) receptive field and dilated convolution: the ASPP network generates multi-scale features through multiple parallel dilated convolutions; for an input two-dimensional feature image In terms of the output of any point As shown in equation (1): （1） wherein, denotes the dilation rate, i.e. the sampling step, of the atrous convolution; denotes the convolution kernel; denotes that there are parameters in the convolution kernel, denotes the convolution kernel size; The first layer receptive field The calculation is shown in equation (2): （2） wherein, represents the convolution kernel size of the i-th layer, represents the convolution kernel size of the i-th layer, represents the cumulative of the convolution step size from the 1st layer to the i-th layer; represents the cumulative of the convolution step size from the 1st layer to the i-th layer; (b) H-CE module: The global average pooling technique is adopted to convert the feature map into a size that allows the network to fully utilize the information sharing mechanism between channels; the calculation method of the global average pooling technique is shown in equation (3): （3） wherein, represents the output value of the convolution kernel after the convolution conversion; is the feature vector after global pooling, W represents the width of the input feature map, H represents the height of the input feature map, and C represents the channel number of the input feature map.​ The excitation operation is performed on the output image to capture the dependence between channels and automatically adjust the feature weight; the flattening processing of the tensor enhances the interaction between channels, and the result after the flattening processing is input into the fully connected layer of the excitation operation to obtain the final feature vector; The feature map With the excitation vector The feature between each channel is weighted and fused, and the calculation formula is shown as formula (4), and finally the output of H-CE module is obtained : （4） wherein, represents an element-wise multiplication operation, represents an excitation vector obtained under the th channel by the excitation operation; (c) constructing a hierarchical compressed excitation ASPP structure block Constructing a difference matrix to make the network obtain the global optimal solution: first, using the cross log likelihood ratio clustering algorithm to cluster the features into similar classes, then calculating the weight coefficients between the adjacent two classes , according to the category corresponding relationship of the clustering results, sequentially write the weight coefficients into the correlation matrix ; then from the global of the image, calculate the weight coefficients between the current class to which the image belongs and all classes , obtain the global matrix , as shown in formula (5); finally, subtract the global matrix from the correlation matrix to construct the difference matrix , as shown in formula (6): （5） （6） In the formula, denotes a diagonal matrix, η denotes the number of classes after clustering, denotes the current class number ; According to the difference matrix of formula (6), the characteristic value and the characteristic vector corresponding to the characteristic value are calculated, the smallest characteristic value is , The corresponding characteristic vector is The threshold value is set according to the size of the receptive field to stratify the network, and with the continuous training of the network, the hierarchical design structure of the network is continuously optimized, and the hierarchical design is constrained by the number of non-zero rows of the difference matrix . The hierarchical optimization problem of the network is converted into solving the minimum value of formula (7) at the low, medium and high levels of the network: （7） In the formula, represents the correlation matrix trained through the network; The minimum eigenvalue The corresponding eigenvector After multiple iterations of training the network under constraints, Constantly loop updates, when the solution The minimum distance between the correlation matrix The current correlation matrix is recorded ; Correlation matrix When entering the H-CE module, the H-CE module processes useful features and suppresses the useless feature part; (d) fusing multi-scale feature information The decoding end fuses local and overall multi-scale feature information for the network through layer-by-layer up-sampling and skip connection technology, and calculates a final depth estimation value; (e) constructing a loss function Through a mask matrix, each pixel value in the image is recalculated, only the loss on the effective points is calculated, and the pixels of the invalid points are ignored; a scale-invariant error loss function is used as the loss function of the network, as shown in equation (8): （8） wherein, is the depth map predicted by the network; is the GT data; and is the pixel value of each point in the image; is the number of pixels in the entire image; is the scale factor. 4.The monocular indoor depth estimation algorithm based on deep network and motion information according to claim 1, characterized in that: The depth information fusion of the ASPP network and the double-discriminative mode motion estimation is realized in step (6), and the depth estimation in various indoor scenes is realized, which is specifically as follows: for the moving objects in the dynamic scene, the MV information is calculated by combining the double-discriminative mode motion estimation block, the non-motion part is shielded by using the mask mechanism, the motion part is subjected to Gaussian denoising, the motion points are taken as depth clues to solve the depth, the depth of the dynamic area is effectively estimated, and then the static depth information obtained by using the ASPP network is combined to realize the consideration of static and dynamic scenes, and the depth estimation in various indoor scenes is realized.

Images