A depth map and posture optimization method and system

Abstract

A depth map and posture optimization method and system, comprising the following steps: preparing a data sequence; inputting image data into an extractor to extract image features and context information respectively; obtaining an initial depth map and posture, as well as photometric measurement cost from image features; inputting the photometric measurement cost into a grouped dual enhancement module to obtain an enhancement cost; inputting the enhancement cost, the initial depth map and posture, and the context information into a multi-level perceptual optimizer to obtain an accurate depth map and posture after iterative optimization; calculating errors based on prediction results and true values, and iteratively optimizing the entire deep neural network until the error converges.

Description

Technical Field

[0001] The present invention belongs to the field of computer vision and is a method and system for optimizing a depth map and a posture based on a grouped dual enhancement module and a multi-level perceptual optimizer. Background Art

[0002] Structure from Motion (SfM) aims to accurately calculate the depth map and relative pose of a set of camera images, so as to model the 3D structure of the corresponding scene. Traditional SfM optimization solutions generally rely on bundle adjustment (BA). However, traditional BA methods require good image key points and feature matching between images, which results in their performance being severely limited by lighting conditions and structural textures.

[0003] In order to solve these problems, some methods and systems that apply deep learning to solve SfM optimization problems have emerged in recent years. Early methods directly used neural networks to regress input images to predict results, but the effect was not ideal due to the lack of basic geometric constraints. Recent methods try to combine the advantages of deep learning with traditional optimization methods. These methods can be mainly divided into two categories: indirect update methods and direct update methods. However, indirect update methods are only applicable to cases where the objective function is relatively simple, and it is difficult to make good use of the advantages of neural networks. On the other hand, existing direct update methods rely on volume costs or traditional photometric costs. However, the former has too much computational complexity, and the latter ignores spatial information, making the optimization process easily affected by noise and making the optimization unstable. In addition, due to computational resource constraints, the optimizer of the existing system only uses convolution operations with a small receptive field, which makes the optimizer poor in capturing the features of large targets, resulting in incomplete prediction results. Summary of the invention

[0004] The present invention aims to solve the problems in the existing methods that the optimization process is unstable and the optimizer has poor perception ability of large targets.

[0005] The technical solution adopted by the present invention is: a depth map and posture optimization method, comprising the following steps:

[0006] S1, prepare a sequence including RGB image data, real depth map data and real camera pose data, and extract a target image It, a plurality of adjacent images Iri of It, the real pose of It relative to Iri, and the real depth map of It;

[0007] S2, extract features from It and Iri to obtain image features Ft and Fri, perform depth extraction on It to obtain depth context information as the depth hidden state, splice It and Iri and perform pose extraction on the spliced ​​image to obtain pose context information as the pose hidden state;

[0008] S3, the image feature Ft is passed through the depth map network to obtain the initial depth map, and Ft and Fri are concatenated and input into the pose network to obtain the initial pose;

[0009] S4, calculating the photometric cost in the feature space according to the initial depth map and the initial pose, and performing cost enhancement on the photometric cost to obtain a depth enhancement cost and a pose enhancement cost;

[0010] Input the deep hidden state, the depth enhancement cost and the initial depth map into the deep multi-level perceptual optimizer to obtain a new initial depth map, use the new initial depth map and the initial pose to calculate the new photometric cost, use the new photometric cost to update the depth enhancement cost and then update the initial depth map, iteratively update the initial depth map m times, where m is a positive integer;

[0011] Input the pose hidden state, pose enhancement cost and initial pose into the pose multi-level perceptual optimizer to get a new initial pose, use the initial depth map and the new initial pose to calculate the new photometric cost, use the new photometric cost to update the pose enhancement cost and then update the initial pose, iterate and update the initial pose n times, n is a positive integer;

[0012] S5, repeat S4 k times to obtain the final depth map and pose, where k is a positive integer;

[0013] S6, constructing a depth map loss and a pose loss using the loss function, obtaining a total loss based on the depth map loss and the pose loss, and updating the weights of the depth multi-level perception optimizer and the pose multi-level perception optimizer using the total loss;

[0014] S7, repeat S1 to S6 until the total loss converges, obtain the training model, input the image to be tested into the training model, and obtain the predicted depth map and pose.

[0015] Furthermore, in S4, the photometric measurement cost is calculated as follows: according to the initial depth map D, the initial pose Pi and the camera intrinsic parameter K, the reprojected two-dimensional coordinates on Fri are calculated, and then Fri is sampled according to the obtained two-dimensional coordinates, and the L2 norm of the sampled feature map and Ft is calculated to obtain the photometric measurement cost.

[0016] Furthermore, in S4, the calculation methods of the depth enhancement cost and the pose enhancement cost are: encoding the photometric measurement cost, grouping by channel, performing global average pooling on each group, obtaining a statistical vector of each group, and then using the statistical vector to perform spatial enhancement and channel enhancement on the cost of the group to which it belongs, and splicing the results of each group by channel to obtain the total enhancement cost.

[0017] Furthermore, the specific steps for obtaining a new initial depth map or initial pose in S4 are as follows: first, the depth enhancement cost and the initial depth map or the pose enhancement cost and the initial depth map are projected into the feature space, and then the optimized features and the depth hidden state or the optimized features and the pose hidden state are input into the depth multi-level perceptual optimizer or the pose multi-level perceptual optimizer to obtain a new depth hidden state or a new pose hidden state, and then the depth map network or the pose network is used to obtain a depth map increment or a pose increment according to the new depth hidden state or the new pose hidden state, and the increment is added to the initial depth map or the initial pose to update the initial depth map or the initial pose.

[0018] Furthermore, in S6, the calculation formula of the loss function L is:

[0019] L=L psoe +L depth ,

[0020]

[0021]

[0022] Among them, D * and Pi * represents the true depth and pose, γ represents the attenuation coefficient, k represents the total number of iterations, t represents the tth iteration, D t and Pi t Represents the predicted depth and pose, D(x) represents the pixel coordinates in D, K is the camera intrinsic parameter, proj is the projection function, and Iri is the pixel value of the image sampled according to the coordinates.

[0023] Furthermore, the deep multi-level perception optimizer and the pose multi-level perception optimizer are both composed of dilated convolutions and gated recurrent units with different expansion coefficients.

[0024] Furthermore, the calculation method of the spatial enhancement operation is: perform an inner product on the statistical vector and the cost element vector one by one to obtain the spatial enhancement weight, then scale and transform the spatial enhancement weight, and then multiply the cost by the sigmoid function to obtain the spatial enhancement cost. The calculation method of the channel enhancement operation is: perform a one-dimensional convolution operation on the statistical variable to obtain the channel weight vector b, and then multiply b by the spatial enhancement cost by the sigmoid function to obtain the double enhancement cost.

[0025] Furthermore, the multi-level perception structure is composed of dilated convolutions and gated recurrent units with different expansion coefficients.

[0026] A depth map and posture optimization system comprises an image feature extraction module, a depth context extraction module, a posture context extraction module, a depth grouping double enhancement module, a posture grouping double enhancement module, a depth multi-level perception optimizer module and a posture multi-level perception optimizer module; the image feature extraction module is used to extract image features, the context information extraction module is used to extract context information as a hidden state, the grouping double enhancement module enhances the cost, the depth multi-level perception optimizer module is used to update the depth map, and the posture multi-level perception optimizer module is used to update the posture.

[0027] Furthermore, the image feature extraction module and the context information extraction module use the residual network structure, the deep graph network and the pose network adopt the two-dimensional convolutional neural network structure, the deep multi-level perceptual optimizer and the pose multi-level perceptual optimizer are both composed of dilated convolutions and gated recurrent units with different expansion coefficients, the output layer of the deep multi-level perceptual optimizer contains the deep graph network, and the output layer of the pose multi-level perceptual optimizer contains the pose network.

[0028] Compared with the prior art, the present invention has the following advantages and effects:

[0029] 1. The grouped dual enhancement module involved in the model of the present invention can effectively enhance the spatial semantic information of the photometric measurement cost and the interdependence between channels, thereby reducing the interference of potential noise on the subsequent optimization process and making the prediction results clearer and more accurate.

[0030] 2. The multi-level perception optimizer involved in the model of the present invention improves the optimizer's ability to capture the features of large targets in the optimization features through an efficient multi-level perception structure, thereby making the prediction results of large targets more complete and accurate. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] The drawings constituting a part of this application are used to provide a further understanding of the present invention. The illustrative embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute improper limitations on the present invention.

[0032] Figure 1 It is a schematic diagram of the model of the present invention;

[0033] Figure 2 It is a structural diagram of the grouped double-enhancement module;

[0034] Figure 3 The implementation details of the grouped double enhancement module for each group;

[0035] Figure 4 This is the core structure diagram of the multi-level perceptual optimizer;

[0036] Figure 5 The ablation experiment results of the grouped dual enhancement module and the multi-level perceptual optimizer are shown in Figure 2.

[0037] Figure 6 The figure shows the comparison between the depth estimation results of the present invention and other methods. DETAILED DESCRIPTION

[0038] In order to make the purpose, technical solution and advantages of the present invention clearer, the present invention will be further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0039] Embodiment 1:

[0040] like Figure 1 As shown, a depth map and posture optimization method and system based on a grouped dual enhancement module and a multi-level perceptual optimizer includes the following steps.

[0041] Step 1, data preparation, is divided into two steps:

[0042] 1.1. Prepare a sequence including RGB image data, real depth map data and real camera pose data, such as the KITTI unmanned driving dataset, which contains a sequence of image data, real depth map data and real camera pose data acquired by a vehicle camera and various sensors.

[0043] 1.2, extract the target image It, its adjacent image Iri (i ranges from 1 to 2), the true position Pi of It relative to Iri, and the true depth map D of It from the prepared data of 1.1. Preferably, the size of the image is preprocessed to be 320 pixels high and 960 pixels wide.

[0044] Step 2: Image feature extraction and context information extraction: Use the feature extraction module and the context extraction module to extract image features and context features from It and Iri in step 1.2.

[0045] 2.1, Extraction of image features. Preferably, the structure of the feature extraction module adopts the existing residual network 18 designed for image classification tasks, and the size of the feature map output by it is one eighth of the original image size, and the number of channels is 128. Thus, the feature maps Ft and Fri of It and Iri are obtained respectively.

[0046] 2.2, extraction of context information. Preferably, the structure of the context extraction module adopts the existing residual network 18 designed for image classification tasks. Among them, the input of the deep context extraction module is It, and the input of the pose context extraction module is the concatenation of It and Iri in the channel dimension. The size of the feature map output by the module is one eighth of the original image size, and the number of channels is 160. In this way, the deep context information and pose context information are obtained as the initial deep hidden state and pose hidden state, respectively.

[0047] Step 3, construction and enhancement of photometric cost.

[0048] 3.1, Construction of photometric cost. In the first iteration, the feature map Ft obtained in step 2.1 is input into the depth map network to obtain the initial depth map D, and Ft and Fri are concatenated in the channel dimension and input into the pose network to obtain the initial pose Pi. Then, the reprojected 2D coordinates on Fri are calculated based on the camera intrinsic parameter K, and Fri is sampled based on the obtained 2D coordinates. Finally, the sampled feature map and Ft are used to calculate the L2 norm to obtain the photometric cost.

[0049] 3.2, Enhancement of photometric cost. The cost is enhanced using a grouped double enhancement module, and its overall structure is shown in the figure below. Figure 2 As shown in , the specific operation is as follows: First, encode the photometric cost. Preferably, the cost encoder uses convolutional layers with kernel sizes of 1 and 3, respectively, and the output size remains unchanged. Then divide the cost into 32 groups by channel. Then, the operation for each group is as follows Figure 3 As shown in the figure, we first perform global average pooling to obtain a statistical vector, and then use the statistical vector to perform spatial enhancement and channel enhancement on the cost of the group. The calculation method for spatial enhancement is: do the inner product of the statistical vector and the cost to obtain the spatial enhancement weight, then scale and transform the spatial enhancement weight, and then multiply it with the cost through the sigmoid function to obtain the spatial enhancement cost; the calculation method for channel enhancement is: perform a one-dimensional convolution operation on the statistical variable to obtain the channel weight vector b, and then multiply the weight vector b with the spatial enhancement cost through the sigmoid function to obtain the double enhancement cost. Finally, the results of each group are spliced ​​by channel to obtain the total enhancement cost.

[0050] Step 4, iteratively update the depth map and pose. Use the multi-level perceptual optimizer to update the depth map and pose. The core structure of the multi-level perceptual optimizer is as follows: Figure 4 As shown, both the deep multi-level perceptual optimizer and the pose multi-level perceptual optimizer are composed of multiple dilated convolutions and gated recurrent units with different expansion coefficients. Preferably, dilated convolutions with expansion coefficients of 1 and 4 are used; the output layer of the deep multi-level perceptual optimizer contains a depth map network, and the output layer of the pose multi-level perceptual optimizer contains a pose network. First, the total enhancement cost and the initial depth map are projected into the feature space and concatenated as optimized features, and then the optimized features and hidden states are input into the multi-level perceptual structure to obtain a new hidden state. Then, the depth map network or the pose network is used to obtain a depth map increment or a pose increment according to the new hidden state. Finally, the increment is added to the initial depth map or initial pose to obtain an updated initial depth map or initial pose. The specific iterative update process is as follows:

[0051] 4.1, Update the depth map. Input the depth hidden state, enhancement cost and initial depth map into the deep multi-level perceptual optimizer to obtain an updated depth map and a new depth hidden state, which are used as the initial depth map and depth hidden state of the next iteration respectively. Repeat this step m times, where m is a positive integer, preferably 4.

[0052] 4.2, Update pose. Input the pose hidden state, enhancement cost and initial pose into the pose multi-level perceptual optimizer to obtain the updated pose and new pose hidden state, which are used as the initial pose and pose hidden state of the next iteration respectively. Repeat this step n times, where n is a positive integer, preferably 4.

[0053] 4.3, repeat steps 4.1 to 4.2 k times, where k is a positive integer, preferably 10. The final predicted depth map and pose are obtained.

[0054] Step 5: Loss function calculation and network parameter update. Using the depth map and pose obtained in step 4, as well as the true values ​​of the depth map and pose, the loss is calculated according to the following loss function:

[0055] L=L psoe +L depth ,

[0056]

[0057]

[0058] Among them, D * and Pi * represents the true depth and pose, γ represents the attenuation coefficient, k represents the total number of iterations, and t represents the tth iteration. Preferably, k is 3 and γ is 0.85. After the above loss calculation, back propagation is performed to update the network parameters.

[0059] Step 6: Repeat steps 1 to 5 until the loss function converges. For example, iterate 40 cycles on the KITTI dataset to obtain a depth map and pose optimization model based on a grouped dual enhancement module and a multi-level perceptual optimizer. Figure 5 The ablation experiment result diagram of the present invention is shown in FIG. 1 , and the bold values ​​represent the best results. It can be seen that each module of the present invention effectively improves the performance of the baseline, and when all modules are combined, the best performance can be achieved. Figure 6 The depth estimation results of the present invention are compared with those of the existing advanced methods. The bold values ​​represent the best results. It can be seen that the accuracy of the present invention exceeds that of the existing algorithms.

[0060] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A depth map and pose optimization method, characterized in that: The following steps are involved: S1, prepare a sequence including RGB image data, real depth map data and real camera pose data, and extract a target image It, a plurality of adjacent images Iri of It, the real pose of It relative to Iri, and the real depth map of It; S2, extract features from It and Iri to obtain image features Ft and Fri, perform depth extraction on It to obtain depth context information as the depth hidden state, splice It and Iri and perform pose extraction on the spliced ​​image to obtain pose context information as the pose hidden state; S3, the image feature Ft is passed through the depth map network to obtain the initial depth map, and Ft and Fri are concatenated and input into the pose network to obtain the initial pose; S4, calculating the photometric cost in the feature space according to the initial depth map and the initial pose, and performing cost enhancement on the photometric cost to obtain a depth enhancement cost and a pose enhancement cost; Input the deep hidden state, the depth enhancement cost and the initial depth map into the deep multi-level perceptual optimizer to obtain a new initial depth map, use the new initial depth map and the initial pose to calculate the new photometric cost, use the new photometric cost to update the depth enhancement cost and then update the initial depth map, iteratively update the initial depth map m times, where m is a positive integer; Input the pose hidden state, pose enhancement cost and initial pose into the pose multi-level perceptual optimizer to get a new initial pose, use the initial depth map and the new initial pose to calculate the new photometric cost, use the new photometric cost to update the pose enhancement cost and then update the initial pose, iterate and update the initial pose n times, n is a positive integer; The calculation method of the photometric cost is as follows: according to the initial depth map D, the initial pose Pi and the camera intrinsic parameter K, the reprojected two-dimensional coordinates on Fri are calculated, and then Fri is sampled according to the obtained two-dimensional coordinates, and the L2 norm of the sampled feature map and Ft is calculated to obtain the photometric cost; The calculation methods of depth enhancement cost and pose enhancement cost are both as follows: encode the photometric cost, group them by channel, perform global average pooling on each group, obtain the statistical vector of each group, then use the statistical vector to perform spatial enhancement and channel enhancement on the cost of the group to which it belongs, and concatenate the results of each group by channel to obtain the total enhancement cost; S5, repeat S4 k times to obtain the final depth map and pose, where k is a positive integer; S6, constructing a depth map loss and a pose loss using the loss function, obtaining a total loss based on the depth map loss and the pose loss, and updating the weights of the depth multi-level perception optimizer and the pose multi-level perception optimizer using the total loss; The calculation formula of the loss function L is: L=L psoe +L depth , Among them, D * and Pi * represents the true depth and pose, γ represents the attenuation coefficient, k represents the total number of iterations, t represents the tth iteration, and D t and Pi t represents the predicted depth and pose, D(x) represents the pixel coordinates in D, K is the camera intrinsic parameter, proj is the projection function, and Iri is the pixel value of the image sampled according to the coordinates; S7, repeat S1 to S6 until the total loss converges, obtain the training model, input the image to be tested into the training model, and obtain the predicted depth map and pose.

2. The depth map and pose optimization method according to claim 1, characterized in that: The specific steps of obtaining the new initial depth map or initial pose in S4 are: first, project the depth enhancement cost and the initial depth map or the pose enhancement cost and the initial depth map into the feature space, splice them as optimized features, and then input the optimized features and the depth hidden state or the optimized features and the pose hidden state into the depth multi-level perception optimizer or the pose multi-level perception optimizer to obtain a new depth hidden state or a new pose hidden state, then use the depth map network or the pose network to obtain a depth map increment or a pose increment according to the new depth hidden state or the new pose hidden state, and add the increment to the initial depth map or the initial pose to update the initial depth map or the initial pose.

3. The depth map and pose optimization method according to claim 1, characterized in that: The depth multi-level perception optimizer and the posture multi-level perception optimizer are both composed of dilated convolutions and gated recurrent units with different expansion coefficients.

4. The depth map and pose optimization method according to claim 1, characterized in that: The calculation method of the spatial enhancement is: performing an inner product on the statistical vector and the cost to obtain the spatial enhancement weight, then scaling and transforming the spatial enhancement weight, and then multiplying the cost by the sigmoid function to obtain the spatial enhancement cost; the calculation method of the channel enhancement is: performing a one-dimensional convolution operation on the statistical variable to obtain the channel weight vector b, and then multiplying the weight vector b by the spatial enhancement cost by the sigmoid function to obtain the double enhancement cost.

5. A depth map and pose optimization system, characterized in that: It includes an image feature extraction module, a depth context extraction module, a posture context extraction module, a depth grouping double enhancement module, a posture grouping double enhancement module, a depth multi-level perception optimizer module and a posture multi-level perception optimizer module; The image feature extraction module is used to extract image features, the context information extraction module is used to extract context information as hidden state, the depth grouping double enhancement module enhances the depth cost, the depth multi-level perceptual optimizer module is used to update the depth map, the pose grouping double enhancement module enhances the pose cost, and the pose multi-level perceptual optimizer module is used to update the pose.

6. The depth map and pose optimization system according to claim 5, characterized in that: The image feature extraction module and the context information extraction module use the residual network structure, the deep graph network and the pose network adopt the two-dimensional convolutional neural network structure, the deep multi-level perceptual optimizer and the pose multi-level perceptual optimizer are both composed of dilated convolutions and gated recurrent units with different expansion coefficients, the output layer of the deep multi-level perceptual optimizer contains the deep graph network, and the output layer of the pose multi-level perceptual optimizer contains the pose network.

Images