Depth and pose joint estimation method and medium based on spatial information aggregation

Abstract

The present invention relates to a method and medium for joint depth and pose estimation based on spatial information aggregation. The method comprises the following steps: obtaining a monocular image sequence, inputting a joint depth and pose estimation model, and outputting a depth prediction map and camera pose prediction results. The model processes the input as follows: obtaining a depth spatial prior and a semantic feature set; concatenating the RGB information of adjacent frames in batches to obtain target frame features and reference frame features; calculating the affinity of the target frame features with respect to the reference frame features in a sliding window, and calculating the feature flow and confidence; calculating absolute feature position coordinates, aggregating the feature flow, absolute feature position coordinates, and depth spatial priors to perform depth feature analysis to obtain a position feature set; and hierarchically injecting the position feature set into the semantic feature set to obtain a deeply fused feature expression, thereby obtaining a depth prediction map and camera pose prediction results. Compared with the prior art, the present invention can further improve the accuracy and robustness of joint depth and pose estimation.

Description

Technical Field

[0001] The present invention belongs to the technical field of unsupervised depth and pose joint estimation, and in particular relates to a depth and pose joint estimation method and medium based on spatial information aggregation. Background Art

[0002] With the rapid development of robotics and autonomous driving technologies, deep pose joint estimation algorithms used to determine the position and posture of robots, autonomous vehicles or other autonomous systems in three-dimensional space have become one of the current research hotspots. The information obtained through these algorithms plays a vital role in achieving accurate navigation, environmental perception and task execution.

[0003] Traditional pose estimation algorithms are typically based on sensor data, such as inertial measurement units (IMUs), global positioning systems (GPS), and vision sensors. However, these sensors are often affected by factors such as environmental conditions, occlusion, and noise, resulting in limited accuracy and stability of pose estimation. To address these issues, joint depth-pose estimation algorithms that incorporate spatial information have emerged in recent years. These algorithms combine the three-dimensional spatial information provided by depth sensors (such as lidar or RGB-D cameras) with other information provided by traditional sensors to achieve more accurate and robust pose estimation, with broad application prospects in areas such as robotic navigation, environmental modeling, and target tracking. Depth sensors can provide accurate three-dimensional position and shape information of objects in the environment. Compared with traditional two-dimensional image sensors, this information is richer and more robust. By fusing depth information with other sensor data, the accuracy of pose estimation can be effectively improved, and performance can be improved in complex environments. Chinese invention patent CN111105432B discloses an unsupervised, end-to-end driving environment perception method based on deep learning. This method uses a binocular camera for image acquisition to obtain training data. Two consecutive stereo images of equal size in the training data are used to train an optical flow estimation network, a pose estimation network, a depth estimation network, and motion segmentation. The outputs of the three networks are used for rigid registration, and the output of the pose estimation network is optimized. The output of the depth estimation network and the output of the optimized pose estimation network are used to calculate the rigid flow caused by camera motion. This flow consistency is then checked against the output of the optical flow estimation network to perform motion segmentation. However, this method relies heavily on the quality and diversity of the training data. In a real autonomous driving environment, a variety of unforeseen factors may affect model performance. Therefore, a redesigned method for joint depth and pose estimation is needed to further improve the accuracy and robustness of joint depth and pose estimation to meet the growing demands of autonomous systems. Summary of the Invention

[0004] The purpose of the present invention is to overcome the defects of the above-mentioned prior art and provide a method and medium for joint depth and pose estimation based on spatial information aggregation, so as to further improve the accuracy and robustness of joint depth and pose estimation.

[0005] The purpose of the present invention can be achieved by the following technical solutions:

[0006] The present invention provides a method for joint depth and pose estimation based on spatial information aggregation, comprising the following steps:

[0007] Obtain a monocular image sequence of a road scene, extract the RGB information of adjacent frames and input it into a trained joint depth and pose estimation model, and output a depth prediction map and camera pose prediction results. The joint depth and pose estimation model includes a parallel depth estimation network, a camera pose estimation network, and a feature stream extraction backbone network. The output end of the feature stream extraction backbone network is sequentially provided with a confidence self-perception feature stream estimator, a spatial information aggregator, and a hierarchical position code injector. The specific process of the joint depth and pose estimation model processing input information to obtain output is as follows:

[0008] S1. Input the RGB information of adjacent frames into the depth estimation network to obtain the depth space prior D;

[0009] S2. After splicing the RGB information of adjacent frames by channel, input it into the camera pose estimation network to obtain a semantic feature set

[0010] S3, after splicing the RGB information of adjacent frames in batches, input the feature flow extraction backbone network, and perform feature separation in batches to obtain the target frame feature F t and reference frame features F s ;

[0011] S4, the target frame feature F t and the reference frame feature F s Input confidence self-perception feature flow estimator to calculate the target frame feature F t Regarding the reference frame feature F s The affinity of the sliding window A i , based on affinity A i Calculating feature flow and pixel-wise confidence C i ;

[0012] S5. Calculate the absolute feature position coordinates pixel by pixel Based on the confidence C i , using the spatial information aggregator to The absolute feature position coordinates And the depth spatial prior D obtained in step S1 is used to perform deep feature aggregation to obtain the position feature set

[0013] S6, injecting the position feature set into the layered position coding injector The semantic feature set obtained by hierarchical injection step S2 In the deep fusion feature expression F k , the deep fusion feature expression F k After average pooling, the data is input into a multi-layer perceptron to obtain the depth prediction map and camera pose prediction results.

[0014] Furthermore, the depth estimation network and the camera pose estimation network have the same structure, both including an encoder based on resnet-50 and a decoder based on a convolutional neural network.

[0015] Furthermore, the feature flow extraction backbone network is constructed based on a residual convolutional neural network, and during the training process of the deep pose joint estimation model, the weights of the feature flow extraction backbone network are frozen to keep its parameters unchanged.

[0016] Furthermore, in step S4, the target frame feature F t Regarding the reference frame feature F s The affinity of the sliding window A i The calculation formula is:

[0017]

[0018] in, and They are the reshaped target frame features and reference frame features respectively.

[0019] Furthermore, the target frame feature F t and the reference frame feature F s The specific process of reshaping is as follows:

[0020] The target frame feature F t and the reference frame feature F s Perform linear normalization and normalize the reference frame feature F s Expand the window with side length d to obtain Then the normalized target frame feature F t Reshape into To match size.

[0021] Furthermore, in step S4, the confidence C i is calculated as follows:

[0022]

[0023] Among them, A i is the affinity, and p is the position enumeration within the window.

[0024] Furthermore, in step S5, the absolute feature position coordinates is calculated as follows:

[0025]

[0026] Among them, h is the feature map height, w is the feature map width, Meshgrid(·) is the grid function, and Concat(·) is the concatenation function.

[0027] Furthermore, in step S5, the position feature set is obtained The process is as follows:

[0028]

[0029] in, With learnable parameters Two consecutive convolutional layers, C i is the pixel-by-pixel confidence.

[0030] Furthermore, in step S6, the position feature set is injected into the hierarchical position coding injector. Inject the semantic feature set into k layers In the deep fusion feature expression F k , for the position embedding feature of the i-th layer and semantic features have:

[0031]

[0032] Among them, γ i is the learnable gating parameter, f i (·,Θ i ) is a learnable parameter The combination of convolutional layer, activation function layer and downsampling layer, f c (·) is the channel reduction function implemented by the convolution layer with a convolution kernel size of 1×1, and F0 is initialized to Zero matrices of the same shape.

[0033] The present invention also provides a computer-readable storage medium having a computer program stored thereon, wherein the program implements the above method when executed by a processor.

[0034] Compared with the prior art, the present invention has the following beneficial effects:

[0035] 1. The present invention obtains the depth prediction map and camera pose prediction results through the depth-pose joint estimation model. Specifically, the depth-pose joint estimation model calculates the target frame feature F through the confidence self-perception feature flow estimator t About the reference frame feature F s The affinity of the sliding window A i , based on affinity A i Calculating feature flow and pixel-wise confidence C i , the dense feature correspondence can be re-weighted; the feature flow is aggregator Absolute feature position coordinates And deep spatial prior D performs deep feature aggregation to obtain position feature set The two-dimensional feature flow and three-dimensional spatial layout can be integrated into the homogeneous representation of position clues. The introduction of spatial priors can also help improve the accuracy and robustness of the model. The position feature set is injected into the layered position encoding injector. Hierarchical injection of semantic feature sets In the deep fusion feature expression F k , we can selectively embed the rich geometric representations of shallower layers into the refined semantic abstractions of deeper layers, thereby effectively improving the accuracy of depth and pose estimation.

[0036] 2. The present invention obtains the depth space prior D through the depth estimation network and obtains the semantic feature set through the camera pose estimation network The depth estimation network and the camera pose estimation network are in parallel and have the same structure. They both include an encoder based on ResNet-50 and a decoder based on a convolutional neural network. They can extract rich semantic information and geometric texture information through the pre-trained network skeleton, which is helpful for the calculation of subsequent feature flow matching in the model and the modeling of spatial correlation information.

[0037] 3. During the training process of the depth pose joint estimation model, the present invention freezes the weights of the feature flow extraction backbone network and keeps its parameters unchanged, which can reduce the amount of trainable parameters of the model and accelerate model training while ensuring accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 It is a structural diagram of the depth pose joint estimation model.

[0039] Among them, I t and I r are the target frame and the reference frame respectively, d is the depth prediction map, c is the number of feature map channels, h is the feature map height, w is the feature map width, [R, t] represents the camera pose, where R is the rotation matrix and t is the translation vector. DETAILED DESCRIPTION

[0040] The present invention is described in detail below with reference to the accompanying drawings and specific embodiments. This embodiment is implemented based on the technical solution of the present invention, and provides a detailed implementation method and specific operation process, but the protection scope of the present invention is not limited to the following embodiments.

[0041] Example:

[0042] Traditional pose estimation network architecture is as follows Figure 1 As shown in the upper part of , the CNN-based feature extractor extracts features of shape c×h×w from the input image, where c is the number of feature map channels, h is the feature map height, and w is the feature map width. Dimensionality reduction and average pooling are then performed to obtain a representation of the camera pose [R, t]. This embodiment provides a method for joint depth and pose estimation based on spatial information aggregation, including the following steps:

[0043] Obtain a monocular image sequence of a road scene, extract the RGB information of adjacent frames and input it into a trained depth-position joint estimation model, and output a depth prediction map and camera pose prediction results. The structure of the depth-position joint estimation model proposed in this embodiment is as follows: Figure 1 As shown in the lower part, the specific process of processing input information to obtain output is as follows:

[0044] S1. Input the RGB information of adjacent frames into the depth estimation network to obtain the depth space prior D.

[0045] S2: After splicing the RGB information of adjacent frames by channel, input it into the camera pose estimation network to obtain the semantic feature set

[0046] Images have three RGB channels, and adjacent frames are spliced ​​channel by channel. The depth estimation network and camera pose estimation network are parallel and have the same structure, both consisting of an encoder based on Resnet-50 and a decoder based on a convolutional neural network. Pre-trained network skeletons can extract rich semantic and geometric texture information, which facilitates the calculation of subsequent feature flow matching in the model and the modeling of spatial correlation information. However, based on the specific use case, the network must be fine-tuned in a specific dataset to obtain the most appropriate network parameter weights. Model inference in specific scenarios is then performed to achieve more accurate predictions. Directly using pre-trained weights does not guarantee the full performance of the network. Fine-tuning generally refers to trying different learning rates, optimizers, and training times, performing repeated training, and selecting the data with the best indicators as the final optimal network weights.

[0047] S3, after splicing the RGB information of adjacent frames in batches, input the feature flow extraction backbone network, and perform feature separation in batches to obtain the target frame feature F t and reference frame features F s .

[0048] During the training process, a batch of images is input each time, and adjacent frames are spliced ​​in batches. The feature flow extraction backbone network is built based on the residual convolutional neural network. During the training process of the deep pose joint estimation model, the weights of the feature flow extraction backbone network are frozen to keep its parameters unchanged. This can reduce the number of trainable parameters and accelerate training while ensuring accuracy. The target frame features obtained by batch separation and reference frame features The shape of is c×h×w. The target frame I t and reference frame I r At the same time, the model is input, and the model reconstructs a target frame based on the reference frame, and then optimizes the network based on the photometric error between the reconstruction result and the original target frame.

[0049] S4, the reshaped target frame feature F t and reference frame features F s Input confidence self-aware feature flow estimator to calculate the affinity A of the sliding window of the reshaped target frame feature with respect to the reference frame feature i , based on affinity A i Calculating feature flow and pixel-wise confidence C i .

[0050] For the target frame feature F t and reference frame features F s Perform linear normalization and normalize the reference frame feature F s Expand the window with a side length of d to get a shape of h×w×c×d 2 Reference frame features

[0051]

[0052] Then the normalized target frame feature F t Reshape into To match

[0053] Then the characteristic affinity A of stage i i The calculation is as follows:

[0054]

[0055] Confidence C i is calculated as follows:

[0056]

[0057] Where p is the position enumeration within the window.

[0058] S5. Calculate the absolute feature position coordinates pixel by pixel Based on the confidence level C i , using spatial information aggregator to Absolute feature position coordinates And the depth spatial prior D obtained in step S1 is used to perform deep feature aggregation to obtain the position feature set

[0059] Absolute feature position coordinates is calculated as follows:

[0060]

[0061] Among them, Meshgrid(·) is the mesh function, and Concat(·) is the concatenation function.

[0062] Get location feature set The process is as follows:

[0063]

[0064] in, With learnable parameters Two consecutive convolutional layers are used to map a two-dimensional or three-dimensional position vector to a higher embedding dimension. The introduction of spatial position prior information helps to improve the accuracy and robustness of the model.

[0065] S6, position feature set is injected into the layered position encoding injector The semantic feature set obtained by hierarchical injection step S2 In the deep fusion feature expression F k , express the deep fusion feature F k After average pooling, the data is input into the multi-layer perceptron MLP for decoding to obtain the depth prediction map and camera pose prediction results.

[0066] Position feature sets are injected into the layered position encoding injector Inject semantic feature sets into k layers In the deep fusion feature expression F k , for the position embedding feature of the i-th layer and semantic features have:

[0067]

[0068] Among them, γ i is a learnable gating parameter that allows the network to selectively inject position embeddings into semantic cues, f i (·,Θ i ) is a learnable parameter The combination of convolutional layer, activation function layer and downsampling layer, f c (·) is the channel reduction function implemented by the convolution layer with a convolution kernel size of 1×1, and F0 is initialized to Zero matrices of the same shape.

[0069] The above method effectively improves the accuracy and robustness of scene parsing, and can simultaneously output accurate pixel-level depth prediction and camera pose prediction results, providing strong technical support for real-time environmental perception and decision-making in intelligent transportation systems, mobile robots and autonomous driving, and has broad application prospects.

[0070] If the above method is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, and other media that can store program code.

[0071] The above description of the embodiments is intended to facilitate understanding and use of the invention by those skilled in the art. It will be apparent that those skilled in the art can readily make various modifications to these embodiments and apply the general principles described herein to other embodiments without requiring inventive effort. Therefore, the present invention is not limited to the above-described embodiments. Improvements and modifications made by those skilled in the art based on the disclosure of the present invention, without departing from the scope of the present invention, should be within the scope of protection of the present invention.

Claims

1. A method for joint depth and pose estimation based on spatial information aggregation, characterized in that: The following steps are involved: Obtain a monocular image sequence of a road scene, extract the RGB information of adjacent frames and input it into a trained joint depth and pose estimation model, and output a depth prediction map and camera pose prediction results. The joint depth and pose estimation model includes a parallel depth estimation network, a camera pose estimation network, and a feature stream extraction backbone network. The output end of the feature stream extraction backbone network is sequentially provided with a confidence self-perception feature stream estimator, a spatial information aggregator, and a hierarchical position code injector. The specific process of the joint depth and pose estimation model processing input information to obtain output is as follows: S1. Input the RGB information of adjacent frames into the depth estimation network to obtain the depth space prior ; S2. After splicing the RGB information of adjacent frames by channel, input it into the camera pose estimation network to obtain a semantic feature set ; S3, after splicing the RGB information of adjacent frames in batches, input the feature flow extraction backbone network, and perform feature separation in batches to obtain the target frame features and reference frame features ; S4, the target frame features and the reference frame features Input confidence self-perception feature flow estimator to calculate the target frame features Regarding the reference frame features The affinity of the sliding window , based on affinity Calculating feature flow and pixel-wise confidence ; S5. Calculate the absolute feature position coordinates pixel by pixel , based on the confidence , using the spatial information aggregator to , the absolute feature position coordinates And the depth space prior obtained in step S1 Perform deep feature aggregation to obtain a set of position features ; S6, injecting the position feature set into the layered position coding injector The semantic feature set obtained by hierarchical injection step S2 In the deep fusion feature expression , the deep fusion feature expression After average pooling, the data is input into the multi-layer perceptron to obtain the depth prediction map and camera pose prediction results; In step S5, the absolute feature position coordinates is calculated as follows: in, is the feature map height, is the feature map width, is the grid function, is the splicing function; In step S5, the position feature set is obtained The process is as follows: in, With learnable parameters Two consecutive convolutional layers, is the pixel-by-pixel confidence.

2. A method for joint depth and pose estimation based on spatial information aggregation according to claim 1, characterized in that: The depth estimation network and the camera pose estimation network have the same structure, both including an encoder based on ResNet-50 and a decoder based on a convolutional neural network.

3. The method for joint depth and pose estimation based on spatial information aggregation according to claim 1, characterized in that: The feature flow extraction backbone network is constructed based on a residual convolutional neural network, and during the training process of the depth pose joint estimation model, the weights of the feature flow extraction backbone network are frozen to keep its parameters unchanged.

4. The method for joint depth and pose estimation based on spatial information aggregation according to claim 1, characterized in that: In step S4, the target frame features Regarding the reference frame features The affinity of the sliding window The calculation formula is: in, They are the reshaped target frame features and reference frame features respectively.

5. The method for joint depth and pose estimation based on spatial information aggregation according to claim 4, characterized in that: The target frame features and the reference frame features The specific process of reshaping is as follows: The target frame features and the reference frame features Perform linear normalization and normalize the reference frame features According to the side length The window is expanded to obtain , and then normalize the target frame features Reshape into , to match size.

6. The method for joint depth and pose estimation based on spatial information aggregation according to claim 1, characterized in that: In step S4, the confidence is calculated as follows: in, For affinity, Enumeration for positions within the window.

7. The method for joint depth and pose estimation based on spatial information aggregation according to claim 1, characterized in that: In step S6, the position feature set is injected into the hierarchical position coder. point Layer injects the semantic feature set In the deep fusion feature expression , for the Layer position embedding features and semantic features ,have: in, is the learnable gating parameter, With learnable parameters The combination of convolutional layer, activation function layer and downsampling layer, is the channel reduction function implemented by the convolution layer with a convolution kernel size of 1×1, Initialized with Zero matrices of the same shape.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images