An optical flow pose estimation method based on contextual semantics and pixel uncertainty

Abstract

The present invention discloses an optical flow pose estimation method based on contextual semantics and pixel uncertainty. The method utilizes an end-to-end monocular visual odometry network architecture to deeply couple the optical flow network and the pose network. The optical flow map, contextual semantic features and uncertainty map output by the optical flow network are used as input information of the pose network. The modules of the two are deeply coupled and jointly trained and optimized to achieve reliable and accurate estimation of the camera pose. At the same time, by introducing contextual semantic features and pixel-level uncertainty estimation mechanism in parallel, adaptive focusing and dynamic filtering of key areas are achieved. The complementary fusion of contextual semantic feature information and motion information is achieved, and the semantic expression ability of the optical flow features is enhanced. At the same time, the element-by-element addition operation promotes gradient flow and improves the training stability of the model. In particular, in scenes with dynamic objects or complex backgrounds, the module can effectively distinguish between foreground and background and reduce interference from irrelevant areas.

Description

Technical Field

[0001] The present invention belongs to the technical fields of robot navigation and positioning, computer vision and three-dimensional reconstruction, and in particular relates to an optical flow pose estimation method based on contextual semantics and pixel uncertainty. Background Art

[0002] Monocular visual odometry is a key technology for autonomous robot navigation and precise positioning. Its basic principle is to estimate the camera's position and pose by analyzing the pixels between consecutive image frames and calculating the movement of feature points within them, thereby inferring the camera's translation and rotation. Currently, there are two main approaches to monocular visual odometry: one uses traditional feature extraction and optical flow calculations for pose estimation, and the other uses deep learning to train a neural network model to estimate camera pose.

[0003] In traditional methods, feature extraction extracts key feature points from an image and estimates camera motion by matching pairs of key points in consecutive images. It is robust to changes in illumination and scale, but performs poorly in low-texture scenes. Optical flow, on the other hand, does not extract discrete feature points, but optimizes the camera pose by constructing a photometric error function to minimize the pixel grayscale of the current frame after projection from the previous frame, assuming photometric consistency. It also performs poorly in scenes with large photometric changes. Deep learning-based methods learn features and estimate poses directly from image sequences, without manually designing a feature extraction and matching process, and directly output the relative pose changes of the camera. A classic example is DeepVO, which uses a combination of CNN and RNN to learn camera pose changes from monocular image sequences end-to-end. Compared with traditional methods, it improves accuracy and robustness, but still faces challenges in complex scenes.

[0004] Traditional geometry-based monocular visual odometry methods, typified by ORB-SLAM2, rely on a manually designed feature extraction and matching process, typically including: first, detecting image keypoints and generating descriptors using the ORB feature extractor; then performing feature matching using the Hamming distance between descriptors; estimating the camera pose using the PnP algorithm; and finally, optimizing the pose and map points through bundle adjustment. However, these methods suffer from the following drawbacks: they are not robust to dynamic scenes; moving objects disrupt feature matching relationships, affecting pose estimation accuracy; in low-texture areas such as the sky and walls, the lack of valid feature points leads to unstable estimation; they rely heavily on manual parameter adjustment and lack adaptability; and their modular, non-end-to-end architecture is prone to error accumulation, leading to overall performance degradation.

[0005] Monocular visual odometry methods based on deep learning primarily rely on deep neural networks to directly regress the camera pose from images. These methods include direct regression and optical flow-assisted methods. The direct regression method uses a CNN to extract features from two image frames and output translation and rotation parameters through a fully connected layer (K. Konda and R. Memisevic, “Learning visual odometry with a convolutional network,” in International Conference on Computer Vision Theory and Applications, vol. 2, pp. 486–490, SciTePress, 2015).

[0006] ”(S. Wang, R. Clark, H. Wen, and N. Trigoni, “Deepvo: Towards end-to-end visual odometry with deep recurrent convolutional neural networks,” in2017 IEEE international conference on robotics and automation (ICRA), pp.2043–2050, IEEE, 2017.); Optical flow-assisted methods such as TartanVO and DROID-VO first use the optical flow network to estimate the inter-pixel motion, then combine the camera intrinsic parameters to form intermediate features to input the pose network, and finally regress the camera pose (Wang, Wenshan, Yaoyu Hu, and Sebastian Scherer. "Tartanvo: A generalizablelearning-based vo." Conference on Robot Learning . PMLR, 2021.) (Teed, Zachary, and Jia Deng. "Droid-slam: Deep visual slam for monocular, stereo, and rgb-dcameras." Advances in neural information processing systems34 (2021): 16558-16569.). Although learning-based methods have certain end-to-end characteristics and show good generalization capabilities, they also have shortcomings: the lack of attention mechanism for dynamic domains or low-texture areas affects the stability and robustness of pose estimation; the failure to fully exploit the semantic information in the image limits the model's ability to understand complex scenes; the overall network structure is complex and consumes a lot of computing resources, which is not conducive to real-time deployment on resource-constrained devices.

[0007] Therefore, the existing technology has not yet been able to ensure real-time performance while taking into account dynamic robustness, texture adaptability and structural simplicity, and there is still much room for improvement. Summary of the Invention

[0008] In response to the shortcomings of the existing technology, the present invention provides an optical flow pose estimation method based on contextual semantics and pixel uncertainty, aiming to effectively solve the various challenges faced by existing visual odometry methods when estimating camera pose in complex environments.

[0009] The present invention is achieved through the following technical solutions:

[0010] An optical flow pose estimation method based on contextual semantics and pixel uncertainty includes the following steps:

[0011] (1) Use a monocular camera to capture continuous video frame sequences, and preprocess the input images;

[0012] (2) Input two consecutive frames of images into the shared weight pre-trained dual-path ResNetFPN and cyclic RNN optical flow estimation network to obtain the optical flow map; then input the current frame into the ResNetFPN network to extract the contextual semantic feature information; and obtain the initial optical flow uncertainty map through the internal convolution layer, and then obtain the final optical flow map and uncertainty map through the RNN network cyclic estimation;

[0013] (3) Convert the camera internal parameters into the internal parameter layer; and combine the optical flow output by the optical flow network Figure 1 The two layers are concatenated into an optical flow intrinsic reference layer, and then cropped and scaled to 1 / 4 resolution.

[0014] (4) The scaled optical flow intrinsic reference layer and uncertainty map are used as inputs to the pose prediction network into the U-type network architecture, and the contextual semantic feature information and the features extracted by the intrinsic reference layer in the second layer of the encoder are input together into the contextual semantic attention module of the encoder placed in the U-type network architecture for attention fusion operation;

[0015] (5) The pixel-level camera pose is predicted through the decoder in the U-like network architecture, the local pose is filtered using the uncertainty map, and finally the global pose and local pose are fused by a hierarchical weighted method to obtain the final camera pose.

[0016] Furthermore, the preprocessing in step (1) is specifically as follows: adjusting the normalized pixel value range of the input image to [0, 1] as the input of the optical flow network, wherein the resolution of each frame image is H×W, the number of channels is 3, and the frame rate is kept unchanged.

[0017] Specifically, the contextual semantic feature information in step (2) is extracted through the ResNet network, which contains image semantics and texture information, and is used to enhance the semantic expression ability of the subsequent pose network.

[0018] Specifically, the uncertainty information of the uncertainty map in step (2) is applied to the screening of pixel-level poses to suppress noise.

[0019] Furthermore, the conversion of the camera intrinsic parameters into the intrinsic parameter layer in step (3) is specifically as follows: the camera intrinsic parameters include focal length and optical center, and an intrinsic parameter layer of the same size as the optical flow map is generated according to the focal length and optical center parameters, and the intrinsic parameter layer is spliced with the optical flow map to obtain the optical flow intrinsic parameter layer.

[0020] Furthermore, the step (4) specifically includes the following sub-steps:

[0021] (4.1) The optical flow intrinsic parameter layer is subjected to feature extraction through three layers of convolution, i.e. the first layer is downsampled to 60×80 with a stride of 2, and the second and third layers have a stride of 1, with an output channel number of 32;

[0022] (4.2) Contains five layers of residual modules from layer 1 to layer 5, in which a contextual semantic attention CAPA module is added to the second layer of the encoder. It has two input optical flow intrinsic parameters, the features and semantic features extracted from the second layer, and realizes the complementary fusion of contextual semantic feature information and motion information through channel attention and pixel attention. The fused features are then used to extract features using the following three layers of residual modules. The number of output channels is 64, 128, 128, 256, and 256, respectively, and the feature map resolution decreases to 1×2 layer by layer.

[0023] Furthermore, the step (4.2) is specifically as follows: the contextual semantic attention module includes two sub-modules: channel attention CA and pixel attention PA; first, the contextual semantic features are extracted and integrated through convolutional layers and ReLU functions to maintain dimensional consistency, and then the corresponding element addition operation is performed on the features extracted from the second layer with the optical flow intrinsic reference to achieve preliminary complementarity between semantic information and motion information, and the added features are input into the contextual semantic attention module; the module adjusts the weights of different channels through global pooling and attention weight calculation, and multiplies the added features with the calculated channel weights element by element to generate enhanced channel features; then, the channel features are used as the input of the pixel attention PA sub-module to generate a pixel feature attention map, which is multiplied element by element with the channel features to obtain pixel enhanced features; finally, the pixel enhanced features are added to the features extracted from the second layer to obtain the final attention features.

[0024] Furthermore, the U-type network architecture in step (4) is specifically as follows: a U-type encoding-decoding network design is adopted, wherein the encoder backbone is ResNet32, which is used to receive the spliced optical flow map, the intrinsic reference layer and the contextual semantic features; and the decoder restores the spatial resolution through skip connections and outputs a pixel-level pose prediction map.

[0025] Furthermore, the step (5) specifically includes the following sub-steps:

[0026] (5.1) Since the uncertainty map has been reduced to the same size as the spliced features, it is first necessary to upscale it to a pixel-wise translation uncertainty map and a rotation uncertainty map through two learnable convolutional layers, which represent the uncertainty distribution of each translation and rotation component, respectively, thereby providing independent uncertainty estimates for each dimension of camera translation and rotation;

[0027] (5.2) The right half of the U-shaped network architecture is responsible for decoding and predicting the camera translation and rotation at the pixel level. It is then fused with the uncertainty map output by the optical flow network to select the most robust local translation and rotation. At the same time, a global camera pose estimation network is added in parallel to the local camera pose estimation network at the encoder output. The global pose translation and rotation are directly predicted based on high-level features. The camera pose is finally obtained by fusing the local pose and the global pose in a weighted combination.

[0028] (5.3) First, the camera translation and rotation as well as the pixel translation uncertainty map and rotation uncertainty map are divided into small pixel blocks, and then the translation and rotation of the pixel with the lowest uncertainty are indexed from each small pixel block, and based on the current uncertainty, the Softmax function is used to calculate the weight of each pixel pose, and finally a more robust local pose translation and rotation is obtained by weighted summation; the global pose prediction module takes the high-level features of the encoder as input, and then integrates the global features through two independent fully connected layers to estimate the translation and rotation of the global pose, and fused with the translation and rotation of the local pose for weighted summation to obtain the final camera pose; the weight of the global pose prediction value and the weight of the local pose prediction value are each 0.5.

[0029] The beneficial effects of the present invention are as follows:

[0030] 1. This invention constructs a tightly coupled joint architecture for optical flow estimation and pose regression. This architecture utilizes an end-to-end deep network design, integrating multiple processes, including optical flow prediction, contextual semantic feature extraction, attention mechanisms, and pose decoding. This avoids the problem of error propagation and accumulation between modules in traditional methods. Furthermore, the introduction of a contextual semantic attention module (CAPA) and an uncertainty filtering module achieves adaptive coupled optimization of optical flow network information and enhances scene semantic understanding. The optimal local pixel-level pose information obtained through uncertainty filtering significantly enhances the network's robustness to interference from data outside the training domain. Finally, while simplifying the architecture, this approach significantly improves system stability and integration. This invention designs an explicit contextual semantic feature guidance mechanism. By constructing a semantic weight map and fusing it with the encoded feature map, the model is guided to focus on key areas strongly related to pose estimation (such as static landmarks and fixed structures), dynamically suppressing attention to less relevant areas. This mechanism significantly improves the model's understanding and feature utilization efficiency in complex environments, particularly in dynamic scenes, weakly textured areas, and blurred images, maintaining stable and robust estimation performance.

[0031] 2. By introducing a contextual semantic attention module and an uncertainty screening mechanism, this invention significantly improves the model's responsiveness to static and texture-rich areas, effectively suppressing the effects of dynamic interference and image blur. Furthermore, the network is constructed using lightweight residual modules, combined with a reduced-size optical flow map input strategy, reducing model complexity and computational load. This allows the system to maintain high accuracy while achieving higher operational efficiency, making it more suitable for real-time applications.

[0032] 3. This invention uses the uncertainty information of the optical flow network to screen the effective area, ensuring the model's accuracy and generalization capabilities. Furthermore, through lightweight structure and module optimization, it can still achieve efficient operation on resource-constrained platforms such as embedded devices. It has excellent real-time performance and generalization capabilities, making it suitable for visual positioning tasks in a variety of complex scenarios such as autonomous driving, robot navigation, and augmented reality. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0034] Figure 1 It is the overall framework diagram of the present invention;

[0035] Figure 2 Contextual semantic attention module diagram of the present invention;

[0036] Figure 3 This is a diagram of the local pose network architecture of the present invention. DETAILED DESCRIPTION

[0037] Exemplary embodiments are described in detail herein, with examples illustrated in the accompanying drawings. When the following description refers to the drawings, identical numerals in different drawings represent identical or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments are not intended to represent all embodiments consistent with this application.

[0038] This example describes a coupled optical flow pose estimation network based on contextual semantics and pixel uncertainty. Figure 1 This system uses an end-to-end deep learning network to deeply couple the optical flow network and the pose network. Using the optical flow, contextual semantic features, and uncertainty features output by the optical flow network as input to the pose network, the system deeply couples the two modules for joint training and optimization, addressing the accuracy and robustness issues of traditional monocular visual odometry in dynamic scenes, changing lighting, and ambiguous conditions. The following details its implementation, including system structure, method steps, and technical results.

[0039] Scenario description: Any monocular camera with known intrinsic parameters can acquire image and video frames through the monocular camera, calculate the relative position of the camera using only two frames of images, and finally infer the motion trajectory of the camera in the entire video stream.

[0040] 1. System Configuration

[0041] The product structure of this system consists of the following core components, which are closely connected through data flow and calculation logic:

[0042] (1) Monocular camera

[0043] Function: Capable of continuously capturing video frames and inputting them into the system as data sequences.

[0044] Positional relationship: It can be installed at the front end or top of any autonomous mobile device (such as a robot or unmanned vehicle), with the lens facing unobstructed.

[0045] Parameters: The camera's intrinsic parameters are known, the frame rate is fixed, and it can output RGB three-channel images with unlimited resolution and RGB three-channel output.

[0046] (2) Optical flow network module

[0047] Function: Responsible for predicting the optical flow map between adjacent video frames and representing the motion of each pixel.

[0048] Connection relationship: The optical flow network receives continuous frames output by the monocular camera , output optical flow map, contextual semantic feature information and uncertainty information to the pose estimation network.

[0049] Implementation: Based on pre-trained optical flow estimation network, such as Sea-RAFT network (Wang Y, Lipson L, Deng J. Sea-raft: Simple, efficient, accurate raft for optical flow[C] / / European Conference on Computer Vision. Cham: Springer Nature Switzerland,2024: 36-54.). Figure 1 After the monocular camera captures continuous image frames, the image data is passed as input to the optical flow network in sequence to extract contextual semantic information and uncertainty features. The optical flow internal parameters are then spliced and input together with the contextual semantic feature information and uncertainty information into the pose network for joint training and inference. Finally, the computing device outputs the camera motion pose corresponding to the current frame.

[0050] (3) Pose network module

[0051] Function: The core estimates global and local pose information, is responsible for extracting optical flow maps and intrinsic reference layer features, and fuses contextual semantic feature information and uncertainty maps from the optical flow network, combining different modules for pose optimization and regression.

[0052] Composition: Contains ResNet32 encoder, contextual semantic attention module, global camera motion pose prediction branch including fully connected layer, uncertainty screening local camera motion pose prediction branch

[0053] Connection relationship: such as Figure 1 As shown in the pose network, the pose network receives the feature map after the fusion of optical flow and camera intrinsic parameters, the contextual semantic feature information extracted by the optical flow network and the uncertainty map, completes the contextual semantic feature fusion through the contextual semantic attention module, and uses the uncertainty screening module to complete the local optimal pose screening. Finally, the local pose is combined with the global pose weighted to obtain the output camera six-degree-of-freedom pose estimation result.

[0054] 2. Methods and Steps

[0055] This system estimates the camera pose through the following specific steps, which are clear and operational:

[0056] Step 1: Data Collection

[0057] Use a monocular camera to capture a continuous sequence of video frames ,in Indicates the Frame image, frame number Increment from 1 to Each frame has a resolution of H × W (H is height, H is width, in pixels) and 3 channels (RGB format). The frame rate must be stable (e.g., 30 frames per second) to ensure temporal continuity. The input image must be preprocessed and normalized to a pixel value range of [0, 1] before being fed into the optical flow network.

[0058] Step 2: Prediction of optical flow map, contextual semantic feature information and uncertainty map

[0059] like Figure 1 The optical flow network shown in the figure is to convert two consecutive frames of images into Input the shared weight pre-trained dual-path ResNetFPN and recurrent RNN optical flow estimation network to obtain the optical flow map , the two dimensions represent the horizontal and vertical motion components. At the same time, according to the current frame Input into the ResNetFPN network to extract Deep contextual semantic feature information at all times This feature can better reflect the high-level semantic information in the image, such as the category of the object, the context of the scene, etc. At the same time, the RNN network will also obtain the uncertain graph , the uncertainty map is convolved to obtain a pixel uncertainty map, and this uncertainty information is used to filter the pixel-level pose to suppress the influence of noise.

[0060] The present invention inputs each frame of image into processing with a resolution of 480×640 and a format of RGB three channels to ensure the standardization of input data.

[0061] Step 3: Intrinsic reference layer generation and feature splicing

[0062] like Figure 1 As shown, according to the camera internal parameters (focal length and optical center ) Generate internal reference layer (used for subsequent training of different camera data to improve generalization ability), the optical flow map and the intrinsic reference layer are spliced in the channel dimension (i.e., the optical flow intrinsic reference) to generate input features Finally, the image is aligned and interpolated to scale to 1 / 4 of the resolution to improve training efficiency.

[0063] Step 4: Fusion of optical flow intrinsic encoder and contextual semantic feature information

[0064] like Figure 1 The main process of the encoder of the pose network shown is as follows: the optical flow intrinsic parameters and uncertainty map scaled in step 3 are used as the input of the pose prediction network into the U-type network architecture, and the input optical flow intrinsic parameters are The features extracted from the second layer of the encoder in the U-shaped network architecture are input into the contextual semantic features together with the contextual semantic features for attention fusion operation, specifically:

[0065] (4.1) The optical flow intrinsic parameter features are first extracted through an initialized 3-layer convolution (the first layer has a stride of 2 and is downsampled to 60×80, and the second and third layers have a stride of 1), with an output channel number of 32.

[0066] (4.2) Then, a residual network consisting of five layers of residual modules (layer1 to layer5) is constructed, in which a contextual semantic attention CAPA module is added to the second layer of the encoder, which has two input optical flow internal reference second layer features and semantic features Through channel attention and pixel attention, the complementary fusion of contextual semantic feature information and motion information is achieved, enhancing the semantic expression ability of optical flow features. Subsequently, the fused features are further extracted using the following three layers of residual modules, with the number of output channels being 64, 128, 128, 256, and 256, respectively. The final feature map resolution is reduced to 1×2 layer by layer.

[0067] The contextual semantic attention module (CAPA module) of step (4.2) is as follows:

[0068] The contextual semantic attention module: Figure 2 As shown in the figure, it includes two sub-modules: channel attention (Context attention, CA) and pixel attention (Pixel attention, PA).

[0069] First, contextual semantic feature information Feature extraction and integration are achieved through convolutional layers and ReLU functions to maintain dimensional consistency, and then The corresponding element addition operation is performed to achieve the initial complementarity of semantic information and motion information. The added features are input into the contextual semantic attention module. This module adjusts the weights of different channels through global pooling and attention weight calculation, and multiplies the added features with the calculated channel weights element by element to generate enhanced channel features. , which is calculated as follows: .

[0070] Afterwards, The input pixel attention submodule generates a pixel feature attention map, which is compared with Element-wise multiplication to obtain pixel enhancement features .

[0071] Finally and Add up to get the final attention feature , and its output formula is as follows: .

[0072] The core idea of the contextual semantic attention module is to perform a two-stage refinement of the input feature map, achieving a complementary fusion of contextual semantic feature information and motion information, enhancing the semantic expressiveness of optical flow features. This module is particularly effective in scenes with dynamic objects or complex backgrounds, effectively distinguishing foreground from background and reducing interference from irrelevant areas. Furthermore, the element-by-element addition operation promotes gradient flow and improves model training stability.

[0073] Step 5: Optical flow intrinsic parameter decoder and uncertainty screening module

[0074] In the optical flow pose estimation method, such as Figure 3 As shown in the figure, the local pose estimation network uses the uncertainty screening module of the optical flow intrinsic parameter decoder to utilize the deep features and uncertainty information of the encoder to accurately predict the camera pose and improve the estimation robustness and accuracy.

[0075] (5.1) The decoder uses the last layer of features of the encoder As input, multi-stage features are fused through U-shaped network and skip connection to restore the pixel-level translation of the original spatial resolution size and rotation , uncertainty graph The dimension is increased to pixel translation uncertainty map through two layers of convolution and pixel rotation uncertainty map , providing confidence estimates for translation and rotation.

[0076] (5.2) The uncertainty screening module will 、 、 and Divided into indivual The small pixel block with the lowest index uncertainty is the translation and the amount of rotation , use the Softmax function to calculate the weight , weighted summation to obtain the translation of the local pose and the amount of rotation .

[0077] Step 6: Camera pose calculation

[0078] First, use Figure 1 The global pose prediction module of the pose network shown in the figure contains two fully connected branches (each branch structure is 512→128→32→3) to regress the translation parameters respectively. (3D vector) and rotation parameters , whose input is the output feature of the last layer of the encoder The flattened form of the global pose is finally output and the amount of rotation .

[0079] Finally, the local pose and and global pose and Fusion weighted to get the final camera pose , ,in .

[0080] 7. Network training

[0081] (a) Data preparation

[0082] Load the training dataset and validation dataset, which contain the following data: Image data contains two consecutive frames of RGB images. Pose truth: the camera's true translation and rotation parameters, which are converted into Lie algebra form through transformation. and The camera internal parameters include focal length and optical center parameters. , used to generate the internal reference layer .

[0083] (b) Input data preprocessing:

[0084] Random cropping is applied to the image and intrinsic reference layers to improve generalization during training. The images are then resized to a uniform size of 240×320, and the camera intrinsics are used to ensure that the cropped regions match the field of view. If optical flow data is available, it is downsampled and its flow volume is scaled to match the resolution and value of the cropped images.

[0085] (c) Model training parameters:

[0086] The training process is divided into two stages. In the first stage, the present invention first uses an analogy enhancement strategy for the real optical flow and pose in combination with a separate pose network to train 200 epochs, with a batch size of 256 and a learning rate of 1e-4. In the second stage, the present invention loads the pose network trained in the first stage as a pre-trained weight into the global pose estimation network of CPVO, and combines the image experiment analogy enhancement strategy to jointly optimize the optical flow and pose coupling network. The number of training iterations is 100, the batch size is 128, and the learning rate is 1e-5. The input image size of this optical flow network is 240×320, and the input image size of the pose network is 120×160, which reduces the GPU memory occupancy with minimal loss of precision. The present invention uses the AdamW optimizer, =0.9, =0.999, and introduced a multi-step learning rate scheduler (MultiStepLR). The first stage of training decayed at the 60th and 100th epochs, and the second stage decayed at the 50th epoch, with a decay rate of 0.2. The models of the present invention were all implemented based on PyTorch and trained on a single NVIDIA RTX 4090 GPU.

[0087] (d) Loss function:

[0088] During the model training process, the setting of the loss function directly affects the optimization effect of the network model parameters. It is unreasonable to recover the absolute motion scale of the camera from the monocular camera alone. Although the absolute scale can be obtained by overfitting the model, its generalization performance is extremely poor. Therefore, the present invention uses the camera translation between two frames as the Perform scale normalization, the expression is as follows:

[0089] ;

[0090] in, To avoid division by zero errors.

[0091] The final loss function of the present invention is as follows:

[0092] ;

[0093] in, represents the normalized predicted translation, represents the normalized true translation, is the predicted rotation amount, is the actual rotation amount, is the predicted pose transformation vector, is the true pose transformation vector. L1 loss is applied to each translation and rotation and the sum is used to obtain the final loss function.

[0094] Since the scale of the monocular camera motion itself cannot be observed from the monocular image sequence, the loss function of the camera pose is , scale ambiguity only affects translation, and the rotation loss remains unchanged. For this loss function, the camera pose loss function is defined as : ;

[0095] in, and is the camera motion prediction value, and is the true value of the camera motion, Avoid miscalculations caused by dividing by zero.

[0096] The protection scope of the present invention is not limited to the above-mentioned specific implementation structure, but also covers equivalent replacements and extensions based on the essence of the present technical solution, including but not limited to the following situations:

[0097] (a) Expansion and lightweight adjustment of the network architecture: such as changes in the number of residual module layers and the position of the attention context semantic attention module;

[0098] (b) Alternative uncertainty modeling strategies: For example, replacing current prediction mechanisms with methods such as Bayesian estimation and information entropy;

[0099] (c) Changes in the way contextual semantics are integrated: for example, introducing Transformer, Cross-Attention, or other contextual encoding modules to achieve semantic guidance;

[0100] (d) Replacement of optimizers and training strategies: such as replacing the optimizer (such as SGD, Adam) and changing the loss function design.

[0101] Therefore, as long as the adjustment does not deviate from the basic concepts and technical points of the present invention regarding optical flow-pose coupling, contextual semantic fusion and pixel-level uncertainty fusion, it should be deemed to be within the scope of protection defined by the claims of the present invention.

Claims

1. An optical flow pose estimation method based on contextual semantics and pixel uncertainty, characterized in that: The following steps are involved: (1) Use a monocular camera to capture continuous video frame sequences, and preprocess the input images; (2) Input two consecutive frames of images into the pre-trained dual-path ResNetFPN and recurrent RNN optical flow estimation network with shared weights to obtain the optical flow map; then input the current frame into the ResNetFPN network to extract the contextual semantic feature information; The initial optical flow uncertainty map is obtained through the internal convolution layer, and then the final optical flow map and uncertainty map are obtained through RNN network cycle estimation. (3) Convert the camera intrinsic parameters into the intrinsic parameter layer; combine them with the optical flow map output by the optical flow network to form the optical flow intrinsic parameter layer, and then crop it and scale it to 1 / 4 resolution; (4) The scaled optical flow intrinsic reference layer and uncertainty map are used as inputs to the pose prediction network into the U-type network architecture, and the contextual semantic feature information and the features extracted by the intrinsic reference layer in the second layer of the encoder are input together into the contextual semantic attention module of the encoder placed in the U-type network architecture for attention fusion operation; (5) The pixel-level camera pose is predicted through the decoder in the U-like network architecture, the local pose is filtered using the uncertainty map, and finally the global pose and local pose are fused by a hierarchical weighted method to obtain the final camera pose.

2. The optical flow pose estimation method according to claim 1, wherein The preprocessing in step (1) is specifically as follows: adjusting the normalized pixel value range of the input image to [0, 1] as the input of the optical flow network, wherein the resolution of each frame image is H×W, the number of channels is 3, and the frame rate is kept unchanged.

3. The optical flow pose estimation method according to claim 1, wherein The contextual semantic feature information in step (2) is extracted through the ResNet network, which contains image semantics and texture information and is used to enhance the semantic expression capability of the subsequent pose network.

4. The optical flow pose estimation method according to claim 1, wherein The uncertainty map in step (2) has uncertainty information that is used for filtering pixel-level poses to suppress noise.

5. The optical flow pose estimation method according to claim 1, wherein The step (3) of converting the camera intrinsic parameters into the intrinsic parameter layer is specifically as follows: the camera intrinsic parameters include focal length and optical center, and an intrinsic parameter layer of the same size as the optical flow map is generated according to the focal length and optical center parameters, and the intrinsic parameter layer is spliced with the optical flow map to obtain the optical flow intrinsic parameter layer.

6. The optical flow pose estimation method according to claim 1, wherein The step (4) specifically includes the following sub-steps: (4.1) The optical flow intrinsic parameter layer is subjected to feature extraction through three layers of convolution, i.e. the first layer is downsampled to 60×80 with a stride of 2, and the second and third layers have a stride of 1, with an output channel number of 32; (4.2) Contains five layers of residual modules from layer 1 to layer 5. A contextual semantic attention (CAPA) module is added to the second layer of the encoder. It takes two inputs, the features extracted from the second layer of the optical flow intrinsic reference and the semantic features, and uses channel attention and pixel attention to achieve complementary fusion of contextual semantic feature information and motion information. The fused features are then used to extract features using the following three layers of residual modules. The number of output channels is 64, 128, 128, 256, and 256, respectively, and the feature map resolution decreases to 1×2 layer by layer.

7. The optical flow pose estimation method according to claim 6, wherein: The step (4.2) is specifically as follows: the contextual semantic attention module includes two sub-modules: channel attention CA and pixel attention PA; first, the contextual semantic features are extracted and integrated through the convolution layer and the ReLU function to maintain dimensional consistency, and then the corresponding element addition operation is performed on the features extracted from the second layer with the optical flow intrinsic reference to achieve preliminary complementarity between semantic information and motion information, and the added features are input into the contextual semantic attention module; the module adjusts the weights of different channels through global pooling and attention weight calculation, and multiplies the added features with the calculated channel weights element by element to generate enhanced channel features; then, the channel features are used as the input of the pixel attention PA sub-module to generate a pixel feature attention map, which is multiplied element by element with the channel features to obtain pixel enhanced features; Finally, the pixel enhanced features are added to the features extracted from the second layer to obtain the final attention features.

8. The optical flow pose estimation method according to claim 1, wherein: The U-shaped network architecture in step (4) is specifically as follows: a U-shaped encoding-decoding network design is adopted, wherein the encoder backbone is ResNet32, which is used to receive the spliced optical flow map, the intrinsic reference layer and the contextual semantic features; Its decoder restores spatial resolution through skip connections and outputs a pixel-level pose prediction map.

9. The optical flow pose estimation method according to claim 1, wherein The step (5) specifically includes the following sub-steps: (5.1) Since the uncertainty map has been reduced to the same size as the spliced features, it is first necessary to upscale it to a pixel-wise translation uncertainty map and a rotation uncertainty map through two learnable convolutional layers, which represent the uncertainty distribution of each translation and rotation component, respectively, thereby providing independent uncertainty estimates for each dimension of camera translation and rotation; (5.2) The right half of the U-shaped network architecture is responsible for decoding and predicting the camera translation and rotation at the pixel level. Subsequently, it is fused with the uncertainty map output by the optical flow network to select the most robust local translation and rotation. At the same time, a global camera pose estimation network is added in parallel with the local camera pose estimation network at the encoder output. The global pose translation and rotation are directly predicted based on high-level features. Finally, the camera pose is obtained by fusing the local pose and the global pose in a weighted combination. (5.3) First, divide the camera translation and rotation as well as the pixel translation uncertainty map and rotation uncertainty map into small pixel blocks. Then, from each small pixel block, index the translation and rotation of the pixel with the lowest uncertainty. Based on the current uncertainty, use the Softmax function to calculate the weight of each pixel pose. Finally, a more robust local pose translation and rotation is obtained through weighted summation. The global pose prediction module takes the high-level features of the encoder as input, then integrates the global features through two independent fully connected layers to estimate the translation and rotation of the global pose, and then fuses them with the translation and rotation of the local pose to obtain the final camera pose. The weight of the global pose prediction value and the weight of the local pose prediction value are both 0.5.

Images