Visual positioning method and system based on multi-modal information fusion

Abstract

The invention discloses a visual positioning method and system based on multi-modal information fusion, and belongs to the field of computer vision. The method comprises the following steps: acquiring an RGB image and a depth image in the same scene, extracting semantic features from the RGB image through a multi-modal feature extraction network, and extracting structural features from the depth image; performing weight distribution and deep fusion on the semantic features through a feature balance module to obtain final multi-modal fusion features; predicting three-dimensional scene coordinates of each two-dimensional pixel point and uncertainty corresponding to the three-dimensional scene coordinates on the basis of the final multi-modal fusion features; and calculating the six-degree-of-freedom pose of the camera based on the predicted coordinates. According to the method, the problems of texture dependence and structure fuzziness in the existing single-mode visual positioning method are solved.

Description

Technical Field

[0001] The present invention belongs to the field of computer vision, and in particular, relates to a visual positioning method and system based on multimodal information fusion. Background Art

[0002] Visual localization is a key computer vision technique that enables 6-DOF camera pose estimation in a known environment from a query image.

[0003] In the traditional visual localization framework, the scene coordinate regression (SCoRe) method directly regresses the 3D scene coordinates with 1D uncertainty corresponding to each pixel in the query image by training a convolutional neural network (CNN). Compared with structure-based methods, the SCoRe method does not require a database and shows better localization performance on small-scale indoor scenes.

[0004] The SCoRe method has two serious problems: texture dependence and structural ambiguity, which significantly affect the localization effect.

[0005] Texture dependency. The SCoRe method relies on identifying unique features of objects to recover scene coordinates. However, the appearance features of different objects may be similar, which interferes with the network training process and causes the regressed scene coordinates to be unable to effectively distinguish these objects. To alleviate this problem, recent studies have proposed a variety of improvements. For example, FDANet uses information in low-level features to distinguish similar image patches; EAAINet extracts global affine information by simultaneously identifying a fixed number of similar and dissimilar image patches, thereby improving positioning accuracy. However, due to the inherent limitations of a single RGB modality, these methods still face the problem of decreased efficiency when processing a large number of similar texture image patches.

[0006] Structural ambiguity. The current unimodal SCoRe method uses RGB images as input to generate scene coordinates, and then uses the RANSAC-based PnP algorithm to estimate the camera pose. In this process, the RGB image only has semantic information, and the feature extraction stage lacks structural information, resulting in inaccurate scene coordinate predictions, which often contain a large number of noise points with high uncertainty. These noise points will reduce the accuracy of the RANSAC algorithm in the PnP solution, thereby affecting the final camera pose estimation.

[0007] In order to solve the above problems, the present application proposes a visual positioning method and system based on multimodal information fusion. Summary of the invention

[0008] The present invention aims to overcome the deficiencies of the prior art and proposes a visual positioning method and system based on multimodal information fusion to achieve the following objectives: solving the problems of texture dependence and structural ambiguity existing in the existing single-modal visual positioning method.

[0009] In order to achieve the above object, the technical solution adopted by the present invention is:

[0010] A visual positioning method based on multimodal information fusion, the method comprising:

[0011] Step S1, obtaining an RGB image and a depth image of the same scene, and extracting semantic features from the RGB image and structural features from the depth image through a multimodal feature extraction network;

[0012] Step S2: weighting and deeply fusing the semantic features and semantic features through a feature balancing module to obtain a final multimodal fusion feature;

[0013] Step S3: predicting the three-dimensional scene coordinates of each two-dimensional pixel and its corresponding uncertainty based on the final multimodal fusion features;

[0014] Step S4: Calculate the 6-DOF position and posture of the camera based on the coordinates predicted in step S3.

[0015] Preferably, in step S1, the multimodal feature extraction network includes a semantic feature extractor. Based on the semantic feature extractor, the RGB image I n ∈R 3×H×W First, the dimension is expanded and the resolution is reduced through the ResNet18 algorithm, and then the global context information is integrated through the PSPNet network, and finally the semantic features are obtained through three-layer upsampling. The formula is as follows:

[0016]

[0017] Among them, Φ R ResNet18 algorithm; Φ P represents the PSPNet network; Φ U Represents three-layer upsampling.

[0018] Preferably, in step S1, the multimodal feature extraction network includes a structural feature extractor. Based on the structural feature extractor, the depth image D is firstly n ∈R 1×H×W Convert to point cloud P according to camera model parameters n ∈R 3×N , and then downsample the point cloud to get a point cloud subset And store the indexes of Q sampling points in Corr, which is expressed by the following formula:

[0019]

[0020] Where D(·) represents the downsampling operation;

[0021] Then, the encoder part of the PointNet++ network is used to abstract the point cloud features layer by layer, and then the decoder part is used to interpolate layer by layer to finally obtain the structural features. The formula is as follows:

[0022]

[0023] Among them, Φ encoder represents the encoder part of the PointNet++ network; Φ decoder Represents the decoder part of the PointNet++ network.

[0024] Preferably, in step S2, the feature balancing module includes a point-by-point reweighting module, and the point-by-point reweighting module is used to jointly encode the semantic features and the structural features and generate weights for itself, and the process is as follows:

[0025] Semantic features through the index in Corr Downsample to obtain the same structural features Aligned Semantic Feature Sequences The formula is as follows:

[0026]

[0027] Where M(·) represents the flattening and alignment operations;

[0028] Then the structural features are respectively processed through convolutional layers and average pooling layers. and semantic feature sequences The encoded feature sequence is concatenated along the channel dimension and then activated by the Sigmoid function to generate the weight H c ∈R 1×Q ;

[0029] At the same time, define H D =1-H C ∈R I×Q ;

[0030] Apply the weight to each element of the feature sequence to generate the final reweighted feature sequence, which is expressed by the following formula:

[0031]

[0032] in, Represents element-wise multiplication; A sequence of heavily weighted features representing semantic features; A sequence of heavily weighted features representing structural features.

[0033] Preferably, in step S2, the feature balancing module includes a hierarchical fusion module. Based on the hierarchical fusion module, the reweighted feature sequences of the semantic features and structural features are encoded at different levels. At each level, the semantic features and structural features are fused together to generate multi-level fusion features through feature concatenation and weighted summation. These fusion features are then connected together to generate the final multimodal fusion features, which are expressed by the following formula:

[0034]

[0035] Among them, F rgbd represents the final multimodal fusion feature; F1, F2, and F3 represent the generated multi-level fusion features respectively; Indicates feature connection.

[0036] Preferably, in step S3, a linear coordinate regression layer is used to predict the three-dimensional scene coordinates of each two-dimensional pixel point and its corresponding uncertainty.

[0037] Preferably, step S4 includes: after obtaining the predicted coordinates through step S3, using the RANSAC algorithm to iteratively screen these coordinate points, and in each iteration process, retaining high-confidence coordinate points, that is, points whose uncertainty is less than or equal to a preset threshold, and removing low-confidence noise points, that is, points whose uncertainty is greater than the preset threshold.

[0038] Preferably, step S4 comprises: calculating the 6-DOF pose of the camera by combining the selected high-confidence coordinate points and the corresponding two-dimensional pixel positions through a PnP algorithm, which is expressed by the following formula:

[0039] T=[R n ∣t n ];

[0040] Among them, R represents the rotation matrix, t represents the translation vector, and T represents the position of the camera.

[0041] At the same time, the present invention also proposes a visual positioning system based on multimodal information fusion, the system includes a processor, and the processor is used to implement the above-mentioned visual positioning method based on multimodal information fusion through program execution.

[0042] The technical effects of the present invention are:

[0043] 1. Improve positioning accuracy: By fusing the semantic information of RGB images and the structural information of depth images, the present invention can more accurately regress scene coordinates, especially in complex scenes with similar or repeated textures, showing excellent positioning accuracy. Compared with traditional single-modality methods, the present invention significantly reduces positioning errors caused by texture dependence when processing complex indoor environments.

[0044] 2. Enhanced feature expression capability: The point-by-point reweighting and layered fusion technology of the present invention fully utilizes the complementarity of different modal information, dynamically adjusts feature weights at different distances and scales, and ensures the generation of highly distinguishable fusion features in diverse scenes. This enhanced feature expression capability enables the network to better adapt to different scene changes, thereby improving the accuracy of scene coordinate regression.

[0045] 3. Reduce noise interference: By introducing the RANSAC algorithm to filter the scene coordinates, the present invention can effectively remove noise points with high uncertainty and only retain coordinate points with high confidence. This screening mechanism reduces the interference of noise on pose estimation, thereby improving the stability and accuracy of camera pose.

[0046] 4. Stronger adaptability: The present invention enhances the adaptability of the network in different scenarios through the fusion of multimodal information and hierarchical processing of features. Whether it is a scene with rich textures or an environment with complex geometric structures, the method of the present invention can provide a stable positioning effect and show strong robustness. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Figure 1 The present invention is a flowchart of a visual positioning method based on multimodal information fusion according to an embodiment of the present invention. DETAILED DESCRIPTION

[0048] The following is a further detailed description of the specific implementation of the present invention by describing the embodiments with reference to the accompanying drawings, with the purpose of helping those skilled in the art to have a more complete, accurate and in-depth understanding of the inventive concept and technical solution of the present invention and to facilitate its implementation. In order to make the technical solution of the present invention clearer, the present invention is explained by the following embodiments.

[0049] A visual positioning method based on multimodal information fusion (MMFNet), combined with Figure 1 The method flow shown is explained.

[0050] Step S1: Obtain the RGB image and depth image of the same scene, and apply I n ∈R 3×H×W and D n ∈R 1 ×H×wIt is represented by , where the number of channels of the RGB image is 3, the number of channels of the depth image is 1, and the number of channels is usually represented by C; H×W represents the resolution, H represents the height, and W represents the width. Then, the semantic features are extracted from the RGB image and the structural features are extracted from the depth image through a multimodal feature extraction network.

[0051] Specifically, the multimodal feature extraction network of this embodiment includes a semantic feature extractor and a structural feature extractor.

[0052] Based on the semantic feature extractor, the RGB image I n ∈R 3×H×W First, the dimension is expanded and the resolution is reduced by the ResNet18 algorithm (or its variant), and then the global context information is integrated through the PSPNet network, and finally the semantic features are obtained through three-layer upsampling. The formula is as follows:

[0053]

[0054] Among them, Φ R ResNet18 algorithm; Φ P represents the PSPNet network; Φ U Represents three-layer upsampling. The ResNet18 algorithm is a classic deep convolutional neural network model, which is characterized by the introduction of residual blocks, which solves the problems of gradient vanishing and gradient explosion in deep convolutional neural networks; PSPNet (Pyramid Scene Parsing Network) is a deep learning architecture for image semantic segmentation. Its core idea is to use the pyramid pooling (Pyramid Pooling Module) module to perform pooling operations on the input feature map using pooling kernels of different scales to effectively obtain multi-scale context information. The ResNet18 algorithm and the PSPNet network are both commonly used by those skilled in the art, and this application will not be repeated here. This application mainly improves the accuracy of semantic feature extraction through the combination of the ResNet18 algorithm, the PSPNet network, and three-layer upsampling.

[0055] Based on the structural feature extractor, this embodiment first transforms the depth image D n ∈R 1×H×W Convert to point cloud P according to camera model parameters n ∈R 3×n , where the superscript 3 indicates that the number of channels (dimensions) is 3, N indicates the number of points, and the other formulas are similar. Then the point cloud is downsampled to obtain a point cloud subset The indexes of the Q sampling points are stored in Corr for subsequent alignment with RGB pixels, which can be expressed by the following formula:

[0056]

[0057] Where D(·) represents the downsampling operation.

[0058] Then, the encoder part of the PointNet++ network is used to abstract the point cloud features layer by layer, and then the decoder part is used to interpolate layer by layer to finally obtain the structural features. The formula is as follows:

[0059]

[0060] Among them, Φ encoder represents the encoder part of the PointNet++ network; Φ decoder Represents the decoder part of the PointNet++ network. The PointNet++ network is a prior art for those skilled in the art. In this embodiment, the application of the PointNet++ network enhances the model's ability to extract point cloud features, especially in complex three-dimensional structural scenes, thereby improving the accuracy of structural feature extraction.

[0061] Step S2: weighting and deeply fusing the semantic features and semantic features through a feature balancing module to obtain a final multimodal fusion feature.

[0062] Specifically, the feature balancing module of this embodiment includes a point-by-point reweighting module and a hierarchical fusion module.

[0063] The point-by-point reweighting module is used to jointly encode semantic features and structural features and generate weights for itself, that is, dynamically adjust feature weights based on the characteristic that the observation error of the depth image increases with distance. Specifically, semantic features are given higher weights in areas with longer distances, while structural features are given higher weights in areas with closer distances. This reweighting mechanism effectively combines the advantages of semantic information and structural information, and improves the adaptability of the network at different distances and scenarios. The process of point-by-point reweighting is as follows:

[0064] Semantic features through the index in Corr Downsample to obtain the same structural features Aligned Semantic Feature Sequences The formula is as follows:

[0065]

[0066] Here, M(·) represents the flattening and alignment operations.

[0067] Then the structural features are respectively processed through convolutional layers and average pooling layers. and semantic feature sequences The encoded feature sequence is concatenated along the channel dimension and then activated by the Sigmoid function to generate the weight H c ∈R 1×Q .

[0068] At the same time, define H D =1-H C ∈R I×Q To ensure the structural characteristics and semantic feature sequences contribution.

[0069] Apply the weight to each element of the feature sequence to generate the final reweighted feature sequence, which is expressed by the following formula:

[0070]

[0071] in, Represents element-wise multiplication; A sequence of heavily weighted features representing semantic features; A sequence of heavily weighted features representing structural features.

[0072] Based on the hierarchical fusion module, the weighted feature sequences of the semantic features and structural features are encoded at different levels (assuming that there are 3 layers in total in this embodiment). At each level, the semantic features and structural features are fused together to generate multi-level fusion features through feature splicing and weighted summation. These fusion features are then connected together to generate the final multimodal fusion features, which are expressed by the following formula:

[0073]

[0074] Among them, F rgbd represents the final multimodal fusion feature; F1, F2, and F3 represent the generated multi-level fusion features respectively; This hierarchical processing method enhances the complementarity between features and ensures that the optimal feature expression can be obtained in different scenarios.

[0075] The point-by-point reweighting and hierarchical fusion technology of the present invention fully utilizes the complementarity of different modal information, dynamically adjusts feature weights at different distances and scales, and ensures the generation of highly distinguishable fusion features in diverse scenes. This enhanced feature expression capability enables the network to better adapt to different scene changes, thereby improving the accuracy of scene coordinate regression.

[0076] By fusing the semantic information of RGB images and the structural information of depth images, the present invention can more accurately regress scene coordinates and enhance the adaptability in different scenes, especially in complex scenes with similar or repeated textures, showing excellent positioning accuracy and strong robustness. Compared with traditional single-modal methods, the present invention significantly reduces the positioning error caused by texture dependence when processing complex indoor environments.

[0077] Step S3: predicting the three-dimensional scene coordinates of each two-dimensional pixel and its corresponding uncertainty based on the final multimodal fusion features.

[0078] Specifically, this embodiment uses a linear coordinate regression layer to predict the three-dimensional scene coordinates of each two-dimensional pixel point and its corresponding uncertainty

[0079] Step S4: Calculate the 6-DOF position and posture of the camera based on the coordinates predicted in step S3.

[0080] After the predicted coordinates are obtained through step S3, the RANSAC algorithm (RANSAC, Random Sample Consensus, is an iterative algorithm based on random sampling for estimating mathematical model parameters) is used to iteratively screen these coordinate points. In each iteration, high-confidence coordinate points, i.e., points with uncertainty less than or equal to a preset threshold, are retained, and low-confidence noise points, i.e., points with uncertainty greater than a preset threshold, are removed. This screening mechanism reduces the interference of noise on pose estimation, thereby improving the stability and accuracy of camera pose.

[0081] Then, the PnP algorithm (Perspective-n-Point, used to solve the object motion positioning problem from 3D to 2D point pairs, that is, the coordinates of the object in the world coordinate system and the pixel coordinates of the object in the camera's image plane are known, and the camera's pose is solved), combined with the selected high-confidence coordinate points and the corresponding two-dimensional pixel positions, calculates the camera's 6-DOF pose, which is expressed by the following formula:

[0082] T=[R n ∣t n ];

[0083] Among them, R represents the rotation matrix, t represents the translation vector, and T represents the camera pose. This process is iteratively optimized to make the predicted camera pose match the actual scene, thereby improving the accuracy and stability of pose estimation.

[0084] At the same time, the present invention also proposes a visual positioning system based on multimodal information fusion, the system includes a processor, and the processor is used to implement the above-mentioned visual positioning method based on multimodal information fusion through program execution.

[0085] The present invention is described above by way of example in conjunction with the accompanying drawings. Obviously, the specific implementation of the present invention is not limited to the above-mentioned methods. As long as various non-substantial improvements are made using the method concept and technical solution of the present invention; or the above concept and technical solution of the present invention are directly applied to other occasions without improvement, they are all within the protection scope of the present invention.

Claims

1. A visual positioning method based on multimodal information fusion, characterized in that: The method comprises: Step S1, obtaining an RGB image and a depth image of the same scene, and extracting semantic features from the RGB image and structural features from the depth image through a multimodal feature extraction network; Step S2: weighting and deeply fusing the semantic features and semantic features through a feature balancing module to obtain a final multimodal fusion feature; Step S3: predicting the three-dimensional scene coordinates of each two-dimensional pixel and its corresponding uncertainty based on the final multimodal fusion features; Step S4: Calculate the 6-DOF position and posture of the camera based on the coordinates predicted in step S3.

2. The visual positioning method based on multimodal information fusion according to claim 1, characterized in that: In step S1, the multimodal feature extraction network includes a semantic feature extractor. Based on the semantic feature extractor, the RGB image I n ∈R 3×H×W First, the dimension is expanded and the resolution is reduced through the ResNet18 algorithm, and then the global context information is integrated through the PSPNet network, and finally the semantic features are obtained through three-layer upsampling. The formula is as follows: Among them, Φ R ResNet18 algorithm; Φ P represents the PSPNet network; Φ U Represents three-layer upsampling.

3. A visual positioning method based on multimodal information fusion according to claim 1 or 2, characterized in that: In step S1, the multimodal feature extraction network includes a structural feature extractor. Based on the structural feature extractor, the depth image D is firstly n ∈R 1×H×W Convert to point cloud P according to camera model parameters n ∈R 3×N , and then downsample the point cloud to get a point cloud subset And store the indexes of Q sampling points in Corr, which is expressed by the following formula: Where D(·) represents the downsampling operation; Then, the encoder part of the PointNet++ network is used to abstract the point cloud features layer by layer, and then the decoder part is used to interpolate layer by layer to finally obtain the structural features. The formula is as follows: Among them, Φ encoder represents the encoder part of the PointNet++ network; Φ decoder Represents the decoder part of the PointNet++ network.

4. The visual positioning method based on multimodal information fusion according to claim 3, characterized in that: In step S2, the feature balancing module includes a point-by-point reweighting module, which is used to jointly encode semantic features and structural features and generate weights for itself, and the process is as follows: Semantic features through the index in Corr Downsample to obtain the same structural features Aligned Semantic Feature Sequences The formula is as follows: Where M(·) represents the flattening and alignment operations; Then the structural features are respectively processed through convolutional layers and average pooling layers. and semantic feature sequences The encoded feature sequence is concatenated along the channel dimension and then activated by the Sigmoid function to generate the weight H c ∈R 1×Q ; At the same time, define H D =1-H C ∈R I×Q ; Apply the weight to each element of the feature sequence to generate the final reweighted feature sequence, which is expressed by the following formula: in, Represents element-wise multiplication; A reweighted feature sequence representing semantic features; A sequence of heavily weighted features representing structural features.

5. The visual positioning method based on multimodal information fusion according to claim 4, characterized in that: In step S2, the feature balancing module includes a hierarchical fusion module. Based on the hierarchical fusion module, the weighted feature sequences of the semantic features and structural features are encoded at different levels. At each level, the semantic features and structural features are fused together to generate multi-level fusion features through feature concatenation and weighted summation. These fusion features are then connected together to generate the final multimodal fusion features, which are expressed by the following formula: Among them, F rgbd represents the final multimodal fusion feature; F1, F2, and F3 represent the generated multi-level fusion features respectively; Indicates feature connection.

6. The visual positioning method based on multimodal information fusion according to claim 5, characterized in that: In step S3, a linear coordinate regression layer is used to predict the three-dimensional scene coordinates of each two-dimensional pixel and its corresponding uncertainty.

7. The visual positioning method based on multimodal information fusion according to claim 6, characterized in that: The step S4 includes: after obtaining the predicted coordinates through the step S3, using the RANSAC algorithm to iteratively screen these coordinate points, and in each iteration process, retaining high-confidence coordinate points, that is, points whose uncertainty is less than or equal to a preset threshold, and removing low-confidence noise points, that is, points whose uncertainty is greater than the preset threshold.

8. The visual positioning method based on multimodal information fusion according to claim 7, characterized in that: The step S4 includes: calculating the 6-DOF pose of the camera by combining the selected high-confidence coordinate points and the corresponding two-dimensional pixel positions through the PnP algorithm, which is expressed by the following formula: T=[R n ∣t n ]; Among them, R represents the rotation matrix, t represents the translation vector, and T represents the position of the camera.

9. A visual positioning system based on multimodal information fusion, characterized in that: The system includes a processor, and the processor is used to implement a visual positioning method based on multimodal information fusion according to any one of claims 1 to 8 through program execution.

Images