A visual positioning method based on pose space texture continuity mapping

Abstract

This invention discloses a visual localization method based on pose space texture continuity mapping, belonging to the fields of robotics and computer vision. In the mapping stage, image intrinsic parameters, pose, and sparse point cloud are first obtained through sparse reconstruction. Image features are extracted using a pre-trained model, and pose encoding and decoding models are trained to ensure consistency between pose encoding and image features in spatial metrics. Simultaneously, the decoding model outputs a multi-peak distribution to model pose ambiguity. Subsequently, a mapping model from image features to pose encoding is trained and optimized end-to-end to construct a scene map. In the localization stage, features of the image to be localized are extracted, pose encoding is obtained through the mapping model, and the pose distribution is output by the decoding model. A unique location or multiple candidate poses are determined based on the number of peaks in the distribution. This invention effectively improves the robustness and accuracy of robot localization in repetitive texture scenes by maintaining the continuity alignment between the pose space and texture space.

Description

TECHNICAL FIELD

[0001] The present application relates to a visual positioning method based on pose space texture continuity mapping, which studies visual positioning problems, belongs to the field of robot technology and computer vision BACKGROUND

[0002] The visual positioning problem studies how to use computer vision technology to solve the pose estimation problem of the robot body in the field of mobile robots, that is, the "where am I" problem of the robot. Traditional visual positioning methods are generally based on explicit feature extraction, matching and geometric estimation, which have problems such as large map representation occupation and difficult positioning in weak texture scenes. The learning-based visual positioning method abstracts the visual positioning as a mapping learning problem, learns the mapping relationship from image to pose, which can simplify the positioning process and reduce the map storage, but in the repeated texture or weak texture scene, this mapping itself has ambiguity (one-to-many mapping), how to solve this problem is very crucial.

[0003] The two research works closest to this paper are:

[0004] Bui, Mai et al. proposed to directly learn a multimodal distribution to solve the scene ambiguity problem, but this method is too direct and easy to overfit, this paper proposes to decompose this mapping problem into a multi-stage mapping, which can improve the effect of visual positioning mapping learning in repeated texture scenes.

[0005] Yoli Shavit et al. proposed an image, feature and pose encoding and decoding method [2], which has certain structural similarity with the method in this paper, but the pose encoding and decoding of this method does not solve the continuity contradiction of space, but solves the space occupation of image storage and subsequent optimization problem, and cannot solve the repeated texture problem in visual positioning.

[0006] [1] Bui, Mai and Birdal, Tolga and Deng, Haowen and Albarqouni, Shadi and Guibas, Leonidas and Ilic, Slobodan and Navab, Nassir, Multimodal Inference for 6D Camera Relocalization and Object Pose Estimation, ECCV 2020;

[0007] [2] Yoli Shavit and Yosi Keller, Camera Pose Auto-Encoders for Improving Pose Regression, ECCV 2022. SUMMARY

[0008] The present application mainly solves the problem of robot positioning in a repetitive texture scene. The essence of the positioning difficulty is the contradiction between the continuous pose space and the continuous texture representation. The present application solves this problem by learning a pose space mapping with continuous texture.

[0009] The technical scheme adopted by the present application is a visual positioning method based on pose space texture continuity mapping, comprising the following steps:

[0010] Mapping stage:

[0011] Step S1: Collect scene images and obtain the intrinsic parameters, pose and sparse point cloud of each frame of image based on sparse reconstruction technology;

[0012] Step S2: Extract image features using a pre-trained image representation model to construct an image feature space;

[0013] Step S3: Train the pose encoding model and the pose decoding model so that the pose encoding output by the pose encoding model is consistent with the image features of the corresponding image in terms of spatial measurement, and the pose decoding model outputs a multi-peak distribution of poses;

[0014] Step S4: Train a mapping model from image features to pose encoding to establish a mapping relationship between images and pose encoding;

[0015] Step S5: Train an image-to-pose prediction model end-to-end based on the image representation model, pose encoding model, pose decoding model and mapping model;

[0016] Positioning stage:

[0017] Step T1: Obtain the image to be positioned and extract its image features using the image representation model;

[0018] Step T2: Input the image features into the mapping model to obtain the corresponding pose encoding;

[0019] Step T3: Input the pose encoding into the pose decoding model to obtain the pose distribution of the robot in the current scene;

[0020] Step T4: Output the positioning result according to the number of peaks of the pose distribution. If it is a single-peak distribution, output a unique position, and if it is a multi-peak distribution, output multiple candidate poses.

[0021] Further, in step S3, the pose encoding model and the pose decoding model are trained by constraining the metric consistency between the pose encoding and the image features to ensure that the continuity of the pose space and the continuity of the texture space are aligned.

[0022] Furthermore, the pose decoding model adopts a multi-peak distribution modeling method to output multiple means and variances, which are used to describe multiple pose assumptions that the robot may have in repetitive texture scenes.

[0023] Furthermore, the image representation model is a deep neural network model pre-trained on a large-scale dataset, used to extract image features that are discriminative and robust.

[0024] Furthermore, the sparse reconstruction techniques include, but are not limited to, SLAM or Structure from Motion methods, used to obtain the intrinsic parameters, pose, and sparse point cloud of the scene.

[0025] A visual localization system based on pose space texture continuity mapping, comprising:

[0026] The image acquisition module is used to acquire scene images during the mapping phase and to acquire images to be located during the localization phase.

[0027] The map building module is used to execute the mapping stage steps S1 to S5 to generate a scene map.

[0028] The positioning module is used to execute the positioning stage steps T1 to T4 and output the robot's pose distribution.

[0029] The storage module is used to store image representation models, pose encoding models, pose decoding models, mapping models, and scene maps.

[0030] Furthermore, when the positioning module outputs a multi-peak distribution, it sends multiple candidate poses to the robot's decision module, and the robot eliminates ambiguity through movement or sensor fusion.

[0031] A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the visual localization method based on pose space texture continuity mapping.

[0032] A mobile robot includes the aforementioned visual positioning system for robust visual positioning in repetitive textured scenes.

[0033] The effects that this invention can achieve are:

[0034] Visual positioning is a key problem that needs to be solved in applications such as mobile robots, and it is also a core scientific problem in the field of embodied intelligence. Effectively providing position in repetitive texture scenes can enable robots to perform tasks robustly and stably in subsequent practical applications, which is also a prerequisite for robots to actively explore their environment. Attached Figure Description

[0035] Figure 1 This is a flowchart of the visual positioning training phase.

[0036] Figure 2 This is a process diagram of the visual positioning stage. Detailed Implementation

[0037] The present invention will now be described in detail with reference to the accompanying drawings and embodiments.

[0038] A visual localization method based on pose space texture continuity mapping is presented. The visual localization problem mainly includes two steps: first, mapping, i.e., constructing and representing a scene map; second, localization, i.e., estimating the robot's pose in the scene based on image information. In this invention, we will introduce the mapping scheme and the localization scheme respectively. In the mapping stage, the system input is uniformly acquired scene images, and the output is a map representation of the scene; in the localization stage, the input is the scene map and the image to be localized, and the output is the pose of the image in the scene map.

[0039] Mapping:

[0040] Based on existing sparse reconstruction techniques (SLAM / Structure from Motion), we obtain the intrinsic parameters (K), pose (P), and sparse point cloud of the scene image.

[0041] Mapping training:

[0042] Image features can be obtained using pre-trained image representation models (such as image retrieval models trained on large datasets). This ensures that similar images are close in distance in the representation space, and dissimilar images are far apart in the representation space. For image I features;

[0043] Position encoding model training (core): Training the pose encoding model based on the representation model. and decoding model ,

[0044] To ensure the continuity of the mapping, the pose encoding needs to have a similar spatial metric to the corresponding image features. .

[0045] Since images observed from different poses may have the same texture, the pose decoding model is a multi-peak distribution model. The output is the mean and variance of a multimodal distribution.

[0046] Representation remapping: Constructing and training an image-to-pose encoding mapping model based on the image pre-trained representation model and pose encoding / decoding model. ;

[0047] End-to-end training: Based on the above training, train a model of the image to the pose: .

[0048] position:

[0049] Extracting the representation f of image I using an image representation model

[0050] The pose code e corresponding to the representation f is calculated using a feature mapping model.

[0051] The pose decoding model is used to obtain the pose distribution p. If the distribution is unimodal, it means that the position is determined. If it is multimodal, it means that the robot may be in an ambiguous position. Multiple poses must be returned for the upper-level algorithm to decide how to eliminate ambiguity.

[0052] The implementation process requires two steps: preliminary map construction and online real-time positioning.

[0053] In the initial map building phase, implementers need to use ordinary cameras or panoramic cameras (handheld or mounted on a mobile device) to collect data evenly in the scene where the robot needs to operate, and then build the map based on the mapping steps proposed in this method.

[0054] During the localization phase: The robot needs to be equipped with a camera and a computing unit (preferably with a GPU). After the camera acquires an image, it is transmitted to the computing unit. The computing unit executes the localization steps in sequence and returns the robot's localization distribution. If the distribution is a single-peak distribution, it means that the robot's pose is determined. If it is a multi-peak distribution, it means that the robot is in a repetitive scene area and needs to obtain additional information through its own movement (not within the scope of this patent description) to determine its pose.

Claims

1. A visual localization method based on pose space texture continuity mapping, characterized in that, Includes the following steps: Mapping phase: Step S1: Acquire scene images and obtain the intrinsic parameters, pose, and sparse point cloud of the scene for each frame based on sparse reconstruction technology. Step S2: Extract image features using a pre-trained image representation model and construct an image feature space; Step S3: Train the pose encoding model and the pose decoding model so that the pose encoding output by the pose encoding model is consistent with the image features of the corresponding image in terms of spatial metric, and the pose decoding model outputs a multi-peak distribution of the pose. Step S4: Train the mapping model of image features to pose encoding, and build the mapping relationship between image and pose encoding; Step S5: Based on the image representation model, pose encoding model, pose decoding model and mapping model, train the image to pose prediction model end-to-end; Positioning phase: Step T1: Obtain the image to be located and extract its image features using the image representation model; Step T2: Input the image features into the mapping model to obtain the corresponding pose encoding; Step T3: Input the pose encoding into the pose decoding model to obtain the pose distribution of the robot in the current scene; Step T4: Output the localization result based on the number of peaks in the pose distribution. If it is a single-peak distribution, output a unique position; if it is a multi-peak distribution, output multiple candidate poses.

2. The visual localization method based on pose space texture continuity mapping according to claim 1, characterized in that, In step S3, the pose encoding model and the pose decoding model are trained by constraining the consistency of the metric between pose encoding and image features to ensure that the continuity of the pose space is aligned with the continuity of the texture space.

3. The visual localization method based on pose space texture continuity mapping according to claim 1, characterized in that, The pose decoding model adopts a multi-peak distribution modeling method, outputting multiple means and variances to describe multiple pose assumptions that the robot may have in repetitive texture scenes.

4. The visual localization method based on pose space texture continuity mapping according to claim 1, characterized in that, The image representation model is a deep neural network model pre-trained on a large-scale dataset, used to extract image features that are discriminative and robust.

5. The visual localization method based on pose space texture continuity mapping according to claim 1, characterized in that, The sparse reconstruction techniques include, but are not limited to, SLAM or StructurefromMotion methods, used to obtain the intrinsic parameters, pose, and sparse point cloud of the scene.

6. A visual positioning system based on pose space texture continuity mapping, characterized in that, include: The image acquisition module is used to acquire scene images during the mapping phase and to acquire images to be located during the localization phase. The map building module is used to execute the mapping stage steps S1 to S5 as described in claim 1 to generate a scene map; The positioning module is used to execute positioning stage steps T1 to T4 as described in claim 1 and output the robot's pose distribution. The storage module is used to store image representation models, pose encoding models, pose decoding models, mapping models, and scene maps.

7. The visual positioning system according to claim 6, characterized in that, When the positioning module outputs a multi-peak distribution, it sends multiple candidate poses to the robot's decision module, which then eliminates ambiguity through movement or sensor fusion.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the visual localization method based on pose space texture continuity mapping as described in any one of claims 1 to 5.

9. A mobile robot, characterized in that, Includes the visual positioning system as described in claim 6 or 7, for robust visual positioning in repetitive texture scenes.

Images