Relocation method, device, electronic device and computer-readable storage medium

Abstract

The present application relates to a repositioning method, device, electronic device and computer-readable storage medium, and relates to the field of computer technology. The method includes: obtaining a current observation image, and extracting features from the current observation image to obtain a dense feature map, and then determining the residual distance between each point descriptor in the dense feature map and each cluster center in a pre-created scene map, and determining a feature map C based on the residual distance, each cluster center is obtained by clustering each point in the pre-created scene map according to the scene, and then generating a confidence map corresponding to the current observation image based on the dense feature map, and then determining a region descriptor corresponding to the current observation image based on the confidence map and the feature map C corresponding to the current observation image, and then determining a scene image that meets a first preset condition from the pre-created scene map based on the region descriptor corresponding to the current observation image, and then repositioning based on the scene image that meets the first preset condition.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a relocation method, device, electronic device, and computer-readable storage medium. Background Art

[0002] In recent years, with the emergence of applications such as autonomous memory parking, smart logistics carts, restaurant robot delivery, and autonomous drone navigation, identifying previously visited scenes has become crucial. These applications often involve manually planning a correct movement path and creating a scene map during the first execution of a task (e.g., parking a car in a private parking space). During subsequent autonomous executions, the intelligent robot or self-driving car uses the currently observed scene to determine its current location within the scene map. It then autonomously follows the pre-planned path or navigates autonomously using the scene map to avoid obstacles. Therefore, relocalization—the process of determining the current location within the scene map—is crucial for these applications. In these scenarios, a significant amount of time may pass between the execution of the autonomous navigation task and the creation of the scene map, resulting in significant changes in the surrounding environment. For example, map creation may occur in the morning, while autonomous navigation begins at night; map creation may occur on a sunny day, while autonomous navigation begins on rainy, foggy, or snowy days; or even seasonal variations may occur, resulting in significant differences in the appearance of the scene observed by both systems. In addition, the scenarios of these applications are often very complex. For example, dynamic objects such as pedestrians and vehicles interfere with mapping and autonomous navigation, further increasing the difference in the appearance of the two observed scenes. These dynamic objects may even cause partial occlusion of the scene. At the same time, the repetition of some open scenes or objects with the same texture is also a major challenge, such as empty parking lots, similar design styles of different garages, and repeated and almost identical lamp poles and fences on the road.

[0003] Therefore, the inventors found during the research process that in these scenarios, relocalization directly through the scene map is more complex and has poor accuracy. Summary of the Invention

[0004] The purpose of this application is to provide a relocation method, device, electronic device and computer-readable storage medium to solve at least one of the above technical problems.

[0005] The above-mentioned invention objectives of this application are achieved through the following technical solutions:

[0006] In a first aspect, a relocation method is provided, comprising:

[0007] Acquire a current observation image and extract features from the current observation image to obtain a dense feature map; determine the residual distance between each point descriptor in the dense feature map and each cluster center in a pre-created scene map, and determine a feature map C based on the residual distance, wherein each cluster center is obtained by clustering each point in the pre-created scene map according to the scene;

[0008] Generating a confidence map corresponding to the current observation image based on the dense feature map;

[0009] Determining a region descriptor corresponding to the current observation image based on the confidence map corresponding to the current observation image and the feature map C;

[0010] Determining a scene image that meets a first preset condition from the pre-created scene map based on a region descriptor corresponding to the current observation image;

[0011] Repositioning is performed based on the scene image that meets the first preset condition.

[0012] In a possible implementation, determining the residual distance between each point descriptor in the dense feature map and each cluster center in the pre-created scene map includes:

[0013] Perform convolution processing based on the dense feature map to obtain a feature map F;

[0014] Determine the residual distance between each point descriptor in the feature map F and each cluster center in the pre-created scene map.

[0015] In a possible implementation, determining the region descriptor corresponding to the current observation image based on the confidence map corresponding to the current observation image and the feature map C includes:

[0016] Dividing the confidence map corresponding to the current observation image into regions;

[0017] Generate the minimum outer bounding box for each divided area;

[0018] Based on the confidence levels of each pixel in each minimum outer bounding box, the confidence levels of each region are determined.

[0019] Determine at least one region based on the confidence levels corresponding to the respective regions and in accordance with preset rules;

[0020] Matching the minimum outer bounding box of each region in the at least one region with the feature map C to obtain respective corresponding second regions;

[0021] Regularizing the descriptor at each position in the feature map C;

[0022] The descriptor means corresponding to each second region are calculated, and regularization processing is performed on each descriptor mean to obtain the region descriptors corresponding to each second region.

[0023] In another possible implementation, determining a scene image that meets a first preset condition from the pre-created scene map based on a region descriptor corresponding to the current observation image includes:

[0024] Determining, from the pre-created scene map, a set of region descriptors corresponding to each scene image, wherein the set of region descriptors includes at least one region descriptor;

[0025] performing region matching based on the region descriptors corresponding to the respective second regions and the region descriptor sets corresponding to the respective scene images, and performing region matching according to preset rules to obtain a region matching pair set;

[0026] Determining, based on the set of region matching pairs, spatial consistency scores corresponding to each scene image;

[0027] Based on the spatial consistency scores respectively corresponding to the scene images, scene images that meet a first preset condition are determined from the pre-created scene map.

[0028] In another possible implementation, determining the spatial consistency scores corresponding to the respective scene images based on the set of region matching pairs includes:

[0029] Based on the set of region matching pairs, determining the region center offsets corresponding to the respective region matching pairs;

[0030] Determining an average row offset and an average column offset based on the region center offsets respectively corresponding to the region matching pairs;

[0031] Based on the set of region matching pairs, the region center offsets, the average row offsets, and the column offsets corresponding to each region matching pair, a spatial consistency score corresponding to each scene image is determined. In another possible implementation, there are at least two scene images that meet the first preset condition; the method further includes:

[0032] Get the F descriptor set corresponding to each scene image;

[0033] Determining the similarity between each scene image and the current observation image based on the descriptor set F corresponding to each scene image and the feature map C;

[0034] Based on the similarities between the respective scene images and the current observation image, a scene image that meets a second preset condition is determined from the scene images that meet the first preset condition.

[0035] In another possible implementation, the F descriptor set corresponding to any scene image includes: at least one F descriptor;

[0036] Determining the similarity between any scene image and the current observation image based on the F descriptor set corresponding to any scene image and the feature map C includes:

[0037] Determine the distance between each F descriptor corresponding to any scene image and each pixel in the feature map C;

[0038] Based on the determined distance, determining the distance between each F descriptor corresponding to any one of the scene images and the feature map C;

[0039] Determining the similarity between each F descriptor and the feature map C based on the distance between each F descriptor corresponding to any one of the scene images and the feature map C;

[0040] Based on the similarity between each F descriptor and the feature map C, the similarity between the any scene image and the feature map C is determined as the similarity between the any scene image and the current observation image.

[0041] In another possible implementation, the relocalization based on the scene image that meets the first preset condition includes:

[0042] Repositioning is performed based on the scene image that meets the second preset condition.

[0043] In another possible implementation, the method further includes:

[0044] Normalizing each point descriptor in the feature map F to obtain a normalized result, wherein the feature map F is obtained by convolving the dense feature map;

[0045] Determining a global descriptor corresponding to the current observation image based on the normalization result and the feature map C;

[0046] Determining a scene image that meets a third preset condition from the pre-created scene map based on a global descriptor corresponding to the current observation image;

[0047] The determining of a scene image that meets a first preset condition from the pre-created scene map based on the region descriptor corresponding to the current observation image includes:

[0048] The scene image that meets the first preset condition is determined from the scene map that meets the third preset condition based on the region descriptor corresponding to the current observation image.

[0049] In another possible implementation, the relocalization based on the scene image that meets the second preset condition includes:

[0050] Clustering the scene images that meet the second preset condition based on the co-viewing relationship of the scene map to obtain at least two co-viewing clusters;

[0051] determining, from the at least two common-view clusters, a common-view cluster that satisfies a fourth preset condition based on similarities between each scene image in each common-view cluster and the current observation image;

[0052] Selecting, from the pre-created scene map, a 3D map point corresponding to each scene image in the common view cluster that meets the fourth preset condition, each 3D map point corresponds to an F descriptor;

[0053] Obtain the matching position of the 3D map point in the feature map C whose descriptor F satisfies the ratio test condition; convert the matching position to the image coordinate system corresponding to the current observation image to obtain the matching relationship between the 3D map point and the 2D map point in the current observation image;

[0054] Acquire the pose of a reference image, and determine an initial pose corresponding to the current observation image based on the pose of the reference image and the matching relationship;

[0055] Repositioning is performed based on the initial pose corresponding to the current observation image.

[0056] In another possible implementation, the repositioning based on the initial pose corresponding to the current observation image includes:

[0057] Projection step: projecting the 3D map point corresponding to each scene image in the common view cluster that meets the fourth preset condition into the image coordinate system corresponding to the current observation image according to the initial pose corresponding to the current observation image;

[0058] A screening step: screening out 3D map points whose projection points are within the current observation image to obtain a map point set; a coordinate conversion step: converting the projection point coordinates of each 3D map point in the map point set into the coordinate system of the H feature map of the current observation image to obtain an H descriptor corresponding to each 3D map point;

[0059] Acquisition step: acquiring an H' descriptor corresponding to each 3D map point in the map point set, wherein the H' descriptor is a descriptor corresponding to an H' feature map corresponding to each key frame image when creating a scene map;

[0060] Residual information determining step: determining residual information of each 3D map point in the map point set based on the H' descriptor corresponding to each 3D map point in the map point set and the H descriptor corresponding to each 3D map point;

[0061] Target minimization step: determining the target function based on the residual information of each 3D map point;

[0062] The relocalized pose is determined based on the objective function.

[0063] In another possible implementation, the determining of the repositioned posture based on the objective function includes: a posture updating step: minimizing the objective function to determine the updated posture; and / or a loop execution step: looping the step of using the updated posture as the initial posture corresponding to the current observation image, the projection step, the screening step, the coordinate conversion step, the acquisition step, the residual information determination step, the target minimization step and the posture updating step until the posture converges, and determining the posture obtained when the posture converges as the repositioned posture.

[0064] In a second aspect, a relocation device is provided, comprising:

[0065] A first acquisition module is used to acquire the current observation image;

[0066] A feature extraction module, configured to extract features from the current observation image to obtain a dense feature map;

[0067] A first determination module is configured to determine a residual distance between each point descriptor in the dense feature map and each cluster center in the pre-created scene map, and determine a feature map C based on the residual distance, wherein each cluster center is obtained by clustering each point in the pre-created scene map according to the scene;

[0068] A generating module, configured to generate a confidence map corresponding to the current observation image based on the dense feature map;

[0069] A second determination module is configured to determine a region descriptor corresponding to the current observation image based on the confidence map corresponding to the current observation image and the feature map C;

[0070] A third determining module, configured to determine a scene image that meets a first preset condition from the pre-created scene map based on a region descriptor corresponding to the current observation image;

[0071] A repositioning module is used to perform repositioning based on the scene image that meets the first preset condition.

[0072] In a possible implementation, when determining the residual distance between each point descriptor in the dense feature map and each cluster center in the pre-created scene map, the first determination module is specifically configured to:

[0073] Perform convolution processing based on the dense feature map to obtain a feature map F;

[0074] Determine the residual distance between each point descriptor in the feature map F and each cluster center in the pre-created scene map.

[0075] In a possible implementation, when the second determination module determines the region descriptor corresponding to the current observation image based on the confidence map corresponding to the current observation image and the feature map C, it is specifically configured to:

[0076] Dividing the confidence map corresponding to the current observation image into regions;

[0077] Generate the minimum outer bounding box for each divided area;

[0078] Based on the confidence levels of each pixel in each minimum outer bounding box, the confidence levels of each region are determined.

[0079] Determine at least one region based on the confidence levels corresponding to the respective regions and in accordance with preset rules;

[0080] Matching the minimum outer bounding box of each region in the at least one region with the feature map C to obtain respective corresponding second regions;

[0081] Regularizing the descriptor at each position in the feature map C;

[0082] The descriptor means corresponding to each second region are calculated, and regularization processing is performed on each descriptor mean to obtain the region descriptors corresponding to each second region.

[0083] In another possible implementation, when the third determination module determines the scene image that meets the first preset condition from the pre-created scene map based on the region descriptor corresponding to the current observation image, it is specifically configured to:

[0084] Determining, from the pre-created scene map, a set of region descriptors corresponding to each scene image, wherein the set of region descriptors includes at least one region descriptor;

[0085] performing region matching based on the region descriptors corresponding to the respective second regions and the region descriptor sets corresponding to the respective scene images, and performing region matching according to preset rules to obtain a region matching pair set;

[0086] Determining, based on the set of region matching pairs, spatial consistency scores corresponding to each scene image;

[0087] Based on the spatial consistency scores respectively corresponding to the scene images, scene images that meet a first preset condition are determined from the pre-created scene map.

[0088] In another possible implementation, when determining the spatial consistency scores corresponding to the respective scene images based on the set of region matching pairs, the third determination module is specifically configured to:

[0089] Based on the set of region matching pairs, determining the region center offsets corresponding to the respective region matching pairs;

[0090] Determining an average row offset and an average column offset based on the region center offsets respectively corresponding to the region matching pairs;

[0091] Based on the set of region matching pairs, the region center offsets, the average row offsets, and the column offsets corresponding to each of the region matching pairs, a spatial consistency score corresponding to each of the scene images is determined. In another possible implementation, the number of scene images that meet the first preset condition is at least two; the device further includes: a second acquisition module, a fourth determination module, and a fifth determination module, wherein the second acquisition module is configured to acquire the F descriptor set corresponding to each of the scene images;

[0092] The fourth determination module is configured to determine the similarity between each scene image and the current observation image based on the descriptor set F corresponding to each scene image and the feature map C;

[0093] The fifth determination module is configured to determine, based on the similarities between each of the scene images and the current observation image, a scene image that meets the second preset condition from the scene images that meet the first preset condition. In another possible implementation, the F descriptor set corresponding to any scene image includes: at least one F descriptor;

[0094] When determining the similarity between any scene image and the current observation image based on the F descriptor set corresponding to any scene image and the feature map C, the fourth determination module is specifically configured to:

[0095] Determine the distance between each F descriptor corresponding to any scene image and each pixel in the feature map C;

[0096] Based on the determined distance, determining the distance between each F descriptor corresponding to any one of the scene images and the feature map C;

[0097] Determining the similarity between each F descriptor and the feature map C based on the distance between each F descriptor corresponding to any one of the scene images and the feature map C;

[0098] Based on the similarity between each F descriptor and the feature map C, the similarity between the any scene image and the feature map C is determined as the similarity between the any scene image and the current observation image.

[0099] In another possible implementation, when performing relocation based on the scene image that meets the first preset condition, the relocation module is specifically configured to:

[0100] Repositioning is performed based on the scene image that meets the second preset condition.

[0101] In another possible implementation, the apparatus further includes: a normalization processing module, a sixth determination module, and a seventh determination module; wherein,

[0102] The normalization processing module is used to perform normalization processing on each point descriptor in the feature map F to obtain a normalized result, wherein the feature map F is obtained by convolving the dense feature map;

[0103] The sixth determination module is configured to determine a global descriptor corresponding to the current observation image based on the normalization result and the feature map C;

[0104] The seventh determining module is configured to determine a scene image that meets a third preset condition from the pre-created scene map based on a global descriptor corresponding to the current observation image;

[0105] Wherein, when the third determination module determines the scene image that meets the first preset condition from the pre-created scene map based on the region descriptor corresponding to the current observation image, it is specifically configured to:

[0106] The scene image that meets the first preset condition is determined from the scene map that meets the third preset condition based on the region descriptor corresponding to the current observation image.

[0107] In another possible implementation, when performing relocation based on the scene image that meets the second preset condition, the relocation module is specifically configured to:

[0108] Clustering the scene images that meet the second preset condition based on the co-viewing relationship of the scene map to obtain at least two co-viewing clusters;

[0109] determining, from the at least two common-view clusters, a common-view cluster that satisfies a fourth preset condition based on similarities between each scene image in each common-view cluster and the current observation image;

[0110] Selecting, from the pre-created scene map, a 3D map point corresponding to each scene image in the common view cluster that meets the fourth preset condition, each 3D map point corresponds to an F descriptor;

[0111] Obtain the matching position of the 3D map point in the feature map C whose descriptor F satisfies the ratio test condition; convert the matching position to the image coordinate system corresponding to the current observation image to obtain the matching relationship between the 3D map point and the 2D map point in the current observation image;

[0112] Acquire the pose of a reference image, and determine an initial pose corresponding to the current observation image based on the pose of the reference image and the matching relationship;

[0113] Repositioning is performed based on the initial pose corresponding to the current observation image.

[0114] In another possible implementation, the relocalization module includes: a projection unit, a screening unit, a coordinate conversion unit, an acquisition unit, a residual information determination unit, a target minimization unit, and a determination unit, wherein the projection unit is configured to project the 3D map point corresponding to each scene image in the common view cluster that meets the fourth preset condition into the image coordinate system corresponding to the current observation image according to the initial pose corresponding to the current observation image;

[0115] A screening unit, configured to screen out 3D map points whose projection points are within the current observation image to obtain a map point set;

[0116] a coordinate conversion unit for converting the projection point coordinates of each 3D map point in the map point set into the coordinate system of the H feature map of the current observation image to obtain the H descriptor corresponding to each 3D map point; an acquisition unit for acquiring the H' descriptor corresponding to each 3D map point in the map point set, wherein the H' descriptor is the descriptor corresponding to the H' feature map corresponding to each key frame image when creating the scene map; and a residual information determination unit for determining the residual information of each 3D map point in the map point set based on the H' descriptor corresponding to each 3D map point in the map point set and the H descriptor corresponding to each 3D map point;

[0117] a target minimization unit, configured to determine a target function based on the residual information of each 3D map point;

[0118] A determination unit is used to determine the posture after relocation based on the objective function.

[0119] In another possible implementation, the determination unit includes: a posture update subunit and / or a loop execution subunit, wherein:

[0120] A posture updating subunit is configured to minimize the objective function and determine the updated posture; a loop execution subunit is configured to loop execute the steps of using the updated posture as the initial posture corresponding to the current observation image, the operations performed by the projection unit, the screening unit, the coordinate conversion unit, the acquisition unit, the residual information determination unit, the target minimization unit, and the posture updating unit until the posture converges, and the posture obtained when the posture converges is determined as the posture after repositioning. In a third aspect, an electronic device is provided, which includes:

[0121] one or more processors;

[0122] Memory;

[0123] One or more applications, wherein one or more applications are stored in a memory and configured to be executed by one or more processors, and the one or more programs are configured to: perform operations corresponding to the relocation method shown in any possible implementation of the first aspect.

[0124] In a fourth aspect, a computer-readable storage medium is provided, wherein the storage medium stores at least one instruction, at least one program, code set or instruction set, and the at least one instruction, at least one program, code set or instruction set is loaded and executed by a processor to implement the relocation method as shown in any possible implementation method of the first aspect.

[0125] In summary, this application includes at least one of the following beneficial technical effects:

[0126] The present application provides a repositioning method, device, electronic device and computer-readable storage medium. Compared with the related art, in the present application, a confidence map corresponding to the current observation image is generated through a dense feature map, and then the residual distance between each point descriptor in the dense feature map and each cluster center in the pre-created scene map is determined, and the feature map C is determined based on the residual distance. Based on the feature map C and the confidence map, the region descriptor corresponding to the current observation image can be determined, that is, the scene map that meets the first preset condition can be determined from the pre-created scene map through the region descriptor, so that repositioning can be performed in the scene map that meets the first preset condition. That is, in the present application, high-level semantic features of the current observation image can be obtained to screen out scene images with higher matching degrees from the pre-created scene map, and then repositioning is performed based on the scene images with higher matching degrees, thereby reducing the complexity of repositioning and improving the accuracy of repositioning. BRIEF DESCRIPTION OF THE DRAWINGS

[0127] Figure 1 This is an architecture diagram of the end-to-end relocation system in an embodiment of the present application;

[0128] Figure 2 A schematic flow chart of a relocation method according to an embodiment of the present application;

[0129] Figure 3 A schematic diagram of the structure of a relocation device according to an embodiment of the present application;

[0130] Figure 4 This is a schematic diagram of the device structure of an electronic device in an embodiment of the present application. DETAILED DESCRIPTION

[0131] The present application is further described in detail below with reference to the accompanying drawings.

[0132] This specific embodiment is merely an explanation of the present application and is not a limitation of the present application. After reading this specification, those skilled in the art may make non-creative modifications to the present embodiment as needed, but as long as they are within the scope of the claims of the present application, they are protected by the patent law.

[0133] To make the purpose, technical solutions, and advantages of the embodiments of this application more clear, the technical solutions in the embodiments of this application will be clearly and completely described below in conjunction with the drawings in the embodiments of this application. Obviously, the described embodiments are part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0134] In this document, the term "and / or" simply describes a relationship between related objects, indicating that three possible relationships exist. For example, "A and / or B" can represent: A exists alone, A and B exist simultaneously, or B exists alone. Furthermore, the character " / " in this document, unless otherwise specified, generally indicates an "or" relationship between the related objects.

[0135] An embodiment of the present application provides a repositioning method. In the embodiment of the present application, the repositioning method may include: a process of vision-based scene retrieval and a process of repositioning based on the retrieved image, wherein the main purpose of vision-based scene retrieval from a pre-created scene map is to find observation information (images or video sequences) at the same geographical location when establishing the scene map based on current observation information; repositioning based on the retrieved image is to reposition based on the observation information at the same geographical location obtained after retrieval.

[0136] There are three main differences between vision-based scene retrieval and general image retrieval / video retrieval:

[0137] 1. The primary similarity metric for general image and video retrieval is based on factors such as "whether objects are of the same category" and "whether they have similar appearances." However, for vision-based scene retrieval, the primary metric is "whether objects are located in the same geographic location." Even if external factors such as weather and seasons cause significant differences in appearance, as long as the objects are located close enough, the similarity should be high.

[0138] 2. General image retrieval / video retrieval mainly targets foreground objects in images, while vision-based scene retrieval mainly targets background areas in images;

[0139] 3. General image retrieval / video retrieval can be performed offline, while vision-based scene retrieval is often used in fields with strong real-time requirements, such as relocalization and loop detection in SLAM. Therefore, in addition to requiring the scene retrieval algorithm to have low complexity, it is also necessary to have an efficient global representation of the observation information (image or video sequence) to make it easier to calculate and store, such as converting the observation information into a vector or a matrix.

[0140] In the related art, scene retrieval technology is mostly based on image global descriptors, such as the Network Vector of Locally Aggregated Descriptors (NetVlad), the Second-Order Loss and Attention for Image Retrieval (SOLAR) and other methods; the method of using image global descriptors for scene retrieval has the advantages of fast retrieval speed and high storage efficiency, but it ignores the detailed information in the image, and the recall rate of scene retrieval is low. Therefore, in the embodiment of the present application, image global descriptors, regional descriptors based on confidence maps, and point descriptors are used to construct a hierarchical scene retrieval architecture. Furthermore, in the related relocation technology, most of them are based on feature point matching, such as SuperPoint, HF-Net (Hierarchical Localization Network) and other methods. They are highly robust to changes in illumination and perspective, but due to the existence of multi-stage errors, the accuracy of pose calculation during relocation is low. Therefore, the patent of the present invention uses the LM algorithm to directly align and optimize the image, uniformly optimize the errors, and improve the accuracy during relocation.

[0141] Specifically, the embodiment of the present application aims to build an end-to-end relocalization system, which inputs the established scene map and observation image, and outputs the 6-degree-of-freedom (6DOF) pose of the observation image, which represents the current position in the scene map.

[0142] The end-to-end relocalization system is a layered relocalization system with a unified network architecture. Functionally, it consists of two parts: scene retrieval and pose optimization. Scene retrieval uses current observations to identify images that are geographically located at the same time as the scene map is constructed. This reduces computational complexity compared to traditional layered relocalization systems, enhances the coupling between scene retrieval and pose optimization, and results in lower latency.

[0143] Specifically, the scene retrieval obtains several frames of reference images that are in the same geographical location as the current observation from a large number of images in the scene map, and then the initial pose generation method of the embodiment of the present application quickly and accurately calculates an initial pose, and finally the pose optimization branch iteratively calculates the 6DOF pose. In other words, the layered scene retrieval is a coarse-to-fine retrieval architecture that comprehensively considers the global information, salient area information and local point information of the image, and pays attention not only to detailed texture features but also to high-level semantic features; the pose optimization method uses a deep learning network for image alignment, which is conducive to more robust feature extraction; these two parts are designed mainly to solve the problems of high re-positioning failure rate and inaccurate pose calculation when the scene appearance changes ((illumination changes, perspective changes, seasonal changes, weather changes, local occlusion, etc.)

[0144] Specifically, in the embodiment of the present application, the end-to-end relocation system architecture is as follows: Figure 1 As shown, for the current observed image, a Convolutional Neural Network (CNN) backbone network is first used. This CNN backbone network includes but is not limited to Unet, VGG, ResNet, RegNet, MobileNet, and AlexNet. For example, the Unet network is selected as the CNN backbone network. The overall process is divided into three branches. The first branch is the coarse-to-fine scene retrieval branch, which includes three parts: image global feature retrieval, salient region feature reordering, and point matching result reordering. Specifically, after passing through the CNN backbone network, a coarse-to-fine retrieval is performed through image global feature retrieval, salient region feature reordering, and point matching results to obtain a scene map that meets the first preset condition. The global feature retrieval includes: a 1×1 convolution process followed by a NetVLAD network layer to obtain a global descriptor; the salient region feature reordering includes: a 1×1 convolution process to also obtain a point-level descriptor; and the salient region feature reordering includes: a CNN process to obtain a confidence map, followed by a region-level descriptor.

[0145] Furthermore, the second branch is the confidence map branch, which adds L layers of convolutional neural network after the CNN backbone network, where L is between [1,5], and the confidence map obtained is Figure 1 On the one hand, it is used for salient region extraction (also referred to as the salient region feature reordering mentioned above), and on the other hand, it is used to weight the projection points in the pose optimization module. The third branch is the iterative pose optimization branch. The dense feature map is obtained by the CNN backbone network. The projection point features are aligned according to the calculated initial pose. The pose is then updated using the LM algorithm. The above steps are repeated until the pose converges, and the final output is the 6DOF pose, which represents the current position.

[0146] For the embodiment of the present application, the three parts in the coarse-to-fine scene retrieval branch, namely, image global feature retrieval, salient area feature reordering, and point matching result reordering, have increasing computational complexity (retrieval time) and retrieval accuracy. Therefore, the coarse-to-fine scene retrieval strategy designed in the embodiment of the present application can reduce system latency as much as possible while ensuring retrieval accuracy.

[0147] The embodiments of the present application are described in further detail below with reference to the accompanying drawings.

[0148] The embodiment of the present application provides a relocation method that can be applied to scenarios such as autonomous memory parking, intelligent logistics vehicles, restaurant intelligent robot food delivery, and autonomous drone cruising, and is executed by electronic devices, such as Figure 2 As shown, the method may include:

[0149] Step S201: Acquire a current observation image, and extract features from the current observation image to obtain a dense feature map.

[0150] For the embodiment of the present application, after the current observation image is acquired, a dense feature map is obtained after passing through the CNN backbone network.

[0151] Step S202: Determine the residual distance between each point descriptor in the dense feature map and each cluster center in the pre-created scene map.

[0152] Among them, each cluster center is obtained by clustering each point in the pre-created scene map according to the scene. For the embodiment of the present application, after obtaining the dense feature map, the L2 distance between each point descriptor in the dense feature map and each cluster center in the pre-created scene map can be calculated. In the embodiment of the present application, the distance between each point in the dense feature map (using its corresponding channel as a descriptor) and the K cluster centers can also include but is not limited to Manhattan distance, Euclidean distance, and Minkowski distance.

[0153] It should be noted that the pre-created scene map includes, but is not limited to, the vehicle motion trajectory during map creation, the confidence map of the keyframes during map creation, scene map points, the co-visibility relationship between keyframes, the visual relationship between keyframes and map points, the global descriptor, regional descriptor, F descriptor, and H descriptor of the observation image of the keyframe, etc. Step S203: Determine the feature map C based on the residual distance.

[0154] Step S204: Generate a confidence map corresponding to the current observation image based on the dense feature map.

[0155] Specifically, in the embodiment of the present application, after obtaining a feature map (dense feature map) through the CNN backbone network, a confidence map corresponding to the current observation image is then generated through the network model. For example, the feature map (dense feature map) is passed through a convolutional neural network to obtain a confidence map corresponding to the current observation image.

[0156] Step S205: Determine the region descriptor corresponding to the current observation image based on the confidence map and feature map C corresponding to the current observation image.

[0157] Step S206: Determine a scene image that meets a first preset condition from the pre-created scene map based on the region descriptor corresponding to the current observation image.

[0158] For the embodiment of the present application, the first preset condition can be input by the user or pre-set, which is not limited in the embodiment of the present application.

[0159] Step S207: repositioning based on the scene image that meets the first preset condition.

[0160] Furthermore, in addition to determining a scene image that satisfies the first preset condition from a pre-created scene map based on the region descriptor corresponding to the current observed image in step S206 above, a target scene image can also be determined from a pre-created scene map based on the region descriptor corresponding to the current observed image according to preset rules, and repositioning can be performed based on the target scene image. The manner of performing repositioning in the embodiments of the present application is described in detail in the following embodiments.

[0161] An embodiment of the present application provides a repositioning method. Compared with the related art, in the embodiment of the present application, a confidence map corresponding to the current observation image is generated through a dense feature map, and then the residual distance between each point descriptor in the dense feature map and each cluster center in the pre-created scene map is determined, and the feature map C is determined based on the residual distance. Based on the feature map C and the confidence map, the region descriptor corresponding to the current observation image can be determined, that is, the scene map that meets the first preset condition can be determined from the pre-created scene map through the region descriptor, so that repositioning can be performed in the scene map that meets the first preset condition. That is, in the embodiment of the present application, high-level semantic features of the current observation image can be obtained to screen out scene images with higher matching degrees from the pre-created scene map, and then repositioning is performed based on the scene images with higher matching degrees, thereby reducing the complexity of repositioning and improving the accuracy of repositioning.

[0162] Furthermore, in the embodiment of the present application, the specific implementation of step S201 is detailed in the above embodiment.

[0163] Furthermore, in order to better perform feature dimensionality reduction and extraction on each point in the feature map, the residual distance between each point descriptor in the dense feature map and each cluster center in the pre-created scene map is determined. Specifically, it can include: performing convolution processing based on the dense feature map to obtain a feature map F; determining the residual distance between each point descriptor in the feature map F and each cluster center in the pre-created scene map. That is to say, after obtaining the dense feature map through the backbone network, a 1×1 convolution can be performed to obtain the feature map F. Further, the L2 distance between each point in the feature map F (using its corresponding channel as a descriptor) and the K cluster centers is obtained to obtain the feature map C. In an embodiment of the present application, the cluster centers obtained after clustering the points in the pre-created scene map according to the scene are called K cluster centers.

[0164] Furthermore, in the embodiment of the present application, the specific implementation of step S203-step S204 is detailed in the above embodiment.

[0165] Specifically, in step S205, based on the confidence map and feature map C corresponding to the current observation image, the region descriptor corresponding to the current observation image is determined, which may specifically include: step S2051 (not shown in the figure), step S2052 (not shown in the figure), step S2053 (not shown in the figure), step S2054 (not shown in the figure), step S2055 (not shown in the figure), step S2056 (not shown in the figure) and step S2057 (not shown in the figure), wherein,

[0166] Step S2051: Divide the confidence map corresponding to the current observation image into regions.

[0167] For the embodiment of the present application, after obtaining the confidence map corresponding to the current observed image, the confidence map can be tested for connected domains, and the flooding method can be used to generate regions. The judgment criteria are: a pixel and the pixels in its eight neighborhoods (or the pixels in the four neighborhoods), if the difference in confidence between the two does not exceed 10% of the maximum confidence of the image, then the two pixels belong to the same region, and the confidence map corresponding to the current observed image is divided into regions based on the confidence between the pixels.

[0168] Step S2052: Generate a minimum outer bounding box for each divided area.

[0169] For the embodiment of the present application, after the confidence map corresponding to the current observation image is divided into regions, the minimum outer bounding box can be generated only for the irregular region, where the boundary of the regular region is the minimum outer bounding box of the regular region; or the minimum outer bounding box can be generated for each divided region.

[0170] Step S2053: Determine the confidence level corresponding to each region based on the confidence level corresponding to each pixel in each minimum outer bounding box.

[0171] In the embodiment of the present application, after generating a minimum outer bounding box for the generated irregular area, the average value of the pixels within the same bounding box on the confidence map is calculated as the confidence of the area. In the embodiment of the present application, in addition to using the average value of the pixels within the same outer bounding box as the confidence of the area, the maximum confidence value or minimum confidence value of the pixels in the area can also be used as the confidence of the area. In the embodiment of the present application, there is no limitation on the calculation method of the confidence of the area.

[0172] Step S2054: Determine at least one region based on the confidence levels corresponding to the respective regions and in accordance with preset rules.

[0173] For the embodiments of the present application, after obtaining the confidence corresponding to each area through the above embodiments, the areas can be sorted from high to low according to the confidence to obtain the top N1 areas. It is also possible to obtain areas with confidence greater than a preset confidence threshold based on the confidence corresponding to each area. For example, the areas with confidence greater than the preset confidence threshold can be N1 areas.

[0174] Among them, N1 can be between [100, 500]. In the embodiment of the present application, N1 can be input by the user or pre-set, and is not limited in the embodiment of the present application.

[0175] Step S2055: Match the minimum outer bounding box of each area in the at least one area with the feature map C to obtain the corresponding second area.

[0176] For the embodiment of the present application, after obtaining N1 regions, the bounding boxes of the first N1 regions are mapped to the feature map C to obtain N1 regions on the feature map C. In the embodiment of the present application, the corresponding second regions can represent the N1 regions corresponding to the feature map C.

[0177] Furthermore, if the CNN backbone network involved in the embodiment of the present application is a Unet network, the confidence map corresponds one-to-one to each pixel in the feature map C, so the coordinates of the minimum bounding box do not require any other conversion and can be matched directly; if the CNN backbone network involved in the embodiment of the present application is not a Unet network, it is necessary to perform coordinate conversion according to the receptive field of the network before matching.

[0178] Step S2056: Regularize the descriptor at each position in the feature map C.

[0179] It should be noted that the execution order between step S2056 and steps S2051-S2055 is not limited. For example, step S2056 can be executed after step S2055 or before step S2055. Specifically, regularization processing is performed on the descriptor of each position in the feature map C, which can specifically include: performing L2 regularization on the descriptor of each position in the feature map C. In this embodiment of the present application, in addition to performing L2 regularization processing on the descriptor of each position in the feature map C, other regularization processing, such as L1 regularization processing, can also be performed, which is not limited in this embodiment of the present application.

[0180] Step S2057: Calculate the descriptor means corresponding to each second region, and perform regularization processing on each descriptor mean to obtain the region descriptor corresponding to each second region.

[0181] In this embodiment of the present application, after regularizing the descriptor at each position in the feature map C, the descriptor mean in the bounding box of each of the N1 regions in the feature map C is calculated in sequence. The descriptor mean is then L2 regularized once more, and the resulting one-dimensional vector is used as the region descriptor. The center point of each bounding box is calculated as the region center. In this embodiment of the present application, each element in the resulting one-dimensional vector is used to characterize the region descriptor corresponding to each region in the feature map C.

[0182] Furthermore, after obtaining the region descriptors corresponding to each region, the set of N1 region descriptors can be obtained. In order to further reduce the complexity of the calculation and improve the accuracy, the areas that do not meet the specific conditions can be filtered out to obtain n c A set of region descriptors Specifically, in step S206, determining a scene image that meets the first preset condition from a pre-created scene map based on the region descriptor corresponding to the current observation image may include: step S2061 (not shown in the figure), step S2062 (not shown in the figure), step S2063 (not shown in the figure), and step S2064 (not shown in the figure), wherein:

[0183] Step S2061: Determine the region descriptor sets corresponding to the respective scene images from the pre-created scene map.

[0184] The region descriptor set includes at least one region descriptor.

[0185] In this embodiment of the present application, during the mapping process, region descriptors are extracted from the keyframe images according to the aforementioned process (implementation corresponding to steps S2051-S2057), and the region descriptors are stored in the scene map database. Specifically, step S2061 may include obtaining, from the scene map database, a set of region descriptors corresponding to each scene image.

[0186] Step S2062: performing region matching based on the region descriptors corresponding to the respective second regions and the region descriptor sets corresponding to the respective scene images, and performing region matching according to preset rules to obtain a region matching pair set.

[0187] Specifically, the current observation image can obtain n c The region descriptors are matched with each scene image in turn, and the region descriptor set of the scene image in the current region is recorded as Then the set of region matching pairs is determined by the following formula (1):

[0188]

[0189] Where f is the region descriptor, c is the current observation image, r is the candidate image, ||·||2 represents the L2 distance, NNMin represents the one with the smallest distance in NN matching, and P is the set of matching pairs in region matching that meet the mutual minimum principle of cross matching.

[0190] Step S2063: Determine the spatial consistency scores corresponding to the respective scene images based on the region matching pair set.

[0191] The spatial consistency score is used to measure the quality of region matching based on the spatial relationship between the matching regions, penalizing situations where there are too few matching pairs or disorganized matching. A higher spatial consistency score indicates a higher degree of similarity, and the candidate image is ranked higher.

[0192] Specifically, based on the set of region matching pairs, determining the spatial consistency scores corresponding to each scene image may include: determining the region center offset corresponding to each region matching pair based on the set of region matching pairs; determining the average row offset and the average column offset based on the region center offset corresponding to each region matching pair; determining the spatial consistency scores corresponding to each scene image based on the region center offset, average row offset, and column offset corresponding to each region matching pair based on the set of region matching pairs. In an embodiment of the present application, the spatial consistency scores corresponding to each scene image are determined by formula (2), wherein formula (2) includes:

[0193]

[0194] Where SS represents the spatial consistency score of regional matching; n r is the number of regions extracted from the candidate image; P is the region matching set; i, j represent the matching pair numbers when traversing the set P; (r, c) is the region center offset of the matching pair; and They represent the average column offset and average row offset in the P set respectively; the dist(·) function is a distance function, including but not limited to Manhattan distance, Euclidean distance, Minkowski distance, etc., max(·) is the maximum value function, and min(·) is the minimum value function.

[0195] Step S2064: Based on the spatial consistency scores corresponding to the respective scene images, determine the scene images that meet the first preset condition from the pre-created scene map.

[0196] In the embodiment of the present application, after obtaining the spatial consistency scores corresponding to each scene image, scene images that meet the first preset condition are determined from the pre-created scene map. Specifically, in the embodiment of the present application, the images can be re-sorted from high to low according to the spatial consistency scores, and the top M2 images can be selected as a new candidate image set. Alternatively, scene images with a spatial consistency score greater than a score threshold can be selected as scene images that meet the first preset condition.

[0197] Among them, M2 can be between [10,20].

[0198] Furthermore, there are at least two scene images that meet the first preset condition; in order to further improve the recall rate of scene retrieval and the accuracy of relocation, detailed texture features are added based on the scene images that meet the first preset condition obtained by scene retrieval to filter out scene images that meet the second preset condition from the scene images that meet the first preset condition. In an embodiment of the present application, the method may further include: step Sa (not shown in the figure), step Sb (not shown in the figure), and step Sc (not shown in the figure), wherein step Sa, step Sb, and step Sc may be performed after step S206, wherein,

[0199] Step Sa: obtain the F descriptor set corresponding to each scene image.

[0200] After obtaining a scene image that meets the first preset condition through the above embodiment, the F descriptor set corresponding to each candidate scene image is taken out from the scene map, and is recorded as {D F ref , ref∈[1,M2]}.

[0201] Step Sb: Based on the descriptor set F and the feature map C corresponding to each scene image, determine the similarity between each scene image and the current observation image.

[0202] Specifically, the F descriptor set corresponding to any scene image includes: at least one F descriptor; wherein, based on the F descriptor set corresponding to any scene image and the feature map C, determining the similarity between any scene image and the current observed image may specifically include: step Sb1 (not shown in the figure), step Sb2 (not shown in the figure), step Sb3 (not shown in the figure) and step Sb4 (not shown in the figure), wherein step Sb1 determines the distance between each F descriptor corresponding to any scene image and each pixel in the feature map C.

[0203] Specifically, for the candidate image ref, its F description subset D F ref An F descriptor FD i , calculate the distance between it and each pixel in the feature map C of the current observation image in turn, where the distance is determined by the following formula (3):

[0204]

[0205] Among them, F c is the feature map C of the current observation image, p c Represents any point on the feature map, then the different channels on the feature map constitute the descriptor F corresponding to the point c (p c ); ||·||2 represents the L2 distance.

[0206] Step Sb2: Based on the determined distance, determine the distance between each F descriptor and the feature map C corresponding to any scene image.

[0207] Specifically, after calculating the distance between each F descriptor corresponding to any scene image and each pixel in the feature map C, the distance between each F descriptor corresponding to any scene image and the feature map C is obtained according to the following formula (4), where:

[0208]

[0209] Among them, the F descriptor FD i The distance from the current observation image C is Dis(FD i ,C), min2 represents the second smallest value in the solution set, that is, the F descriptor FD is obtained i With the feature map F c The second smallest distance value among all pixel distances. Similarly, the min function obtains the minimum distance value, ε = 1 × 10 -5 .

[0210] Step Sb3: Determine the similarity between each F descriptor and the feature map C based on the distance between each F descriptor and the feature map C corresponding to any scene image.

[0211] Specifically, the similarity between each F descriptor and the feature map C is determined by the following formula (5), where formula (5) is:

[0212]

[0213] Among them, sim(FD i ) is the F descriptor FD i The similarity with the current observed image C, exp(·) is an exponential function, h is a constant, and h is between [0.2, 0.8].

[0214] Step Sb4: Based on the similarity between each F descriptor and the feature map C, determine the similarity between any scene image and the feature map C as the similarity between any scene image and the current observation image.

[0215] Specifically, based on the similarity between each F descriptor and the feature map C, the similarity between any scene map and the feature map C is determined by formula (6), where

[0216] Among them FD i Satisfy Dis(FD i ,C)>δFormula (6)

[0217] Among them, sim(ref, C) represents the point matching similarity between the candidate image ref and the current observation image C, n ref The number of F descriptors extracted for the candidate image ref.

[0218] Furthermore, the similarity between each scene image and the current observation image can be calculated according to steps Sb1-Sb4 above to obtain the similarity between each scene map and the current observation image. In step Sc, based on the similarity between each scene image and the current observation image, scene images that meet the second preset condition are determined from the scene images that meet the first preset condition.

[0219] Specifically, after the similarity between each scene image and the current observation image is calculated through the above embodiment, the images can be sorted from high to low according to the similarity, and the first M3 images can be selected as scene images that meet the second preset condition. Alternatively, based on the similarity between each scene image and the current observation image, scene images with a similarity greater than a preset similarity threshold can be selected as scene images that meet the second preset condition. For example, the scene images with a similarity greater than the preset similarity threshold can be M3 scene images.

[0220] Among them, M3 can be between [3,10].

[0221] Furthermore, the method may further include: step Sd (not shown in the figure), step Se (not shown in the figure) and step Sf (not shown in the figure), wherein step Sd, step Se and step Sf may be performed before step S206, wherein,

[0222] Step Sd: normalize each point descriptor in the feature map F to obtain a normalized result.

[0223] Specifically, after obtaining the feature map F, the NetVLAD network layer first uses a 1×1 convolution kernel to perform point-by-point convolution on the feature map F, and uses Softmax to exponentially normalize the result to obtain the result a.

[0224] Step Se: Based on the normalization result and the feature map C, determine the global descriptor corresponding to the current observation image. Specifically, the residual distance in each cluster is weighted and summed using the above result a as the weight to obtain K vectors, which represent the global representation of the K clusters. Then, L2 regularization is performed on the vector of each cluster. The vectors of the K clusters are then concatenated and globally L2 regularized. Finally, a high-dimensional vector is obtained, which is the global descriptor of the image, that is, the global descriptor corresponding to the current observation image, where K is between [16,128].

[0225] Step Sf: determining a scene image that meets a third preset condition from a pre-created scene map based on a global descriptor corresponding to the current observed image.

[0226] For the embodiment of the present application, during the scene map establishment process, the key frame image needs to also generate a feature map F and a global descriptor according to the above process, and the feature points during the map establishment are mapped to the feature map F. The multiple channels on F constitute the F descriptor. The global descriptor and F descriptor of the key frame are included in the scene map database for storage. That is, based on the global descriptor corresponding to the current observation image, a scene image that meets the third preset condition is determined from the pre-created scene map. Specifically, it may include: obtaining the global descriptors corresponding to each key frame from the scene map database, calculating the L2 distance between the global descriptor of the current observation image and the global descriptors of the massive key frame images in the scene map, and determining the scene map that meets the third preset condition from the pre-created scene map based on the L2 distance.

[0227] Specifically, based on the L2 distance, each scene image is sorted from small to large according to the distance, and the first M1 scene maps are intercepted. The smaller the distance, the greater the possibility of being located at the same geographical location, and the higher the ranking of the key frame. Finally, the top M1 key frames for mapping are selected as the scene map that meets the third preset condition, and M1 is between [20,100]. Alternatively, scene images with an L2 distance less than a preset L2 distance threshold can be selected as the scene map that meets the third preset condition.

[0228] Furthermore, based on the scene global features of the image, the candidate M1 frame images that are in the same geographical location as the current observation image can be quickly retrieved from the massive images in the scene map.

[0229] Among them, based on the above steps Sd-Sf, in order to improve the accuracy of the screened scene images, scene images that meet the third preset condition can be first extracted from the pre-created scene map, such as the M1 frame scene image, and then, based on the M1 frame scene image, the scene images can be further screened. Specifically, in step S206, based on the region descriptor corresponding to the current observation image, the scene image that meets the first preset condition is determined from the pre-created scene map. This can specifically include: based on the region descriptor corresponding to the current observation image, the scene image that meets the first preset condition is determined from the scene map that meets the third preset condition. Furthermore, after obtaining the scene map that meets the first preset condition, the scene map that meets the second preset condition is screened out from the scene map that meets the first preset condition.

[0230] For example, the M2 frame scene image is filtered out from the M1 frame scene map based on the region descriptor, and then the M3 frame scene map is filtered out from the M2 frame scene map.

[0231] Specifically, based on the above embodiment, after obtaining the scene map (M3 frame scene image) that meets the second preset condition through the scene retrieval process, repositioning is performed based on the scene image that meets the first preset condition in step S207, which may specifically include: repositioning based on the scene image that meets the second preset condition.

[0232] Specifically, relocalization based on the scene image that meets the second preset condition may include: step S1 (not shown in the figure), step S2 (not shown in the figure), step S3 (not shown in the figure), step S4 (not shown in the figure), step S5 (not shown in the figure), step S6 (not shown in the figure), and step S7 (not shown in the figure), wherein:

[0233] Step S1: clustering scene images that meet a second preset condition based on the co-viewing relationship of the scene map to obtain at least two co-viewing clusters.

[0234] Specifically, for the M3 images obtained by scene retrieval, clustering is performed according to the co-viewing relationship in the scene map, and the M3 frames of scene images are divided into several co-viewing clusters.

[0235] Step S2: Based on the similarity between each scene image in each common view cluster and the current observation image, determine a common view cluster that meets a fourth preset condition from at least two common view clusters.

[0236] In an embodiment of the present application, the above embodiment can be used to obtain the similarity between each scene image corresponding to the current observation image, and then based on the similarity between each scene image in each common view cluster and the current observation image, a common view cluster that meets the fourth preset condition is determined from at least two common view clusters.

[0237] Specifically, based on the similarity between each scene image in each common view cluster and the current observation image, a common view cluster that meets the fourth preset condition is determined from at least two common view clusters. Specifically, it may include: adding the point matching similarities between the images in each cluster and the current observation image, and selecting the cluster with the highest sum (referred to as common view cluster A) as the common view cluster that meets the fourth preset condition; or, determining the maximum similarity between each scene image in each common view cluster and the current observation image as the maximum similarity of each common view cluster, and determining the maximum similarity among the maximum similarities as the common view cluster that meets the fourth preset condition. For example, after clustering the scene images that meet the second preset condition, three common view clusters are obtained, such as common view cluster 1, common view cluster 2, and common view cluster 3, wherein the maximum similarity in common view cluster 1 is the maximum similarity. The maximum similarity in common view cluster 2 is 50%, the maximum similarity in common view cluster 2 is 80%, and the maximum similarity in common view cluster 3 is 70%, then common view cluster 2 is the common view cluster that meets the fourth preset condition; or the average value of the similarity in each common view cluster is determined, and the common view cluster with the largest average value is selected as the common view cluster that meets the fourth preset condition, and the maximum similarity among the maximum similarities is determined as the common view cluster that meets the fourth preset condition. For example, after clustering the scene images that meet the second preset condition, three common view clusters are obtained, such as common view cluster 1, common view cluster 2 and common view cluster 3, wherein the average value of each similarity in common view cluster 1 is 45%, the average value of each similarity in common view cluster 2 is 60%, and the average value of each similarity in common view cluster 3 is 64%, then the common view cluster that meets the fourth preset condition is common view cluster 3.

[0238] Step S3: Selecting a 3D map point corresponding to each scene image in a common view cluster that meets a fourth preset condition from the pre-created scene map.

[0239] Among them, each 3D map point corresponds to an F descriptor.

[0240] For example, the common view cluster that meets the fourth preset condition can be called common view cluster A. For each image belonging to common view cluster A, its corresponding 3D map point is sequentially taken from the scene map, and each 3D map point corresponds to an F descriptor.

[0241] Step S4: Obtain the matching position of the 3D map point in the feature map C corresponding to the 3D map point whose descriptor F satisfies the ratio test condition.

[0242] For the embodiment of the present application, the F descriptor that satisfies the ratio test condition is that the distance between the descriptor and the feature map C is greater than the ratio factor δ, where δ is between [1.25, 2]. In the embodiment of the present application, the matching position (X) corresponding to the 3D map point in the feature map C for which the F descriptor satisfies the ratio test condition is obtained. FDi ,Y FDiStep S5: convert the matching position into the image coordinate system corresponding to the current observation image to obtain the matching relationship between the 3D map point and the 2D map point in the current observation image.

[0243] Specifically, after obtaining the matching position (X FDi ,Y FDi ), then the matching position (X FDi ,Y FDi ) is converted to the image coordinate system of the current observation image, then the 3D map point corresponding to the F descriptor also forms a matching relationship with the 2D point on the current observation image.

[0244] Furthermore, in the embodiment of the present application, Unet is used as the CNN backbone network, so the feature map C corresponds one-to-one to the pixels of the current observation image, so no additional coordinate transformation is required. If other CNN backbone networks are used, coordinate transformation is required according to the network receptive field, and then the matching relationship between the 3D map points and the 2D map points in the current observation image is obtained.

[0245] Step S6: Obtain the pose of the reference image, and determine the initial pose corresponding to the current observation image based on the pose of the reference image and the matching relationship.

[0246] Specifically, after obtaining the matching relationship between the 3D map points and the 2D map points in the current observation image, the pose of the current observation image is calculated based on the matching relationship between the obtained 3D map points and the 2D map points in the current observation image, and the EPnP (Efficent Perspective-n-Point) algorithm is used. In an embodiment of the present application, each scene image that meets the fourth preset condition can be calculated to obtain a pose in the above manner, and then the initial pose corresponding to the current observation image is determined based on the pose calculated for each scene. It should be noted that when calculating the pose of the current observation image, the PnP-DLT (Perspective-n-Point—Direct Linear Transformation) algorithm, the Singular Value Decomposition (SVD) algorithm, and the Bundle Adjustment (BA) algorithm can also be used, which are not limited in the embodiment of the present application.

[0247] Specifically, after obtaining the poses calculated by each scene image, the poses calculated are weighted averaged based on the similarity between each scene image and the current observation image as a weight to obtain the initial pose corresponding to the current observation image.

[0248] Step S7: repositioning based on the initial pose corresponding to the current observation image.

[0249] For the embodiment of the present application, after the initial posture is quickly and accurately determined by the above method, the number of iterations can be greatly reduced when repositioning is performed using the initial posture corresponding to the current observation image, reducing system experiments and preventing falling into a local optimal solution.

[0250] Specifically, relocalization based on the initial pose corresponding to the current observation image in step S7 may include: projecting 3D map points corresponding to each scene image in the common view cluster that meets the fourth preset condition into the image coordinate system corresponding to the current observation image based on the initial pose corresponding to the current observation image; screening out 3D map points whose projection points are within the current observation image to obtain a map point set; converting the projection point coordinates of each 3D map point in the map point set into the coordinate system of the H feature map of the current observation image to obtain an H descriptor corresponding to each 3D map point; obtaining an H' descriptor corresponding to each 3D map point in the map point set; determining residual information of each 3D map point in the map point set based on the H' descriptor corresponding to each 3D map point in the map point set and the H descriptor corresponding to each 3D map point; determining an objective function based on the residual information of each 3D map point; and determining a pose after relocalization based on the objective function. The H' descriptor is the descriptor corresponding to the H' feature map corresponding to each key frame image when creating the scene map.

[0251] Specifically, a common view cluster that meets the fourth preset condition can be called a common view cluster A. This means that the 3D map points corresponding to each scene image in common view cluster A are projected into the image coordinate system corresponding to the current observation image. The 3D map points corresponding to all images in common view cluster A are extracted from the scene map to form a map point set PM. Since a map point may be observed simultaneously by multiple keyframe images in common view cluster A, each map point in the map point set PM has several corresponding H descriptors in the scene map. Next, based on the calculated initial pose of the current observation image, the 3D map points in the map point set PM are projected onto the image coordinate system of the current observation image. A determination is made as to whether the projected points are within the current observation image. The 3D map points within the current observation image are then selected to form a visible map point set VM.

[0252] The projection point of each map point in the visible map point set VM on the current observation image is converted to the H feature map coordinates of the current observation image, thereby obtaining the corresponding H descriptor. At the same time, there are also several H' descriptors corresponding to the key frame images in the common view cluster A in the scene map for this map point.

[0253] Furthermore, after obtaining the H' descriptor and the H descriptor corresponding to each 3D map point in the map point set, the residual information of each 3D map point is determined. Specifically, the residual information of each 3D map point is calculated using the following formula (7). Then, the objective function is determined based on the residual information of each 3D map point using formula (8).

[0254] in,

[0255]

[0256] Among them, r i is the H descriptor residual of the i-th map point in the visual map point set VM; H is the H descriptor of the H feature map of the current observation image corresponding to the map point; H' k is the kth H' descriptor of the key frame image in the common view cluster A corresponding to the map point in the scene map; w i,k is the product of the confidence weights of the H descriptor and the H' descriptor on the corresponding confidence maps; ||·||2 represents the L2 distance.

[0257] in,

[0258]

[0259] Wherein, ρ is a robust kernel function, and E(R, t) is the objective function of the calculated current observation image pose (R, t). Further, after obtaining the objective function through the above embodiment, in order to obtain the relocated pose more accurately, the relocated pose is determined based on the objective function, which may specifically include: step S771 (not shown in the figure), or step S771 and step S772 (not shown in the figure), wherein,

[0260] Step S771: Minimize the objective function to determine the updated pose.

[0261] Specifically, the objective function E(R, t) can be minimized by an algorithm such as the LM algorithm, the Gauss-Newton algorithm, or the gradient descent algorithm to update the pose of the current observation image, and the updated result is obtained as the pose of the current observation image after relocation.

[0262] Taking the pose update by LM algorithm as an example, the LM (Levenberg-Marquardt) algorithm is used to minimize the above objective function and update the pose of the observation image as follows, as shown in formula (9):

[0263]

[0264] Among them, (R + , t +) is the updated pose, δ is the update amount obtained by the LM algorithm, T is the transpose operation, and ^ is the anti-symmetric operation.

[0265] Furthermore, in order to improve the accuracy of relocalization, step S772 is executed after executing step S771, wherein step S772 is a step of cyclically executing the step of taking the updated posture as the initial posture corresponding to the current observation image, projecting the 3D map points corresponding to each scene image in the common view cluster that meets the fourth preset condition into the image coordinate system corresponding to the current observation image according to the initial posture corresponding to the current observation image, screening out the 3D map points whose projection points are in the current observation image, and obtaining a map point set, and the step of projecting each 3D map point in the map point set. The step of converting the point coordinates into the coordinate system of the H feature map of the current observation image to obtain the H descriptor corresponding to each 3D map point, the step of obtaining the H' descriptor corresponding to each 3D map point in the map point set, the step of determining the residual information of each 3D map point in the map point set based on the H' descriptor corresponding to each 3D map point in the map point set and the H descriptor corresponding to each 3D map point, the step of determining the objective function based on the residual information of each 3D map point, and step S771, until the pose converges, and the pose obtained when the pose converges is determined as the pose after relocation.

[0266] That is, in the embodiment of the present application, after obtaining the updated posture, the map point set PM is re-projected according to the updated posture to obtain a new visible map point set VM and the H descriptor residual r i , thereby obtaining the updated objective function E, and then repeat the above operation and iteratively update the pose until the update amount δ is small enough, then the pose is considered to converge, and the pose (R, t) at the time of final convergence is used as the final 6DOF pose of the current observation image, that is, the result of relocalization.

[0267] The end-to-end relocalization system involved in the above embodiment can be a trained network model. The training of the end-to-end relocalization system is divided into two parts. First, the confidence map branch and the dense feature map branch are trained. The confidence map is used as the weight of the H descriptor according to the above expression for end-to-end training. The loss function is set as the reprojection error when the final visual map point set VM is projected based on the 6DOF pose at the time of convergence of the current observed image and the true pose. Then, the weights of the CNN backbone network are fixed, and the scene retrieval branch is trained. Image similarity is defined based on the global image descriptor. Then, a ternary loss pair is constructed for training. The image similarity of image pairs at the same geographical location in the real scene is much greater than the image similarity of image pairs at different geographical locations in the real scene.

[0268] The above embodiment introduces a relocation method from the perspective of method flow, and the following embodiment introduces a relocation device from the perspective of a virtual module. Please refer to the following embodiment for details.

[0269] The embodiment of the present application provides a repositioning device, such as Figure 3 As shown, the relocation device 30 may include: a first acquisition module 31, a feature extraction module 32, a first determination module 33, a generation module 34, a second determination module 35, a third determination module 36 and a relocation module 37, wherein:

[0270] The first acquisition module 31 is used to acquire the current observation image.

[0271] The feature extraction module 32 is used to extract features from the current observation image to obtain a dense feature map.

[0272] The first determination module 33 is used to determine the residual distance between each point descriptor in the dense feature map and each cluster center in the pre-created scene map, and determine the feature map C based on the residual distance.

[0273] The cluster centers are obtained by clustering the points in the pre-created scene map according to the scene. The generation module 34 is used to generate a confidence map corresponding to the current observation image based on the dense feature map.

[0274] The second determination module 35 is configured to determine a region descriptor corresponding to the current observation image based on the confidence map and the feature map C corresponding to the current observation image.

[0275] The third determining module 36 is configured to determine a scene image that meets a first preset condition from the pre-created scene map based on the region descriptor corresponding to the current observation image.

[0276] The repositioning module 37 is configured to perform repositioning based on a scene image that meets a first preset condition.

[0277] Specifically, when determining the residual distance between each point descriptor in the dense feature map and each cluster center in the pre-created scene map, the first determination module 33 is specifically used to:

[0278] Perform convolution processing based on the dense feature map to obtain the feature map F;

[0279] Determine the residual distance between each point descriptor in the feature map F and each cluster center in the pre-created scene map.

[0280] Specifically, when the second determination module 35 determines the region descriptor corresponding to the current observation image based on the confidence map and the feature map C corresponding to the current observation image, it is specifically used to:

[0281] Divide the confidence map corresponding to the current observation image into regions;

[0282] Generate the minimum outer bounding box for each divided area;

[0283] Based on the confidence levels of each pixel in each minimum outer bounding box, the confidence levels of each region are determined.

[0284] Determine at least one region based on the confidence levels corresponding to the respective regions and in accordance with preset rules;

[0285] Matching the minimum outer bounding box of each region in at least one region with the feature map C to obtain the corresponding second region;

[0286] Regularize the descriptor at each position in the feature map C;

[0287] Calculate the descriptor means corresponding to each second region, and perform regularization processing on each descriptor means to obtain the region descriptors corresponding to each second region. Specifically, when the third determination module 36 determines the scene image that meets the first preset condition from the pre-created scene map based on the region descriptors corresponding to the current observation image, it is specifically used to:

[0288] Determine, from a pre-created scene map, a set of region descriptors corresponding to each scene image, wherein the set of region descriptors includes at least one region descriptor;

[0289] performing region matching based on the region descriptors corresponding to the respective second regions and the region descriptor sets corresponding to the respective scene images, and performing region matching according to preset rules to obtain a set of region matching pairs; and determining a spatial consistency score corresponding to each scene image based on the set of region matching pairs;

[0290] Based on the spatial consistency scores corresponding to the respective scene images, scene images that meet the first preset condition are determined from the pre-created scene map.

[0291] Specifically, when determining the spatial consistency scores corresponding to the respective scene images based on the region matching pair set, the third determination module 36 is specifically configured to:

[0292] Based on the set of region matching pairs, determining the region center offset corresponding to each region matching pair;

[0293] Based on the region center offsets corresponding to the region matching pairs, the average row offset and the average column offset are determined; based on the set of region matching pairs, the region center offsets, average row offsets and column offsets corresponding to each region matching pair are used to determine the spatial consistency scores corresponding to each scene image.

[0294] In another possible implementation of the embodiment of the present application, there are at least two scene images that meet the first preset condition; the apparatus 30 further includes: a second acquisition module, a fourth determination module, and a fifth determination module, wherein the second acquisition module is configured to acquire a set of F descriptors corresponding to each scene image;

[0295] A fourth determination module is configured to determine the similarity between each scene image and the current observation image based on the descriptor set F and the feature map C corresponding to each scene image;

[0296] The fifth determining module is configured to determine, based on the similarities between each scene image and the current observation image, a scene image that meets the second preset condition from the scene images that meet the first preset condition.

[0297] Specifically, the F descriptor set corresponding to any scene image includes: at least one F descriptor;

[0298] The fourth determination module is specifically configured to:

[0299] Determine the distance between each F descriptor corresponding to any scene image and each pixel in the feature map C; based on the determined distances, determine the distance between each F descriptor corresponding to any scene image and the feature map C;

[0300] Based on the distance between each F descriptor and the feature map C corresponding to any scene image, determine the similarity between each F descriptor and the feature map C;

[0301] Based on the similarity between each F descriptor and the feature map C, the similarity between any scene image and the feature map C is determined as the similarity between any scene image and the current observation image.

[0302] Specifically, when performing relocation based on the scene image that meets the first preset condition, the relocation module 37 is specifically configured to:

[0303] Repositioning is performed based on the scene image that meets the second preset condition.

[0304] Furthermore, the device 30 further includes: a normalization processing module, a sixth determination module, and a seventh determination module; wherein,

[0305] A normalization processing module is used to normalize each point descriptor in the feature map F to obtain a normalized result. The feature map F is obtained by convolving the dense feature map;

[0306] A sixth determination module is used to determine the global descriptor corresponding to the current observation image based on the normalization result and the feature map C;

[0307] a seventh determining module, configured to determine a scene image that meets a third preset condition from a pre-created scene map based on a global descriptor corresponding to the current observation image;

[0308] The third determining module 36 is specifically configured to:

[0309] A scene image that meets the first preset condition is determined from the scene map that meets the third preset condition based on the region descriptor corresponding to the current observation image.

[0310] Specifically, when performing relocation based on the scene image that meets the second preset condition, the relocation module 37 is specifically configured to:

[0311] Based on the co-viewing relationship of the scene map, clustering the scene images that meet the second preset condition to obtain at least two co-viewing clusters;

[0312] Determining a common view cluster that meets a fourth preset condition from at least two common view clusters based on similarities between each scene image in each common view cluster and the current observation image;

[0313] Selecting a 3D map point corresponding to each scene image in a common view cluster that meets a fourth preset condition from a pre-created scene map, each 3D map point corresponding to an F descriptor;

[0314] Obtain the matching position of the 3D map point in the feature map C whose descriptor F satisfies the ratio test condition; convert the matching position to the image coordinate system corresponding to the current observation image to obtain the matching relationship between the 3D map point and the 2D map point in the current observation image;

[0315] Obtain the pose of the reference image, and determine the initial pose corresponding to the current observation image based on the pose of the reference image and the matching relationship;

[0316] Relocalization is performed based on the initial pose corresponding to the current observation image.

[0317] Specifically, the repositioning module 37 includes: a projection unit, a screening unit, a coordinate conversion unit, an acquisition unit, a residual information determination unit, a target minimization unit, and a determination unit, wherein:

[0318] a projection unit for projecting the 3D map points corresponding to each scene image in the common view cluster that meets the fourth preset condition into the image coordinate system corresponding to the current observation image according to the initial pose corresponding to the current observation image; a screening unit for screening out the 3D map points whose projection points are within the current observation image to obtain a map point set; a coordinate conversion unit for converting the projection point coordinates of each 3D map point in the map point set into the coordinate system of the H feature map of the current observation image to obtain an H descriptor corresponding to each 3D map point;

[0319] Acquisition unit: used to obtain the H' descriptor corresponding to each 3D map point in the map point set. The H' descriptor is the descriptor corresponding to the H' feature map corresponding to each key frame image when creating the scene map;

[0320] a residual information determining unit, configured to determine residual information of each 3D map point in the map point set based on an H' descriptor corresponding to each 3D map point and an H descriptor corresponding to each 3D map point; and a target minimization unit, configured to determine a target function based on the residual information of each 3D map point;

[0321] The determination unit is used to determine the pose after relocalization based on the objective function.

[0322] Specifically, the determination unit includes: a posture updating subunit and / or a loop execution subunit, wherein the posture updating subunit is used to minimize the objective function and determine the updated posture;

[0323] The loop execution subunit is used to loop through the steps of taking the updated posture as the initial posture corresponding to the current observation image, the operations performed by the projection unit, the screening unit, the coordinate conversion unit, the acquisition unit, the residual information determination unit, the target minimization unit, and the posture update unit until the posture converges, and the posture obtained when the posture converges is determined as the posture after repositioning.

[0324] It should be noted that the first determination module 33, the second determination module 35, the third determination module 36, the fourth determination module, the fifth determination module, the sixth determination module and the seventh determination module can be different modules, or they can all be the same modules, or they can be partially the same modules; the first acquisition module 31 and the second acquisition module can all be the same modules, or they can all be different modules, which is not limited in the embodiments of the present application.

[0325] An embodiment of the present application provides a repositioning device. Compared with the related art, in the embodiment of the present application, a confidence map corresponding to the current observation image is generated through a dense feature map, and then the residual distance between each point descriptor in the dense feature map and each cluster center in the pre-created scene map is determined based on the residual distance, and the region descriptor corresponding to the current observation image can be determined based on the feature map C and the confidence map, that is, the scene map that meets the first preset condition can be determined from the pre-created scene map through the region descriptor, so that repositioning can be performed in the scene map that meets the first preset condition, that is, in the embodiment of the present application, high-level semantic features of the current observation image can be obtained to screen out scene images with higher matching degrees from the pre-created scene map, and then repositioning is performed based on the scene images with higher matching degrees, thereby reducing the complexity of repositioning and improving the accuracy of repositioning.

[0326] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0327] An electronic device is provided in an embodiment of the present application, such as Figure 4 As shown, Figure 4 The electronic device 400 shown includes a processor 401 and a memory 403. The processor 401 and the memory 403 are connected, for example, via a bus 402. Optionally, the electronic device 400 may further include a transceiver 404. It should be noted that in actual applications, the number of transceivers 404 is not limited to one, and the structure of the electronic device 400 does not constitute a limitation on the embodiments of the present application.

[0328] Processor 401 can be a CPU (Central Processing Unit), a general-purpose processor, a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. It can implement or execute the various exemplary logic blocks, modules, and circuits described in conjunction with the disclosure of this application. Processor 401 can also be a combination that implements computing functions, such as a combination of one or more microprocessors, a combination of a DSP and a microprocessor, etc.

[0329] The bus 402 may include a path for transmitting information between the above components. The bus 402 may be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus. The bus 402 may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 4 Only one thick line is used in the diagram, but this does not mean that there is only one bus or one type of bus.

[0330] The memory 403 can be a ROM (Read Only Memory) or other types of static storage devices that can store static information and instructions, a RAM (Random Access Memory) or other types of dynamic storage devices that can store information and instructions, or an EEPROM (Electrically Erasable Programmable Read Only Memory), a CD-ROM (Compact Disc Read Only Memory) or other optical disc storage, optical disc storage (including compact disc, laser disc, optical disc, digital versatile disc, Blu-ray disc, etc.), a magnetic disk storage medium or other magnetic storage device, or any other medium that can be used to carry or store the desired program code in the form of instructions or data structures and can be accessed by a computer, but is not limited to this. The memory 403 is used to store the application code that executes the solution of the present application, and is controlled by the processor 401 to execute. The processor 401 is used to execute the application code stored in the memory 403 to implement the content shown in the aforementioned method embodiment.

[0331] The electronic devices include, but are not limited to, mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), and in-vehicle terminals (e.g., in-vehicle navigation terminals), as well as fixed terminals such as digital TVs and desktop computers. Servers and the like are also possible. Figure 4 The electronic device shown is merely an example and should not limit the functions and scope of use of the embodiments of the present application.

[0332] The embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon, which, when executed on a computer, enables the computer to execute the corresponding contents of the aforementioned method embodiment. Compared with the related art, in the embodiment of the present application, a confidence map corresponding to the current observation image is generated by a dense feature map, and then a feature map C is determined based on the residual distance between each point descriptor in the dense feature map and each cluster center in the pre-created scene map, and based on the residual distance, a region descriptor corresponding to the current observation image can be determined based on the feature map C and the confidence map, that is, a scene map that meets the first preset condition can be determined from the pre-created scene map by the region descriptor, so that relocation can be performed in the scene map that meets the first preset condition, that is, in the embodiment of the present application, high-level semantic features of the current observation image can be obtained to screen out scene images with a higher degree of matching from the pre-created scene map, and then relocation is performed based on the scene images with a higher degree of matching, thereby reducing the complexity of relocation and improving the accuracy of relocation.

[0333] Those skilled in the art will clearly understand that for the sake of convenience and brevity, the division of the above-mentioned functional modules is only used as an example for illustration. In actual applications, the above-mentioned functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. The specific working processes of the above-mentioned systems, devices, and units can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0334] In the embodiments provided in this application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of the modules or units is only a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0335] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0336] In addition, the functional units in the various embodiments of the present application can be integrated into a processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of a software functional unit. If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor to execute all or part of the steps of the method described in the various embodiments of the present application. The aforementioned storage medium includes various media that can store program code, such as a USB flash drive, a mobile hard disk, a read-only memory, a random access memory, a magnetic disk or an optical disk.

[0337] The above embodiments are merely a detailed introduction to the technical solutions of the present application. However, the description of the above embodiments is only intended to help understand the method and core concept of the present application and should not be construed as limiting the present application. Any changes or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present application should be included in the scope of protection of the present application.

Claims

1. A relocation method, characterized in that: include: Acquire a current observation image, and extract features from the current observation image to obtain a dense feature map; Determining the residual distance between each point descriptor in the dense feature map and each cluster center in the pre-created scene map, and determining a feature map C based on the residual distance, wherein each cluster center is obtained by clustering each point in the pre-created scene map according to the scene; Generating a confidence map corresponding to the current observation image based on the dense feature map; Determining a region descriptor corresponding to the current observation image based on the confidence map corresponding to the current observation image and the feature map C; Determining a scene image that meets a first preset condition from the pre-created scene map based on a region descriptor corresponding to the current observation image; Repositioning is performed based on the scene image that meets the first preset condition.

2. The method according to claim 1, characterized in that Determining the residual distance between each point descriptor in the dense feature map and each cluster center in the pre-created scene map includes: Perform convolution processing based on the dense feature map to obtain a feature map F; Determine the residual distance between each point descriptor in the feature map F and each cluster center in the pre-created scene map.

3. The method according to claim 2, characterized in that The determining, based on the confidence map corresponding to the current observation image and the feature map C, a region descriptor corresponding to the current observation image includes: Dividing the confidence map corresponding to the current observation image into regions; Generate the minimum outer bounding box for each divided area; Based on the confidence levels of each pixel in each minimum outer bounding box, the confidence levels of each region are determined. Determine at least one region based on the confidence levels corresponding to the respective regions and in accordance with preset rules; Matching the minimum outer bounding box of each region in the at least one region with the feature map C to obtain respective corresponding second regions; Regularizing the descriptor at each position in the feature map C; The descriptor means corresponding to each second region are calculated, and regularization processing is performed on each descriptor mean to obtain the region descriptors corresponding to each second region.

4. The method according to claim 3, characterized in that The determining, from the pre-created scene map, a scene image that meets a first preset condition based on the region descriptor corresponding to the current observation image includes: Determining, from the pre-created scene map, a set of region descriptors corresponding to each scene image, wherein the set of region descriptors includes at least one region descriptor; According to a preset rule, region matching is performed based on the region descriptors corresponding to the respective second regions and the region descriptor sets corresponding to the respective scene images to obtain a region matching pair set; Determining, based on the set of region matching pairs, spatial consistency scores corresponding to each scene image; Based on the spatial consistency scores respectively corresponding to the scene images, scene images that meet a first preset condition are determined from the pre-created scene map.

5. The method according to claim 4, characterized in that Determining the spatial consistency scores corresponding to the respective scene images based on the set of region matching pairs includes: Based on the set of region matching pairs, determining the region center offsets corresponding to the respective region matching pairs; Determining an average row offset and an average column offset based on the region center offsets respectively corresponding to the region matching pairs; Based on the set of region matching pairs, the region center offsets, the average row offsets, and the column offsets respectively corresponding to the region matching pairs, a spatial consistency score respectively corresponding to each scene image is determined.

6. The method according to claim 5, characterized in that There are at least two scene images that meet the first preset condition; and the method further includes: Get the F descriptor set corresponding to each scene image; Determining the similarity between each scene image and the current observation image based on the descriptor set F corresponding to each scene image and the feature map C; Based on the similarities between the respective scene images and the current observation image, a scene image that meets a second preset condition is determined from the scene images that meet the first preset condition.

7. The method according to claim 6, characterized in that The F descriptor set corresponding to any scene image includes: at least one F descriptor; Determining the similarity between any scene image and the current observation image based on the F descriptor set corresponding to any scene image and the feature map C includes: Determine the distance between each F descriptor corresponding to any scene image and each pixel in the feature map C; Based on the determined distance, determining the distance between each F descriptor corresponding to any one of the scene images and the feature map C; Determining the similarity between each F descriptor and the feature map C based on the distance between each F descriptor corresponding to any one of the scene images and the feature map C; Based on the similarity between each F descriptor and the feature map C, the similarity between the any scene image and the feature map C is determined as the similarity between the any scene image and the current observation image.

8. The method according to claim 6 or 7, characterized in that The repositioning based on the scene image that meets the first preset condition includes: Repositioning is performed based on the scene image that meets the second preset condition.

9. The method according to any one of claims 1 to 7, characterized in that The method further comprises: Normalizing each point descriptor in a feature map F to obtain a normalized result, wherein the feature map F is obtained by convolving the dense feature map; Determining a global descriptor corresponding to the current observation image based on the normalization result and the feature map C; Determining a scene image that meets a third preset condition from the pre-created scene map based on a global descriptor corresponding to the current observation image; The determining of a scene image that meets a first preset condition from the pre-created scene map based on the region descriptor corresponding to the current observation image includes: The scene image that meets the first preset condition is determined from the scene map that meets the third preset condition based on the region descriptor corresponding to the current observation image.

10. The method according to claim 8, characterized in that The repositioning based on the scene image that meets the second preset condition includes: Clustering the scene images that meet the second preset condition based on the co-viewing relationship of the scene map to obtain at least two co-viewing clusters; determining, from the at least two common-view clusters, a common-view cluster that satisfies a fourth preset condition based on similarities between each scene image in each common-view cluster and the current observation image; Selecting, from the pre-created scene map, a 3D map point corresponding to each scene image in the common view cluster that meets the fourth preset condition, each 3D map point corresponds to an F descriptor; Get the matching position of the 3D map point in the feature map C where the F descriptor meets the ratio test condition; Converting the matching position to an image coordinate system corresponding to the current observation image to obtain a matching relationship between the 3D map point and the 2D map point in the current observation image; Acquire the pose of a reference image, and determine an initial pose corresponding to the current observation image based on the pose of the reference image and the matching relationship; Repositioning is performed based on the initial pose corresponding to the current observation image.

11. The method according to claim 10, characterized in that The repositioning based on the initial pose corresponding to the current observation image includes: Projection step: projecting the 3D map point corresponding to each scene image in the common view cluster that meets the fourth preset condition into the image coordinate system corresponding to the current observation image according to the initial pose corresponding to the current observation image; Screening step: screening out 3D map points whose projection points are within the current observation image to obtain a map point set; Coordinate conversion step: converting the projection point coordinates of each 3D map point in the map point set into the coordinate system of the H feature map of the current observation image to obtain the H descriptor corresponding to each 3D map point; Acquisition step: acquiring an H' descriptor corresponding to each 3D map point in the map point set, wherein the H' descriptor is a descriptor corresponding to an H' feature map corresponding to each key frame image when creating a scene map; Residual information determining step: determining residual information of each 3D map point in the map point set based on the H' descriptor corresponding to each 3D map point in the map point set and the H descriptor corresponding to each 3D map point; Target minimization step: determining the target function based on the residual information of each 3D map point; The relocalized pose is determined based on the objective function.

12. The method according to claim 11, characterized in that The determining of the relocated pose based on the objective function includes: Posture updating step: performing minimization processing on the objective function to determine an updated posture; and / or, Loop execution step: loop execution of the step of using the updated posture as the initial posture corresponding to the current observation image, the projection step, the screening step, the coordinate conversion step, the acquisition step, the residual information determination step, the target minimization step and the posture updating step until the posture converges, and the posture obtained when the posture converges is determined as the posture after repositioning.

13. A repositioning device, characterized in that: include: A first acquisition module is used to acquire the current observation image; A feature extraction module, configured to extract features from the current observation image to obtain a dense feature map; A first determination module is configured to determine a residual distance between each point descriptor in the dense feature map and each cluster center in the pre-created scene map, and determine a feature map C based on the residual distance, wherein each cluster center is obtained by clustering each point in the pre-created scene map according to the scene; A generating module, configured to generate a confidence map corresponding to the current observation image based on the dense feature map; A second determination module is configured to determine a region descriptor corresponding to the current observation image based on the confidence map corresponding to the current observation image and the feature map C; A third determining module, configured to determine a scene image that meets a first preset condition from the pre-created scene map based on a region descriptor corresponding to the current observation image; A repositioning module is used to perform repositioning based on the scene image that meets the first preset condition.

14. An electronic device, characterized in that: It includes: One or more processors; Memory; One or more applications, wherein the one or more applications are stored in the memory and configured to be executed by the one or more processors, and the one or more applications are configured to: execute a relocation method according to any one of claims 1 to 12.

15. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, a relocation method according to any one of claims 1 to 12 is implemented.

Images