A map-free visual positioning method, device, equipment and medium

Abstract

This application discloses a map-free visual localization method, apparatus, device, and medium. The method includes acquiring scene images collected in real time by a robot and extracting an initial feature map from the scene images; enhancing the initial feature map into a voxel feature map using a sparse kernel self-attention mechanism; aggregating the voxel feature map from several dimensions and multiplying the aggregated features element-wise to obtain image feature identifiers; searching for at least one anchor point feature identifier based on the image feature identifiers; and determining a six-degree-of-freedom pose based on the set of anchor point feature identifiers. This application utilizes a sparse kernel self-attention mechanism to enhance the initial feature map using voxels and aggregates the voxel feature map from several dimensions, achieving cross-domain interactive sparse quantization. This overcomes feature interference caused by motion blur or illumination changes, improves the robustness of scene image feature extraction, thereby reducing the probability of matching failure or inaccurate matching results and improving the accuracy of map-free visual localization.

Description

Technical Field

[0001] This application relates to the fields of intelligent robots and computer vision technology, and in particular to a mapless visual positioning method, device, equipment and medium. Background Technology

[0002] Mapless visual localization technology aims to perform similar image retrieval and relative pose regression using a pre-built reference image database, without relying on explicit 3D maps or implicit neural representations of the scene. This mapless visual localization technology offers advantages such as convenient deployment, privacy protection, and easy adaptation to new scenarios, and is widely used in fields such as intelligent robot navigation and autonomous driving.

[0003] However, in complex scenarios such as drastic changes in perspective, differences in lighting conditions, motion blur, and dynamic occlusion, the visual features extracted from scene images captured by the robot's perception system can undergo significant and unstable changes due to factors such as changes in ambient lighting, object occlusion, perspective shifts, or dynamic interference. These drastic fluctuations in visual features can easily lead to feature matching failures or inaccurate matching results, thus affecting the accuracy and overall robustness of map-free visual localization.

[0004] Therefore, existing technologies still need to be improved and enhanced. Summary of the Invention

[0005] The technical problem to be solved by this application is to provide a map-free visual positioning method, apparatus, device and medium to address the shortcomings of the existing technology.

[0006] To address the aforementioned technical problems, the first aspect of this application provides a map-free visual positioning method, wherein the map-free visual positioning method specifically includes: The robot acquires scene images in real time and uses a pre-trained visual encoder to extract the initial feature map of the scene images. The initial feature map is enhanced into a voxel feature map through a sparse kernel self-attention mechanism; The voxel feature maps are aggregated from several dimensions, and image feature identifiers are determined by feature fusion. The several dimensions include one or more of spatial dimension, frequency dimension and channel dimension. Based on the image feature identifier, at least one anchor point feature identifier is searched in the reference image database by debiased anchor point registration. The six-degree-of-freedom pose of the robot is determined based on the anchor point feature identifier and geometric information pose regression.

[0007] The map-free visual localization method, wherein enhancing the initial feature map into a voxel feature map through a sparse kernel self-attention mechanism specifically includes: The nonlocal dependencies of the initial feature map are captured by Gaussian kernel self-attention, and the sparse response map is determined by the sparse maximum function; The sparse response map is residually concatenated with the initial feature map to obtain a voxel feature map.

[0008] The map-free visual localization method, wherein capturing the non-local dependencies of the initial feature map through Gaussian kernel self-attention specifically includes: Based on the initial feature map, construct the query vector, key vector, and value vector; Based on the query vector and the key vector, the similarity between any two feature positions in the initial feature map is calculated using a Gaussian kernel function to obtain a non-local dependency relationship.

[0009] The map-free visual localization method, wherein the plurality of dimensions includes spatial dimension, frequency dimension, and channel dimension, and the aggregation of the voxel feature map from the plurality of dimensions and the determination of image feature identifiers by feature fusion specifically includes: Channel domain summation is performed on the voxel feature map to obtain the spatial dimension feature map; The voxel feature map is corrected for frequency dimension to obtain a frequency dimension feature map; The voxel feature map is subjected to indicator compression and spatial domain summation to obtain the channel response, and the channel response is logarithmically adjusted to obtain the channel dimension feature vector. The spatial dimension feature map, the frequency dimension feature map, and the channel dimension feature vector are fused to obtain an image feature identifier.

[0010] The map-free visual localization method, wherein fusing the spatial dimension feature map, the frequency dimension feature map, and the channel dimension feature vector to obtain an image feature identifier specifically includes: The spatial dimension feature map is transposed and multiplied with the frequency dimension feature map by a dot product and summed in the spatial domain to obtain the spatial-frequency dimension feature vector. The spatial-frequency dimension feature vector and the channel dimension feature vector are multiplied element-wise to obtain the image feature identifier.

[0011] The map-free visual localization method, wherein the step of searching for at least one anchor point feature identifier in the reference image database based on the image feature identifier through bias-free anchor point registration specifically includes: The nearest neighbor feature candidate set is retrieved from the reference image feature identifier library corresponding to the reference image database by means of nearest neighbor graph retrieval; At least one anchor feature identifier is determined based on the nearest neighbor feature candidate set using autoregressive anchor attribution.

[0012] The map-free visual localization method, wherein determining the robot's six-degree-of-freedom pose based on geometric information pose regression according to the anchor point feature identifier specifically includes: The gating vector and value vector corresponding to the anchor point feature identifier are determined using a symmetric exchange-gated multilayer perceptron, and the relative pose error of the robot is determined using the gating vector and value vector. The robot's six-degree-of-freedom pose is obtained by using the reference six-degree-of-freedom pose corresponding to the anchor point feature identifier and the relative pose error to determine the predicted absolute pose.

[0013] A second aspect of this application provides a map-free visual positioning device, wherein the map-free visual positioning device specifically includes: The acquisition module is used to acquire scene images collected by the robot in real time; The control module is used to extract an initial feature map of the scene image using a pre-trained visual encoder, enhance the initial feature map into a voxel feature map through a sparse kernel self-attention mechanism, aggregate the voxel feature map from several dimensions, determine image feature identifiers through feature fusion, search for at least one anchor feature identifier in the reference image database based on the image feature identifiers through debiased anchor registration, and determine the six-degree-of-freedom pose of the robot based on geometric information pose regression according to the anchor feature identifiers, wherein the several dimensions include one or more of spatial dimension, frequency dimension and channel dimension.

[0014] A third aspect of this application provides a computer-readable storage medium storing one or more programs that can be executed by one or more processors to implement the steps in any of the map-free visual positioning methods described above.

[0015] A fourth aspect of this application provides a terminal device, which includes: a processor and a memory; The memory stores a computer-readable program that can be executed by the processor; When the processor executes the computer-readable program, it implements the steps in any of the map-free visual positioning methods described above.

[0016] Beneficial effects: 1. This application utilizes a sparse kernel self-attention mechanism to perform voxel enhancement on the initial feature map and aggregates the voxel feature map from several dimensions, realizing cross-domain interactive sparse quantization, overcoming feature interference caused by motion blur or illumination changes, improving the robustness of scene image feature extraction, thereby reducing the probability of matching process failure or inaccurate matching results, and improving the accuracy of mapless visual positioning.

[0017] 2. This application uses a sparse kernel self-attention mechanism to construct the global sparse affinity response of the initial feature map. Compared with the conventional dot product self-attention mechanism, it can effectively reduce redundant voxel interactions and reduce the computational cost of the feature enhancement process. At the same time, it can accurately capture the non-local dependencies in the initial feature map and ensure the feature enhancement effect.

[0018] 3. This application simultaneously aggregates and fuses voxel feature maps from spatial, frequency, and channel dimensions. Multi-dimensional feature aggregation can comprehensively integrate effective localization information in the image, further suppress feature fluctuations caused by environmental interference, and obtain more robust image feature identifiers.

[0019] 4. This application uses nearest neighbor graph retrieval and autoregressive anchor point attribution to match anchor point feature identifiers for scene images, which can eliminate confounding factors in feature registration and thus further improve the accuracy of mapless visual positioning.

[0020] 5. This application utilizes pose regression based on set information to determine relative pose error, which can enhance the geometric constraints on six-degree-of-freedom pose regression, thereby improving the accuracy of six-degree-of-freedom pose solution and further enhancing the accuracy of map-free visual positioning. Attached Figure Description

[0021] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0022] Figure 1 A flowchart of a map-free visual positioning method provided in an embodiment of this application.

[0023] Figure 2 This is a schematic diagram of the overall process of the map-free visual positioning method provided in the embodiments of this application.

[0024] Figure 3 This is an implementation framework diagram of the map-free visual positioning method provided in the embodiments of this application.

[0025] Figure 4This is a schematic diagram of the mapless visual positioning device provided in the embodiments of this application.

[0026] Figure 5 A schematic block diagram of the terminal device provided in the embodiments of this application. Detailed Implementation

[0027] This application provides a map-free visual positioning method, apparatus, device, and medium. To make the objectives, technical solutions, and effects of this application clearer and more explicit, the following detailed description is provided with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only for explaining this application and are not intended to limit this application.

[0028] Those skilled in the art will understand that, unless specifically stated otherwise, the singular forms “a,” “an,” “the,” and “the” used herein may also include the plural forms. It should be further understood that the term “comprising” as used in this application’s specification means the presence of the stated features, integers, steps, operations, elements, and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof. It should be understood that when this application states that an element is “connected” or “coupled” to another element, it may be directly connected or coupled to the other element, or there may be intermediate elements. Furthermore, “connected” or “coupled” as used herein may include wireless connection or wireless coupling. The term “and / or” as used herein includes all or any unit and all combinations of one or more associated listed items.

[0029] It will be understood by those skilled in the art that, unless otherwise defined, all terms used herein (including technical and scientific terms) have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains. It should also be understood that terms such as those defined in general dictionaries should be understood to have the same meaning as in the context of the prior art, and should not be interpreted in an idealized or overly formal sense unless specifically defined as herein.

[0030] It should be understood that the sequence number and size of each step in this embodiment do not imply the order of execution. The execution order of each process is determined by its function and internal logic, and should not constitute any limitation on the implementation process of this application embodiment.

[0031] Research has shown that visual localization aims to estimate a robot's six degrees of freedom (6DoF) pose in the world coordinate system using query images captured by airborne cameras. This is a key technology for enabling robots to achieve embodied perception and autonomous navigation in their work environments. Traditional visual localization typically relies on pre-reconstructed 3D scene maps. The robot's 6DoF pose is calculated by performing 2D-3D matching between the current query image and landmarks in the 3D scene map. While this map-based method offers high accuracy, it suffers from inherent limitations such as high map construction and storage costs, high computational overhead, and difficulty adapting to dynamic scene changes including lighting variations, seasonal changes, and partial occlusion.

[0032] Therefore, map-free visual localization technology has emerged to meet the practical needs of lightweight deployment and edge computing. Map-free visual localization aims to perform similar image retrieval and relative pose regression using a pre-built reference image database, thus eliminating the need for explicit 3D maps or implicit neural representations of the scene. This technology offers advantages such as convenient deployment, privacy protection, and easy adaptation to new scenarios, and is widely used in fields such as intelligent robot navigation and autonomous driving. However, the visual features extracted from scene images captured by the robot's perception system in existing map-free visual localization methods can undergo significant and unstable changes due to factors such as changes in ambient lighting, object occlusion, viewpoint shifts, or dynamic interference, thereby affecting the accuracy of map-free visual localization.

[0033] Furthermore, since the similarity in the feature space and the proximity in the pose space do not always strictly correspond, confounding factors may cause the six-DOF pose corresponding to the anchor feature identifier based on feature matching to be not the most relevant reference six-DOF pose, thus affecting the accuracy of six-DOF pose regression. Meanwhile, existing map-free visual localization technologies generally use multi-layer perceptrons (MLPs) as pose regressors, but MLPs generally lack geometric constraints and interpretability, resulting in a sharp drop in pose prediction accuracy when faced with unseen viewpoints or scene changes.

[0034] Based on this, embodiments of this application provide a mapless visual localization method. The method includes acquiring scene images collected in real time by a robot and extracting an initial feature map of the scene images using a pre-trained visual encoder; enhancing the initial feature map into a voxel feature map using a sparse kernel self-attention mechanism; aggregating the voxel feature maps from several dimensions and determining image feature identifiers through feature fusion; searching for at least one anchor point feature identifier in a reference image database based on the image feature identifiers through debiased anchor point registration; and determining the robot's six-degree-of-freedom pose based on geometric information pose regression according to the anchor point feature identifiers. This application utilizes a sparse kernel self-attention mechanism to enhance the initial feature map using voxels and aggregates the voxel feature maps from several dimensions, achieving cross-domain interactive sparse quantization. This overcomes feature interference caused by motion blur or illumination changes, improves the robustness of scene image feature extraction, thereby reducing the probability of matching process failure or inaccurate matching results and improving the accuracy of mapless visual localization.

[0035] Furthermore, this application uses nearest neighbor graph retrieval and autoregressive anchor point attribution to match anchor point feature identifiers for scene images. This eliminates cluttered features in feature registration, ensuring that the six-DOF pose corresponding to the anchor point feature identifiers based on feature matching is the most relevant reference six-DOF pose, further improving the accuracy of map-free visual localization. Moreover, during pose regression, a symmetric exchange-gated multilayer perceptron is used to determine the relative pose error, enhancing the geometric constraints on the pose regressor and improving the accuracy of the six-DOF pose solution, further enhancing the accuracy of map-free visual localization.

[0036] The application content will be further explained below with reference to the accompanying drawings and the description of the embodiments.

[0037] This embodiment provides a map-free visual positioning method, such as... Figure 1 and Figure 2 As shown, the map-free visual positioning method specifically includes: S10. Acquire scene images collected by the robot in real time, and extract the initial feature map of the scene images using a pre-trained visual encoder.

[0038] Specifically, the scene image is an image of the current scene captured by an image acquisition device (such as a camera) mounted on the robot. For example, the scene image is an RGB image of the current scene captured by a camera mounted on the robot.

[0039] The initial feature map is obtained by feature map encoding of the scene image, where, for example... Figure 3As shown, a Vision Mamba (ViM) encoder can be used to encode feature maps of the scene image to extract initial feature maps, effectively capturing the global receptive field while maintaining linear computational complexity. The initial feature map extraction process can be represented as follows: , in, Represents a scene image. Represents the initial feature map. Indicates ViM encoder, Indicates the height of the spatial domain. Indicates the width of the spatial domain. Indicates the channel dimension.

[0040] Of course, in practical applications, other network models that can encode features of scene images can also be used to extract initial feature maps, such as Residual Network (ResNet) and Vision Transformer (ViT), etc., without making specific restrictions here.

[0041] S20. The initial feature map is enhanced into a voxel feature map through a sparse kernel self-attention mechanism.

[0042] Specifically, the voxel feature map is obtained by applying a sparse kernel self-attention mechanism to the initial feature map. This captures the non-local dependencies of features at different locations within the initial feature map while filtering out invalid and redundant voxel interactions. The sparse kernel self-attention mechanism is formed using a Gaussian kernel and a sparse response extended self-attention mechanism. This mechanism not only captures the non-local dependencies in the initial feature map but also filters out redundant interaction terms by using sparse response scores based on these non-local dependencies, reducing unnecessary computational overhead.

[0043] In one embodiment, enhancing the initial feature map into a voxel feature map using a sparse kernel self-attention mechanism specifically includes: S21. Capture the nonlocal dependencies of the initial feature map through Gaussian kernel self-attention, and determine the sparse response map using the sparse maximum function; S22. Perform a residual connection between the sparse response map and the initial feature map to obtain a voxel feature map.

[0044] In step S21, nonlocal dependencies are used to reflect the degree of dependency between any two feature positions in the initial feature map. Gaussian kernel self-attention is used to calculate the similarity between any two feature positions in the initial feature map, and sparse maximum value (Sparsemax) is used to capture sparse nonlocal dependencies in the initial feature map. Specifically, when using Gaussian kernel self-attention to capture the nonlocal dependencies in the initial feature map, a query vector, key vector, and value vector can be constructed based on the initial feature map first, and then the global similarity between the query vector and the key vector can be calculated using the Gaussian kernel function.

[0045] Based on this, in one embodiment, capturing the nonlocal dependencies of the initial feature map through Gaussian kernel self-attention specifically includes: Based on the initial feature map, construct the query vector, key vector, and value vector; Based on the query vector and the key vector, the similarity between any two feature positions in the initial feature map is calculated using a Gaussian kernel function to obtain a non-local dependency relationship.

[0046] Specifically, both the query vector and the key vector are obtained by projecting the initial feature map using a learnable projection matrix; that is, as shown in the figure... Figure 3 As shown, after obtaining the initial feature map, the initial feature map is projected using the learnable query projection matrix and the learnable key projection matrix to obtain the query vector, key vector, and value vector, respectively. Then, the exponential Euclidean distance between the query vector and the key vector is calculated using a Gaussian kernel function to obtain the global similarity. Finally, an affinity matrix is ​​constructed based on all the calculated global similarities. The affinity matrix can be represented as: , in, This indicates a learnable query projection matrix. Represents the learnable key projection matrix. Represents the initial feature map. Represents the Gaussian kernel function. Indicates the scaling factor. This represents the affinity matrix.

[0047] Furthermore, in step S22 above, after obtaining the affinity matrix, the sparse response map can be determined by combining the affinity matrix with the value vector of the initial feature map. The value vector is obtained by projecting the initial feature map using a learnable value projection matrix. Multiplying the affinity matrix with the value vector yields the sparse response map.

[0048] However, in practical applications, to further improve the sparsity of voxel feature maps, the Sparsemax activation function can be used instead of Softmax to prevent overfitting when obtaining the sparse response map, thereby enhancing the robustness of features to motion blur and sudden changes in illumination. Specifically, when determining the sparse response map, the affinity matrix is ​​first sparsified using the Sparsemax activation function, and then the sparsified affinity matrix is ​​multiplied by the value vector to determine the sparse response map.

[0049] Based on this, in one embodiment, the sparse response map can be represented as: , in, Represents a sparse response diagram. Represents the affinity matrix. Represents the maximum value function. This represents the threshold cutoff function. This represents the position embedding function corresponding to the value vector. Represent a A one-dimensional matrix.

[0050] Further, in step S23, after obtaining the sparse response map, it can be restored to the image scale of the initial feature map through inverse projection. Then, the restored sparse response map is added to the initial feature map in residual form to obtain a voxel feature map. This voxel feature map can be represented as: , in, Represents voxel feature maps, Represents the initial feature map. Represents a sparse response diagram. This represents the inverse projection weight matrix.

[0051] This application embodiment uses a sparse kernel self-attention mechanism for feature enhancement, which can filter out redundant feature interactions while retaining key feature information. This can reduce the number of parameters and computational load in subsequent calculations, and also reduce the impact of interference factors such as illumination changes and dynamic occlusion on feature robustness.

[0052] S30. Aggregate the voxel feature maps from several dimensions respectively, and determine the image feature identifier through feature fusion.

[0053] Specifically, the dimensions may include one or more of spatial, frequency, and channel dimensions. The aggregated features obtained by aggregating the various dimensions are then fused to achieve a multi-dimensional cross-domain aggregated description. Among them, spatial dimension aggregation is used to extract global information in the spatial direction to preserve the overall spatial structure information of the scene; frequency dimension aggregation is used to select effective feature components from the frequency domain perspective to suppress high-frequency noise; and channel dimension aggregation is used to integrate feature information from different channels to enhance the discriminative channel.

[0054] In one embodiment, the dimensions include spatial, frequency, and channel dimensions. That is, aggregation is first performed in the spatial, frequency, and channel dimensions respectively, and then the aggregated feature vectors are multiplied by a dot product and summed in the spatial domain to achieve a ternary cross-domain aggregation description. Specifically, the aggregation of the voxel feature maps from several dimensions and the determination of image feature identifiers through feature fusion include: Channel domain summation is performed on the voxel feature map to obtain the spatial dimension feature map; The voxel feature map is corrected for frequency dimension to obtain a frequency dimension feature map; The voxel feature map is subjected to indicator compression and spatial domain summation to obtain the channel response, and the channel response is logarithmically adjusted to obtain the channel dimension feature vector. The spatial dimension feature map, the frequency dimension feature map, and the channel dimension feature vector are fused to obtain an image feature identifier.

[0055] Specifically, such as Figure 3 As shown, during spatial dimension aggregation, the voxel feature map is first summed in the channel domain, and then the resulting spatial response is scaled and its dimensions restored. The spatial dimension feature map can be represented as: , , in, Represents spatial dimension feature map, Indicates spatial response, Represents voxel feature maps, and Indicates the scaling factor. Represent a A one-dimensional matrix Represent a A one-dimensional matrix Indicates the height of the spatial domain. Indicates the width of the spatial domain.

[0056] Furthermore, when performing frequency-dimensional aggregation, adaptive frequency convolution can be used to correct the frequency components of the voxel feature map. For example... Figure 3 As shown, the correction process can be as follows: First, perform a fast Fourier transform on the voxel feature map to obtain the frequency domain features; The real and imaginary parts of the frequency domain features are separated, and the real and imaginary parts are concatenated to form channel features; Group convolution operations are performed on channel features to obtain intermediate features, in order to model frequency domain dependencies; The intermediate features are split into channels to obtain the real and imaginary parts after convolution. The real and imaginary parts after convolution are then combined to form the frequency domain features after convolution. The frequency domain features after convolution are transformed to the spatial domain by inverse Fourier transform. The frequency domain features transformed to the spatial domain are then residually connected with the voxel feature map to obtain the frequency dimension feature map, which integrates the original spatial information into the frequency dimension feature map.

[0057] Based on this, the frequency dimension feature map can be represented as: , in, Represents the frequency dimension feature map. This represents the 2D Fast Fourier Transform. This represents the 2D inverse fast Fourier transform. This indicates extracting the real part. This indicates the extraction of the imaginary part. This represents the weight matrix of a 1×1 grouped convolution. Represents 2D convolution. Represents a voxel feature map.

[0058] Furthermore, when performing channel-dimensional aggregation, the voxel feature map can be subjected to indicator compression and spatial domain summation to obtain the channel response, and then the channel response can be normalized to obtain the channel-dimensional feature vector. For example... Figure 3 As shown, the process of obtaining the feature vector of this channel dimension can be as follows: The original spatial activation information of the voxel feature map is extracted using an indicator; The original spatial activation information is subjected to global pooling in the spatial domain to compress the spatial information of each channel into a single scalar, and the initial channel response is determined based on all the single scalars. Apply positive constraints to the initial channel response to obtain the channel response, such that the channel response retains only the positive activation; The channel response is logarithmically adjusted to obtain the channel dimension feature vector.

[0059] Based on this, the feature vector of this channel dimension can be represented as: , , in, Represents the channel-dimensional feature vector. Represents the robustness constant. Indicates channel response, Indicates an indicator function, Represent a A one-dimensional matrix Represent a A one-dimensional matrix Indicates the height of the spatial domain. Indicates the width of the spatial domain. Represents a voxel feature map.

[0060] Furthermore, after obtaining the spatial dimension feature map, frequency dimension feature map, and channel dimension feature vector, the spatial dimension feature map and the frequency dimension feature map are first fused to obtain a spatial-frequency dimension feature vector. Then, the spatial-frequency dimension feature vector and the channel dimension feature vector are multiplied element-wise to obtain the image feature identifier. Specifically, fusing the spatial dimension feature map, the frequency dimension feature map, and the channel dimension feature vector to obtain the image feature identifier includes: The spatial dimension feature map is transposed and multiplied with the frequency dimension feature map by a dot product and summed in the spatial domain to obtain the spatial-frequency dimension feature vector. The spatial-frequency dimension feature vector and the channel dimension feature vector are multiplied element-wise to obtain the image feature identifier.

[0061] Specifically, such as Figure 3 As shown, after obtaining the spatial dimension feature map, frequency dimension feature map, and channel dimension feature vector, the spatial dimension feature map and the frequency dimension feature map are first multiplied by their transposes and summed in the spatial domain to obtain the spatial-frequency dimension feature vector. Then, the spatial-frequency dimension feature vector and the channel dimension feature vector are multiplied by a Hadamard product to obtain the image feature identifier. The image feature identifier can be represented as: , , in, Represents image feature identifiers. Represents the spatial-frequency dimension feature vector. Represents the channel-dimensional feature vector. Represents the frequency dimension feature map. Represents spatial dimension feature map, Represent a A one-dimensional matrix Indicates the height of the spatial domain. Indicates the width of the spatial domain. It represents the Hadamardi (or Hadama) stack.

[0062] S40. Based on the image feature identifier, at least one anchor point feature identifier is searched in the reference image database through debiased anchor point registration.

[0063] Specifically, the reference image database is pre-established and includes multiple reference image feature identifiers. Each reference image feature identifier corresponds to a reference image, which carries a pre-calibrated reference six-DOF pose label. The process for obtaining the reference image feature identifiers is the same as the process for obtaining the image feature identifiers described above, and will not be repeated here. Of course, in practical applications, a set of reference images can also be stored, and when map-free visual localization is needed, the reference image feature identifiers of each reference image can be extracted using steps S10-S3 above to obtain the reference image database. The anchor point feature identifiers found can be one, two, or more, depending on the actual needs; no restrictions are imposed here.

[0064] Furthermore, when searching for anchor point feature identifiers based on the image feature identifiers through debiased anchor point registration, the similarity between the image feature identifier and each reference image feature identifier in the reference image database can be used for selection, or it can be selected through nearest neighbor graph retrieval and autoregressive anchor point attribution.

[0065] In the embodiments of this application, such as Figure 3 As shown, an anchor point feature identifier is selected using a combination of nearest neighbor graph retrieval and autoregressive anchor point attribution. Specifically, the relative pose difference between the scene image and the reference image is estimated using nearest neighbor graph retrieval and autoregressive anchor point attribution, and the anchor point feature identifier is selected based on this relative pose difference. This ensures the geometric consistency between the scene image and the anchor point feature identifier.

[0066] Therefore, the step of searching for at least one anchor point feature identifier in the reference image database based on the image feature identifier through debiased anchor point registration specifically includes: S41. Retrieve a candidate set of nearest neighbor features from the reference image feature identifier library corresponding to the reference image database using nearest neighbor graph retrieval; S42. Determine at least one anchor feature identifier based on the nearest neighbor feature candidate set using autoregressive anchor attribution.

[0067] In step S41, the nearest neighbor graph retrieval is a fast similarity retrieval method based on the nearest neighbor graph. It is used to retrieve a preset number of nearest reference image feature identifiers from the reference image feature identifier library corresponding to the reference image database, forming a nearest neighbor feature candidate set. When retrieving the preset number of nearest reference image feature identifiers, incremental retrieval can be performed based on Euclidean distance, or incremental retrieval based on cosine distance, etc.

[0068] In one embodiment, an incremental retrieval method based on cosine distance is employed. Specifically, when retrieving the nearest neighbor feature candidate set from the reference image feature identifier library corresponding to the reference image database, the Hierarchical Navigable Small World (HNSW) algorithm is used to retrieve a predetermined number of reference image feature identifiers from the reference image database to obtain the nearest neighbor feature candidate set. The HNSW algorithm's retrieval process starts from a random entry point at the top layer of the HNSW graph and iteratively traverses the neighboring reference image feature identifiers with the smallest cosine distance to the image feature identifier, then descends layer by layer until the predetermined number of reference image feature identifiers are identified at the bottom layer. The nearest neighbor feature candidate set can be represented as: , in, Represents the candidate set of nearest neighbor features. Indicates a reference image database, Represents image feature identifiers, Indicates the feature identifier of the reference image. Indicates the first Index of a reference image feature identifier, This represents the number of reference image feature identifiers in the nearest neighbor feature candidate set.

[0069] In step S42, after retrieving the nearest neighbor feature candidate set, an anchor feature identifier matching the scene image is obtained based on the nearest neighbor feature candidate set using autoregressive anchor attribution. Autoregressive anchor attribution is used to extract anchor feature identifiers with geometric consistency with the scene image. Specifically, the process of obtaining anchor feature identifiers matching the scene image based on the nearest neighbor feature candidate set using autoregressive anchor attribution can be as follows: a state-space model is used to model the temporal dependencies of the nearest neighbor feature candidate set to obtain a modeling feature sequence; a multi-head attention mechanism is used to capture the global dependencies of the modeling feature sequence to obtain an attention feature sequence; and a causal mask pointer is used to determine at least one anchor feature identifier based on the attention feature sequence. Furthermore, while determining at least one anchor feature identifier, the confidence score of that anchor feature identifier is also output. The confidence score is derived from the final hidden state after autoregressive anchor attribution processing.

[0070] Furthermore, such as Figure 3 As shown, autoregressive anchor attribution can be performed through an autoregressive anchor attribution unit, which may include several cascaded causal Mamba pointer layers and activation function layers. Each causal Mamba pointer layer includes a self-state space model (Self-SSM), residual connections and layer normalization (Add&Norm), multi-head attention mechanism, Add&Norm, feed forward neural network, Add&Norm, and causal masked pointer. The activation function layer can use the Log Softmax activation function. After obtaining the nearest neighbor feature candidate set, this application inputs the nearest neighbor feature candidate set and image feature identifier into the autoregressive anchor attribution unit, and determines the anchor feature identifier and its corresponding confidence score through the autoregressive anchor attribution unit.

[0071] This application embodiment uses an autoregressive anchor attribution unit for debiased anchor registration, which not only ensures linear computational complexity consistent with the visual Mamba encoder, but also effectively overcomes the limitation of the original Mamba's limited autoregressive capability.

[0072] S50. Based on the anchor point feature identifier, determine the six-degree-of-freedom pose of the robot using geometric information pose regression.

[0073] Specifically, the six-degree-of-freedom pose is the translation vector and rotation quaternion of the robot in the world coordinate system, which can completely describe the robot's current spatial position and pose. After obtaining the matched anchor point feature identifiers, the anchor point images corresponding to the anchor point feature identifiers can be pre-calibrated, and the robot's six-degree-of-freedom pose corresponding to the scene image can be solved by combining the relative pose transformation between the scene image and the anchor point image.

[0074] In this embodiment, when determining the six-DOF pose of the robot based on geometric information pose regression, two decoupled symmetric exchange-gated MLPs are used to regress the translation and rotation errors respectively. Then, the six-DOF pose of the robot is determined using the regressed translation and rotation errors. Therefore, determining the six-DOF pose of the robot based on geometric information pose regression according to the anchor point feature identifier specifically includes: The gating vector and value vector corresponding to the anchor point feature identifier are determined using a symmetric exchange-gated multilayer perceptron, and the relative pose error of the robot is determined using the gating vector and value vector. The robot's six-degree-of-freedom pose is obtained by using the reference six-degree-of-freedom pose corresponding to the anchor point feature identifier and the relative pose error to determine the predicted absolute pose.

[0075] Specifically, the gate vector and value vector are obtained by performing a linear mapping of the anchor feature identifier using a symmetric exchange gating mechanism based on a multilayer perceptron (MLP). That is, first, the anchor feature identifier is linearly mapped using an MLP, and then the linearly mapped anchor feature identifier is segmented into a gate vector and a value vector. Then, the gate vector and value vector are weighted and fused to obtain a fused feature identifier, which is then input into a linear layer to obtain the relative pose error, where, for example... Figure 3 As shown, the fused feature identifier and relative pose error can be expressed as: , , in, Indicates the fusion feature identifier. express Activation function Indicates learnable route weights. Represents the gate vector, Represents a value vector. express Activation function Indicates translation error. Indicates rotational error. This indicates the relative pose error.

[0076] Furthermore, after calculating the relative pose error, the reference six-DOF pose corresponding to the anchor point feature identifier is obtained. Then, by combining the relative pose error with the reference six-DOF pose, the predicted absolute pose can be determined. This predicted absolute pose can be expressed as: , , , in, This indicates the predicted absolute pose. This indicates the predicted absolute translation vector. This indicates the prediction of absolute rotation quaternions. To represent quaternion multiplication, the resulting quaternions are all normalized to a unit length to avoid inaccuracies.

[0077] It should be noted that since single-frame mode and multi-frame mode can select one or more anchor feature identifiers to determine the robot's six-degree-of-freedom (6DOF) pose, when using anchor feature identifiers to determine the robot's 6DOF pose, the number of anchor feature identifiers can be read first. When the number is 1, the single-frame mode determines the robot's 6DOF pose; when the number is greater than or equal to 2, the multi-frame mode determines the robot's 6DOF pose. Specifically, in single-frame mode, the predicted absolute pose determined based on the anchor feature identifiers is directly used as the robot's 6DOF pose. In multi-frame mode, the predicted absolute pose corresponding to each anchor feature identifier is first obtained, then the confidence score corresponding to each anchor feature identifier is used as a weighting coefficient, and the predicted absolute poses corresponding to each anchor feature identifier are weighted and averaged to obtain the target predicted absolute pose, which is then used as the robot's 6DOF pose.

[0078] This application embodiment, by configuring single-frame mode and multi-frame mode, can determine the number of anchor point feature identifiers to be acquired according to actual needs, and call single-frame mode or multi-frame mode according to the number to determine the robot's six-degree-of-freedom pose. This can meet both positioning tasks with high positioning efficiency and positioning tasks with high positioning accuracy. In particular, in environments with visual blur or dynamic interference, multi-frame mode fuses information from multiple anchor point feature identifiers to smooth errors, which can ensure the accuracy of the robot's six-degree-of-freedom pose.

[0079] In summary, this embodiment provides a map-free visual localization method. The method includes acquiring scene images collected in real time by a robot and extracting an initial feature map of the scene images using a pre-trained visual encoder; enhancing the initial feature map into a voxel feature map using a sparse kernel self-attention mechanism; aggregating the voxel feature maps from several dimensions and determining image feature identifiers through feature fusion; searching for at least one anchor point feature identifier in a reference image database based on the image feature identifiers through biased anchor point registration; and determining the robot's six-degree-of-freedom pose based on geometric information pose regression according to the anchor point feature identifiers. This application utilizes a sparse kernel self-attention mechanism to enhance the initial feature map using voxels and aggregates the voxel feature maps from several dimensions, achieving cross-domain interactive sparse quantization, overcoming feature interference caused by motion blur or illumination changes, improving the robustness of scene image feature extraction, thereby reducing the probability of matching process failure or inaccurate matching results and improving the accuracy of map-free visual localization.

[0080] Furthermore, this application uses nearest neighbor graph retrieval and autoregressive anchor point attribution to match anchor point feature identifiers for scene images, which eliminates confounding factors in feature registration. This ensures that the six-DOF pose corresponding to the anchor point feature identifiers based on feature matching is the most relevant reference six-DOF pose, further improving the accuracy of map-free visual localization. Moreover, during pose regression, a symmetric exchange gating mechanism is used to determine the relative pose error, which enhances the geometric constraints on the pose regressor, improves the accuracy of six-DOF pose solving, and further enhances the accuracy of map-free visual localization.

[0081] Furthermore, in practical applications, the map-free visual positioning method provided in this application can utilize a map-free visual positioning model. After acquiring the scene image, the scene image can be input into the map-free visual positioning model, which then outputs the robot's six-degree-of-freedom pose. For example, Figure 2 and Figure 3 As shown, the mapless visual localization model may include a visual encoding module, a cross-domain interactive sparse quantization module, a bias-reduced anchor registration module, and a geometric information pose regression module. The visual encoding module is used to extract the initial feature map of the scene image using a pre-trained visual encoder. The cross-domain interactive sparse quantization module is used to enhance the initial feature map into a voxel feature map through a sparse kernel self-attention mechanism, and aggregate and fuse the voxel feature map from several dimensions respectively. The image feature identifier is determined through feature fusion. The bias-reduced anchor registration module is used to search for at least one anchor feature identifier in the reference image database based on the image feature identifier through bias-reduced anchor registration. The geometric information pose regression module is used to determine the six-degree-of-freedom pose of the robot based on the anchor feature identifier and geometric information pose regression.

[0082] Furthermore, the cross-domain interactive sparse quantization module may include a sparse kernel self-attention mechanism unit and a ternary cross-domain aggregation description unit. The sparse kernel self-attention mechanism unit is used to enhance the initial feature map into a voxel feature map through the sparse kernel self-attention mechanism. The ternary cross-domain aggregation description unit is used to aggregate the voxel feature map from several dimensions respectively, and determine the image feature identifier through feature fusion. The sparse kernel self-attention mechanism unit may include parallel query mapping blocks, key mapping blocks, and value mapping blocks. Both the query mapping blocks and key mapping blocks are connected to a Gaussian function layer, which is implemented by a radial basis function (RBF) and connected to a sparsemax activation function layer. The key-value block is connected to a transpose layer. The outputs of the transpose layer and the sparsemax activation function layer are multiplied. The multiplier, recovery layer, and deconvolution layer are cascaded sequentially, and their outputs are residually concatenated with the initial feature map through an adder to obtain the voxel feature map. The query mapping block, key mapping block, and value mapping block all include convolutional layers and flattening layers.

[0083] like Figure 3 As shown, the ternary cross-domain aggregation description unit includes a spatial branch, a frequency branch, and a channel branch. The spatial branch is used to perform channel domain summation on the voxel feature map to obtain a spatial dimension feature map. The frequency branch is used to perform frequency dimension correction on the voxel feature map to obtain a frequency dimension feature map. The channel branch is used to perform indicator compression and spatial domain summation on the voxel feature map to obtain a channel response, and to perform logarithmic adjustment on the channel response to obtain a channel dimension feature vector. The spatial branch is connected to the frequency branch through a dot product, and its output term is summed in the spatial domain to obtain a spatial-frequency dimension feature vector. The spatial-frequency dimension feature vector and the channel feature vector are multiplied element-wise through a Hadamard product to obtain an image feature identifier.

[0084] Furthermore, such as Figure 3 As shown, the bias-reduction anchor registration module includes a nearest neighbor graph retrieval unit and an autoregressive anchor attribution unit. The nearest neighbor graph retrieval unit is used to retrieve a candidate set of nearest neighbors from the reference image feature identifier library corresponding to the reference image database through nearest neighbor graph retrieval. The autoregressive anchor attribution unit is used to determine at least one anchor feature identifier based on the candidate set of nearest neighbors using autoregressive anchor attribution. Specifically, the nearest neighbor graph retrieval unit uses the HNSW algorithm for retrieval, and the autoregressive anchor attribution unit uses a causal Mamba pointer network for candidate set rearrangement and confidence scoring. The structure of the causal Mamba pointer network has been explained in the above steps.

[0085] Furthermore, such as Figure 3As shown, the geometric information pose regression module can include a translation head and a rotation head, both of which include a gating layer, a symmetric interaction layer, and an activation function layer SiLU.

[0086] Based on the above-described map-free visual positioning method, this embodiment provides a map-free visual positioning device, such as... Figure 4 As shown, the mapless visual positioning device specifically includes: The acquisition module 100 is used to acquire scene images collected by the robot in real time; The control module 200 is used to extract an initial feature map of the scene image using a pre-trained visual encoder, enhance the initial feature map into a voxel feature map through a sparse kernel self-attention mechanism, aggregate the voxel feature map from several dimensions, determine image feature identifiers through feature fusion, search for at least one anchor feature identifier in the reference image database based on the image feature identifiers through debiased anchor registration, and determine the six-degree-of-freedom pose of the robot based on geometric information pose regression according to the anchor feature identifiers, wherein the several dimensions include one or more of spatial dimension, frequency dimension and channel dimension.

[0087] Based on the above-described mapless visual positioning method, this embodiment provides a computer-readable storage medium storing one or more programs that can be executed by one or more processors to implement the steps in the mapless visual positioning method described in the above embodiment.

[0088] Based on the above-mentioned map-free visual positioning method, this application also provides a terminal device, such as... Figure 5 As shown, it includes at least one processor 20; a display screen 21; and a memory 22, and may also include a communications interface 23 and a bus 24. The processor 20, display screen 21, memory 22, and communications interface 23 can communicate with each other via the bus 24. The display screen 21 is configured to display a preset user guide interface in the initial setup mode. The communications interface 23 can transmit information. The processor 20 can invoke logical instructions in the memory 22 to execute the methods described in the above embodiments.

[0089] Furthermore, the logical instructions in the aforementioned memory 22 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium.

[0090] The memory 22, as a computer-readable storage medium, can be configured to store software programs, computer-executable programs, such as program instructions or modules corresponding to the methods in the embodiments of this disclosure. The processor 20 executes functional applications and data processing by running the software programs, instructions, or modules stored in the memory 22, thereby implementing the methods in the above embodiments.

[0091] The memory 22 may include a program storage area and a data storage area. The program storage area may store the operating system and application programs required for at least one function; the data storage area may store data created based on the use of the terminal device. Furthermore, the memory 22 may include high-speed random access memory (RAM) and non-volatile memory. Examples include various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks, as well as transient storage media.

[0092] Furthermore, the specific process of loading and executing multiple instruction processors in the aforementioned storage medium and terminal device has been described in detail in the above method and will not be repeated here.

[0093] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. A map-free visual positioning method, characterized in that, The map-free visual positioning method specifically includes: The robot acquires scene images in real time and uses a pre-trained visual encoder to extract the initial feature map of the scene images. The initial feature map is enhanced into a voxel feature map through a sparse kernel self-attention mechanism; The voxel feature maps are aggregated from several dimensions, and image feature identifiers are determined by feature fusion. The several dimensions include one or more of spatial dimension, frequency dimension and channel dimension. Based on the image feature identifier, at least one anchor point feature identifier is searched in the reference image database by debiased anchor point registration. The six-degree-of-freedom pose of the robot is determined based on the anchor point feature identifier and geometric information pose regression.

2. The map-free visual positioning method according to claim 1, characterized in that, The step of enhancing the initial feature map into a voxel feature map through a sparse kernel self-attention mechanism specifically includes: The nonlocal dependencies of the initial feature map are captured by Gaussian kernel self-attention, and the sparse response map is determined by the sparse maximum function; The sparse response map is residually concatenated with the initial feature map to obtain a voxel feature map.

3. The map-free visual positioning method according to claim 2, characterized in that, The method of capturing the nonlocal dependencies of the initial feature map through Gaussian kernel self-attention specifically includes: Based on the initial feature map, construct the query vector, key vector, and value vector; Based on the query vector and the key vector, the similarity between any two feature positions in the initial feature map is calculated using a Gaussian kernel function to obtain a non-local dependency relationship.

4. The map-free visual positioning method according to claim 1, characterized in that, The aforementioned dimensions include spatial dimension, frequency dimension, and channel dimension. The aggregation of the voxel feature maps from these dimensions, and the determination of image feature identifiers through feature fusion, specifically includes: Channel domain summation is performed on the voxel feature map to obtain the spatial dimension feature map; The voxel feature map is corrected for frequency dimension to obtain a frequency dimension feature map; The voxel feature map is subjected to indicator compression and spatial domain summation to obtain the channel response, and the channel response is logarithmically adjusted to obtain the channel dimension feature vector. The spatial dimension feature map, the frequency dimension feature map, and the channel dimension feature vector are fused to obtain an image feature identifier.

5. The map-free visual positioning method according to claim 4, characterized in that, The process of fusing the spatial dimension feature map, the frequency dimension feature map, and the channel dimension feature vector to obtain the image feature identifier specifically includes: The spatial dimension feature map is transposed and multiplied with the frequency dimension feature map by a dot product and summed in the spatial domain to obtain the spatial-frequency dimension feature vector. The spatial-frequency dimension feature vector and the channel dimension feature vector are multiplied element-wise to obtain the image feature identifier.

6. The map-free visual positioning method according to claim 1, characterized in that, The step of searching for at least one anchor point feature identifier in the reference image database based on the image feature identifier through bias-removing anchor point registration specifically includes: The nearest neighbor feature candidate set is retrieved from the reference image feature identifier library corresponding to the reference image database by means of nearest neighbor graph retrieval; At least one anchor feature identifier is determined based on the nearest neighbor feature candidate set using autoregressive anchor attribution.

7. The map-free visual positioning method according to claim 1, characterized in that, The step of determining the robot's six-degree-of-freedom pose based on geometric information pose regression according to the anchor point feature identifier specifically includes: The gating vector and value vector corresponding to the anchor point feature identifier are determined using a symmetric exchange-gated multilayer perceptron, and the relative pose error of the robot is determined using the gating vector and value vector. The robot's six-degree-of-freedom pose is obtained by using the reference six-degree-of-freedom pose corresponding to the anchor point feature identifier and the relative pose error to determine the predicted absolute pose.

8. A map-free visual positioning device, characterized in that, The map-free visual positioning device specifically includes: The acquisition module is used to acquire scene images collected by the robot in real time; The control module is used to extract an initial feature map of the scene image using a pre-trained visual encoder, enhance the initial feature map into a voxel feature map through a sparse kernel self-attention mechanism, aggregate the voxel feature map from several dimensions, determine image feature identifiers through feature fusion, search for at least one anchor feature identifier in the reference image database based on the image feature identifiers through debiased anchor registration, and determine the six-degree-of-freedom pose of the robot based on geometric information pose regression according to the anchor feature identifiers, wherein the several dimensions include one or more of spatial dimension, frequency dimension and channel dimension.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores one or more programs, which can be executed by one or more processors to implement the steps in the mapless visual positioning method as described in any one of claims 1-7.

10. A terminal device, characterized in that, include: Processor and memory; The memory stores a computer-readable program that can be executed by the processor; When the processor executes the computer-readable program, it implements the steps in the mapless visual positioning method as described in any one of claims 1-7.

Images