A First-Perspective Cognitive Encoding Map Construction and Localization Method Based on Attention Mechanism

Abstract

This invention discloses a first-person cognitive encoding map construction and localization method based on an attention mechanism. The method first captures environmental information to be mapped, and encodes the positional information based on the image's location and pose. Then, it encodes the image information using a ViT architecture to obtain image features. The positional features and image features of consecutive images are input into an attention mechanism to calculate the attention score for each image, simulating the image information reconstructed by the human brain based on positional information. Next, the image information output by the attention mechanism is used for residual connections, and the difference is compared with the real image to calculate the loss and update the network model parameters. Finally, the image to be recognized is input into the trained network model, and by backpropagating the assumed positional information of the input image, the predicted image position is output, completing the localization.

Description

Technical Field

[0001] This invention belongs to the field of image reconstruction technology, and relates to the construction and localization of coded maps, specifically to a method for constructing and localizing first-person cognitive coded maps based on an attention mechanism. Background Technology

[0002] Localization and mapping are fundamental functions of intelligent agents, which rely on onboard sensors to perceive their environment and estimate system state. Examples include autonomous vehicles, delivery drones, and home service robots, which make autonomous decisions by sensing their surroundings. Various mobile and wearable devices, such as smartphones, smart wearables, and IoT devices, provide users with location-based services, from pedestrian navigation in public areas to motion and activity monitoring, and security systems—all of which rely on localization and mapping. Mapping primarily aims to construct a model describing the surrounding environment. The generated map can provide users with an understandable map reference or provide environmental information for robotic tasks, such as navigation planning. Localization refers to estimating the state by comparing perceived data within the constructed map.

[0003] Traditional localization and mapping methods rely on data association results from time-series images, combined with multi-view geometric model constraints, to construct a 3D map and use the constructed map to estimate the position and orientation of the image data. For example, Simultaneous Localization and Mapping (SLAM) first extracts sparse features from the surrounding environment, then performs 3D reconstruction to obtain a 3D map. While moving, it continuously obtains various information about its current position from sensors and performs localization within the constructed 3D map. However, the maps generated by localization and mapping methods only associate significantly similar feature pixels across different images, thus lacking the utilization of image cognitive information and failing to efficiently utilize complete visual image information, resulting in low information utilization. Furthermore, similarity-based data association methods are susceptible to environmental changes, such as dynamic scenes with moving objects or cross-time-period scenes that change over time. Localization and mapping methods lacking scene cognition capabilities still suffer from robustness issues. Summary of the Invention

[0004] To address the shortcomings of existing technologies, this invention proposes a first-view cognitive coding map construction and localization method based on an attention mechanism. This method fully explores the visual information from the first-view perspective, fuses multi-view cognitive coding information based on attention, constructs a cognitive map, and uses this map to understand the cognitive information from multiple perspectives, thereby achieving accurate estimation of image pose information.

[0005] A first-person cognitive coding map construction and localization method based on attention mechanisms includes the following steps:

[0006] Step 1: Use a camera to acquire continuous images from multiple angles and record the camera's internal and external parameters.

[0007] Preferably, the camera is moved and shot at different angles, both horizontally and in pitch, to obtain continuous multi-angle video, which is then decomposed into continuous multi-angle images.

[0008] Step 2: Establish a 3D Cartesian coordinate system [X,Y,Z] for the real world and a 3D Cartesian coordinate system [Xc,Yc,Zc] for the camera's viewpoint. Transform the camera's viewpoint to the real world based on its intrinsic and extrinsic parameters. [Xc,Yc,Zc,1] = Cex[X,Y,Z,1], where Cex represents the camera's extrinsic parameter matrix, containing rotation and translation information. Encode the position information (x,y,z) of the transformed image using Fourier encoding, and then encode the attitude variables (θ,Φ,β) using spherical harmonic functions to obtain the position features.

[0009] Step 3: Encode the multi-angle images obtained in Step 1 using the ViT architecture, converting them into matrix form. Specifically, each original image is divided into multiple sub-images of the same size, arranged into an input sequence and input into a linear projection layer to obtain the feature vector corresponding to each sub-image. After adding positional encoding based on the position of the sub-image in the original image, the image is then passed through an LN layer, a multi-head attention layer, and another LN layer in sequence to complete the transformation from low-dimensional dense feature vectors to high-dimensional sparse feature vectors, thus obtaining image features.

[0010] Step 4: The features of the continuous images obtained in Step 1 are input into the model in the order of shooting time. They are then processed through the attention mechanism and residual calculation to obtain the predicted image features. These features are compared with the true features of the input images, and the loss function value L is calculated. The network parameters are optimized by minimizing the loss function.

[0011] The attention mechanism uses the positional features X of the Nth input image as the Q matrix and the positional features of the N-1 images preceding the current input image as the K matrix, where K = K1, K2, ... K N-1 The image features of the N-1 input images are used as matrix V, where V = V1, V2, ... V N-1 Then according to the formula The system integrates first-person view image information from multiple angles, where s(Ki,Q) represents the similarity between Q and Ki calculated by the dot product model.

[0012] The residual is calculated as LayerNorm(X+Attention(X)), where LayerNorm() is an RNN structure that accelerates convergence by converting the input of each neuron into mean and variance.

[0013] Step 5: Assume the location information of the image to be predicted is X. 1After extracting location and image features through steps 1-3, these features are input into the model trained in step 4. The model then processes the hypothetical information X. 1 Make corrections and output the corresponding predicted location information.

[0014] The present invention has the following beneficial effects:

[0015] 1. This method mimics the process by which the brain builds cognitive maps through groups of place cells, simulates the human brain's use of environmental images, and uses attention to encode and reconstruct images to predict their location information. This reduces the reconstruction process of 3D maps, speeds up the calculation, reduces the amount of computation, and has strong robustness.

[0016] 2. By calculating the feature vector of an image through the ViT architecture, we can efficiently utilize complete visual image information, improve information utilization, and obtain the feature information of the image at the bottom layer. This solves the problem of lack of scene cognition caused by dynamic scenes with moving objects and cross-time scenes that change over time. Attached Figure Description

[0017] Figure 1 A flowchart of a first-person cognitive encoding map construction and localization method based on attention mechanisms;

[0018] Figure 2 This is a schematic diagram of the image feature encoding process in the embodiment;

[0019] Figure 3 This is a schematic diagram of network training in the embodiment. Detailed Implementation

[0020] This method integrates multi-view cognitive coding information based on an attention mechanism to construct a cognitive map, enabling accurate estimation of image pose information. The cognitive map reconstruction simulates the human brain's perception of images from different viewpoints, constructing a scene to predict first-view visual cognitive coding under any viewpoint or pose. The cognitive map-based localization method involves comparing the cognitive coding predicted by prior pose with the cognitive coding based on the actual perceived image, performing multiple backpropagations to correct the prior pose, and estimating the pose information of the image to be judged.

[0021] The invention will be further explained and described below with reference to the accompanying drawings, such as... Figure 1 As shown, the first-person cognitive encoding map construction and localization method based on attention mechanism includes the following steps:

[0022] Step 1: Use a camera to move and capture images of the target environment from different angles, such as horizontal and vertical, to obtain continuous multi-angle video with a resolution of 224x224. Decompose the multi-angle video into frames, and use the resulting images for subsequent network training. Record the camera's intrinsic and extrinsic parameters during shooting.

[0023] Step 2: Establish a 3D Cartesian coordinate system [X,Y,Z] in the real world and a 3D Cartesian coordinate system [Xc,Yc,Zc] from the camera's perspective. Transform the camera's perspective into the real world based on its intrinsic and extrinsic parameters. [Xc,Yc,Zc,1] = Cex[X,Y,Z,1], where Cex represents the camera's extrinsic parameter matrix, containing rotation and translation information. Encode the position information (x,y,z) of the transformed image using the Fourier coding function F() to obtain a feature vector E representing the intrinsic information of the coordinate position. Then, encode the pose variables (θ,Φ,β) using a spherical harmonic function to complete the high-dimensionalization of the original image's position and pose.

[0024] Step 3: Encode the image obtained in Step 1 using the ViT architecture to obtain image features in matrix form. The specific process is as follows: Figure 2 As shown: The image obtained in step 1 is segmented into sub-images of size 16×16. Each sub-image can generate (224×224) / (16×16) = 196 sub-images. The sub-images are arranged into a sequence of length 196 and input into a linear projection layer. Since the dimension of each sub-image is 16×16×3 = 768, the dimension of the linear projection layer is set to 768×768, so that the sequence obtains a feature matrix of size 196×768 after passing through the linear projection layer. After adding positional encoding according to the position of the sub-image in the original image, it is then passed through an LN layer, a multi-head attention layer, and another LN layer in sequence to obtain an image feature of size 196×768. This process efficiently utilizes the visual information in the original image to complete the transformation from a low-dimensional dense feature vector to a high-dimensional sparse feature vector.

[0025] Step 4, as follows Figure 3 As shown, the features of the continuous images obtained in step 1 are input into the model in the order of shooting time. They are then processed through an attention mechanism and residual calculation to obtain the predicted image features. These predicted features are compared with the true features of the input images, and the loss function value L is calculated.

[0026]

[0027] Where S and L represent the predicted image features and the real image features of the input image, respectively, Cov(S, L) represents the covariance, and Var(S) and Var(L) represent the variances of S and L, respectively. The network parameters are optimized by minimizing the loss function to minimize the difference between the predicted image and the real image.

[0028] The attention mechanism uses the positional features X of the Nth input image as the Q matrix and the positional features of the N-1 images preceding the current input image as the K matrix, where K = K1, K2, ... K N-1 The image features of the N-1 input images are used as matrix V, where V = V1, V2, ... V N-1 Then according to the formula The system integrates first-person view image information from multiple angles, where s(Ki,Q) represents the similarity between Q and Ki calculated by the dot product model.

[0029] The residual is calculated as LayerNorm(X+Attention(X)), where LayerNorm() is an RNN structure that accelerates convergence by converting the input of each neuron into mean and variance.

[0030] Step 5: Assume the location information of the image to be predicted is X. 1 After extracting location and image features through steps 1-3, these features are input into the model trained in step 4. The model then processes the hypothetical information X. 1 Make corrections and output the corresponding predicted location information.

[0031] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. Although specific embodiments have been described in this description, it is obvious that various modifications and variations can be made without departing from the inventive spirit and scope of these embodiments, and all such changes and improvements fall within the scope of the claimed invention. The scope of protection of this invention is defined by the appended claims and their equivalents.

Claims

1. A first-person cognitive encoding map construction and localization method based on attention mechanism, characterized by: Specifically, the following steps are included: Step 1: Use a camera to acquire continuous images from multiple angles and record the camera's internal and external parameters; Step 2: Establish three-dimensional Cartesian coordinate systems for the real world and the camera's perspective, respectively, and transform the camera's perspective into the real world based on the camera's intrinsic and extrinsic parameters; then encode the position and pose variables (θ, Φ, β) of the transformed image using Fourier coding and spherical harmonic functions to obtain position features; Step 3: Encode the multi-angle images obtained in Step 1 using the ViT architecture to obtain image features; Step 4: Input the features of the continuous images obtained in Steps 2 and 3 into the attention mechanism and residual calculation in the order of shooting time to obtain the predicted image features. Compare them with the real features of the input images, calculate the loss function L, and optimize the network parameters by minimizing the loss function. The attention mechanism uses the positional features X of the Nth input image as the Q matrix and the positional features of the N-1 images preceding the current input image as the K matrix, where K = K1, K2, ... K N-1 The image features of the N-1 input images are used as matrix V, where V = V1, V2, ... V N-1 Then according to the formula Integrate first-person view image information from multiple angles, where s(Ki, Q) represents the similarity between Q and Ki calculated by the dot product model; The residual is calculated as LayerNorm(X+Attention(X)), where LayerNorm() is an RNN structure that accelerates convergence by converting the input of each neuron into mean and variance. Step 5: Assume the location information of the image to be predicted is X. 1 After extracting location and image features through steps 1-3, these features are input into the model trained in step 4. The model then processes the hypothetical information X. 1 Make corrections and output the corresponding predicted location information.

2. The first-person cognitive coding map construction and localization method based on attention mechanism as described in claim 1, characterized in that: The camera is moved and shot at different angles, both horizontally and vertically, to obtain continuous multi-angle video, which is then broken down into continuous images from multiple angles.

3. The first-person cognitive coding map construction and localization method based on attention mechanism as described in claim 1, characterized in that: [Xc,Yc,Zc,1]=Cex[X,Y,Z,1], where [Xc,Yc,Zc] represents the image position information from the camera's perspective, [X,Y,Z] represents the corresponding coordinates in the real world, and Cex represents the camera's extrinsic parameter matrix, which includes rotation and translation information.

4. The first-person cognitive coding map construction and localization method based on attention mechanism as described in claim 1, characterized in that: The specific process of encoding image information using the ViT architecture is as follows: each original image is divided into multiple sub-images of the same size, arranged into an input sequence and input into a linear projection layer to obtain the feature vector corresponding to each sub-image. After adding positional encoding according to the position of the sub-image in the original image, it is then passed through an LN layer, a multi-head attention layer, and another LN layer in sequence to complete the transformation from low-dimensional dense feature vector to high-dimensional sparse feature vector, thus obtaining image features.

5. The first-person cognitive coding map construction and localization method based on attention mechanism as described in claim 1, characterized in that: The loss function is defined as follows: Where S and L represent the predicted image features and the real image features of the input image, respectively, Cov(S,L) represents the covariance, and Var(S) and Var(L) represent the variances of S and L, respectively; the network parameters are optimized by minimizing the loss function to minimize the difference between the predicted image and the real image.

Images