A visual self-localization method for unmanned systems based on an end-to-end feature optimization model

Abstract

This invention discloses a method for visual self-localization of unmanned systems based on an end-to-end feature optimization model. The method includes the following steps: first, constructing a model; constructing a deep neural network model for end-to-end global pose prediction, namely the Re-PoseNet network, or Re-PoseNet model; second, training the model; training the Re-PoseNet model for a specific scene; and third, applying the model; inputting images captured at the specific scene into the Re-PoseNet model, which returns an end-to-end pose estimation result. This method for visual self-localization of unmanned systems based on an end-to-end feature optimization model has the advantages of high accuracy, good real-time performance, strong robustness, small storage space usage, and applicability to a variety of indoor and outdoor scenarios.

Description

Technical Field

[0001] The present invention belongs to the field of mobile robots and relates to a visual self-positioning method for an unmanned system based on an end-to-end feature optimization model. Background Art

[0002] Self-localization is a crucial technical foundation for fields such as SLAM, autonomous navigation, and augmented reality. It refers to the process by which autonomous unmanned systems calculate their six-degree-of-freedom (6-DoF) pose within a map containing prior environmental information based on information from their onboard sensors. To ensure accurate long-term positioning of mobile robots, self-localization is essential in situations such as severe cumulative errors, pose tracking failures, robot abduction, or power outages and restarts. Position information is also crucial for various intelligent unmanned systems to perform other advanced tasks. Autonomous unmanned systems can generally achieve convenient and efficient self-localization in outdoor environments using satellite positioning systems such as GPS and Beidou. However, due to limitations in satellite signal penetration and positioning accuracy, in GPS-restricted environments, such as indoors, densely populated areas, mountainous areas / forest parks, and special confidentiality-related situations, autonomous unmanned systems must utilize sensors such as cameras or lidar to obtain environmental information. Vision sensors are widely used for self-localization tasks due to their low cost, small size, and wide range of applications. However, in complex dynamic environments, long-term visual self-positioning faces the following difficulties: 1) Changes in natural conditions or human factors in complex environments make the current environmental model of the autonomous unmanned system inconsistent or contradictory with the previously constructed map, resulting in a mismatch between the observed information and the prior map and positioning failure; 2) The presence of highly repetitive structures or sparse textures in the environment causes perceptual confusion or difficulty in capturing effective visual features for the autonomous unmanned system; 3) There are a large number of dynamic targets such as people and vehicles in the actual environment, and the time and number of their appearance are highly uncertain, which will cause great interference to positioning.

[0003] Visual self-localization can be broadly categorized into three types based on pose estimation methods: image retrieval-based methods, indirect estimation methods based on geometric structures, and direct estimation methods based on end-to-end models. Image retrieval-based methods typically consist of a map database consisting of a set of discrete scene images with pose labels. These methods then find the image most similar to the query image in the database and approximate its pose as the query image's pose. This approximate estimation method is considered coarse localization and is more suitable for large-scale scenarios or tasks that do not require high pose accuracy, such as location recognition. Indirect estimation methods based on geometric structures typically construct a 3D point cloud map to represent the environment model, establish corresponding coordinate pairs between the camera image and the 3D environment model, and calculate the camera pose using geometric optimization methods. However, long-term dynamic environments are often subject to negative influences such as changing natural conditions, human factors, and interference from dynamic objects, which can easily lead to erroneous correspondences and thus poor localization results. Furthermore, the large size of the map model and the high computational complexity in large-scale scenarios pose significant challenges to real-time localization. With the rapid development and significant performance of deep learning in computer vision-related fields, direct estimation methods based on end-to-end models have also attracted extensive attention and research. This approach utilizes deep neural networks to construct an end-to-end pose regression model. Specifically, given a query image as input, it directly outputs a 6-DoF pose. PoseNet, a representative model, primarily consists of a visual encoder for feature extraction and a regressor for pose prediction. The visual encoder is implemented using a convolutional neural network and trained end-to-end, implicitly expressing the scene model using the trained network parameters without the need for an explicit map. This groundbreaking network model is simple yet effective, requiring minimal space to store the network weights, and its size does not increase dramatically with scene size. However, in complex and dynamic environments, it faces challenges such as reduced accuracy and insufficient robustness. Some researchers have addressed this challenge by introducing additional sensor information, using continuous image sequences, and combining learning with other tasks. However, these approaches present challenges such as high cost and difficulty balancing accuracy and efficiency.

[0004] In summary, how to construct a learnable end-to-end network model for complex dynamic scenes, so that it pays more attention to scene geometric information and static effective features that are helpful for predicting pose, and realizes global pose estimation based on single-frame images with high robustness and efficiency, is a technical problem that needs to be solved urgently.

[0005] Therefore, it is necessary to design a new visual self-localization method for unmanned systems. Summary of the Invention

[0006] The technical problem to be solved by the present invention is to provide an unmanned system visual self-positioning method based on an end-to-end feature optimization model. The unmanned system visual self-positioning method based on the end-to-end feature optimization model has the advantages of high precision, good real-time performance, strong robustness, small storage space occupation, and applicability to various indoor and outdoor scenarios.

[0007] The technical solutions of the invention are as follows:

[0008] A visual self-localization method for an unmanned system based on an end-to-end feature optimization model comprises the following steps:

[0009] Step 1: Construct the model

[0010] Construct a deep neural network model for end-to-end global pose prediction, namely the Re-PoseNet network, or Re-PoseNet model;

[0011] Step 2: Model training

[0012] Train the Re-PoseNet model for a specific scenario;

[0013] Step 3: Model Application

[0014] The image captured at the specific scene is input into the Re-PoseNet model, and the end-to-end pose estimation result is returned.

[0015] The Re-PoseNet model includes a feature extractor, a feature optimization module and a pose regressor. The feature extractor is based on the residual network ResNet34. The feature optimization module consists of two cross-attention modules and a four-directional long short-term memory module. The pose regressor consists of a fully connected layer.

[0016] The training set used in step 2 is obtained as follows:

[0017] A mobile robot equipped with a visual sensor randomly traverses a scene and collects a set of images and their global poses as a training dataset T to build an implicit map model of the scene. The global pose corresponding to each frame of the image is used as the true value label for subsequent network training, denoted as in Indicates location information. It is the rotation information represented by quaternion.

[0018] The training process is:

[0019] Step (1): For all images I in the training dataset T i Preprocessing is performed to generate training images of size 256×256 with pixel intensities between -1 and 1

[0020] Step (2): Input a set of training images and their pose labels into Re-PoseNet with batch size batch_size=64, obtain the predicted pose [p, q] through the pose regressor, and construct a joint loss function at the same time Calculate the predicted pose [p, q] and its corresponding true value Losses between

[0021] Step (3): Set the initial learning rate of the hyperparameter to 5×10 -5 , the number of iterations is 1000, and the Adam optimizer is used to train the Re-PoseNet network. The loss function is continuously converged and reduced through iterations, and the network parameters are updated until the number of iterations is reached. The loss function is selected The set of model parameters corresponding to the minimum value is used as the optimal implicit map model for the scenario.

[0022] In step 3, the mobile robot operates in the scene in any motion state, and the image I with unknown posture collected at any time is t Perform preprocessing to generate an image of size 256×256 with pixel intensity between -1 and 1 And input it as a query image into the trained Re-PoseNet model with optimal weights;

[0023] Get the query image through the feature extractor Feature map Then the optimized feature vector is obtained through the attention module and long short-term memory unit Finally, the pose regressor outputs the predicted position information With rotation information Obtain end-to-end pose estimation results.

[0024] The specific process of building a deep neural network (Re-PoseNet) for end-to-end global pose prediction is as follows:

[0025] Step 1.1: Design the specific structure of the feature extractor of Re-PoseNet as shown in Table 1. There are 33 convolutional layers and one maximum pooling layer. Taking convolutional layer 1 as an example, its parameter information includes: convolution kernel size is 7×7, number of channels is 64, and stride is 2. The residual layer consists of multiple residual units, each of which contains two convolutional layers with convolution kernel size of 3×3 and stride of 1. For any image I input to the feature extractor, its corresponding feature map can be obtained.

[0026] Table 1 Network structure of Re-PoseNet feature extractor

[0027]

[0028]

[0029] Step 1.2: Use two convolution layers with a convolution kernel of 1×1, a channel of 64, and a stride of 1 to perform convolution operations on the original feature map F, and generate feature maps with the same spatial size but different numbers. and At the same time, a convolution operation is performed on F using a convolution layer with a convolution kernel of 1×1, a channel of 512, and a stride of 1 to generate feature maps of the same size and number.

[0030] Step 1.3: For any position u in the spatial dimension of the feature map F, the global context of the feature at position u is captured by calculating the correlation between the feature at that position and other features in the same row and column:

[0031]

[0032] in, is the vector corresponding to position u in X, is the set Ω u The i-th element in the set The elements in d are composed of vectors corresponding to the same row and column positions as position u in Y, so i = [1, 2, ..., 15], d i，u It is a scalar representing the correlation between features at different positions;

[0033] Step 1.4: Traverse all i and repeat step 1.3 to calculate X u With the set Ω u The correlation d of all elements in i，u , get a set of weight coefficients corresponding to position u;

[0034] Step 1.5: Traverse all positions u in the feature map F space and repeat steps 1.3 and 1.4 to get It saves the correlation degree corresponding to all position features in the feature map F, that is, d i，u ∈D, and then use the Softmax layer to process D in the channel dimension to obtain the attention map

[0035] Step 1.6: Aggregate the attention map A with the feature map Z, and add the aggregation result to the original feature map F through the residual connection to achieve feature adaptation, adjust the network's attention to different feature information, and filter out redundant information to obtain a new feature map. For any position u in the spatial dimension, there is F′ u ∈F′:

[0036]

[0037] Among them, A i，u ∈A, is the set Φ u The i-th element in the set, i = [1, 2, ..., 15] Take the vector at the same row and column as position u in the feature map z, F u and F′ u Respectively represent the features before and after the attention operation at the spatial position u;

[0038] Step 1.7: Repeat steps 1.2 to 1.6 for the feature map F′ to further capture dense context information and readjust the feature weights to obtain a new feature map

[0039] Step 1.8: Use the average pooling layer to aggregate the information of the feature map F″ on each channel to obtain a feature with a dimension of 512×1×1, and output the feature vector through a fully connected layer with 2048 neurons

[0040] Step 1.9: Reshape the eigenvector V′ into a 32×64 matrix. Then, process it using four LSTMs with 256 neurons, using the up, down, left, and right methods. Then, concatenate the four output vectors into a 1024-dimensional eigenvector V.

[0041] Step 1.10: The pose regressor consists of two fully connected layers, containing 3 and 4 neurons respectively. The feature vector V is input into these two fully connected layers, and the predicted position information can be output. With rotation information

[0042] Another way to describe the specific steps of the present invention is as follows:

[0043] A visual self-localization method for an unmanned system based on an end-to-end feature optimization model comprises the following steps:

[0044] Step 1: Use a mobile robot (autonomous unmanned system) equipped with a visual sensor to randomly traverse a scene and collect a set of images and their global poses as a training dataset T to build an implicit map model of the scene. The global pose corresponding to each frame of the image is used as the true value label for subsequent network training, denoted as in Indicates location information. It is the rotation information represented by quaternion;

[0045] Step 2: Build a trainable end-to-end deep neural network (Re-PoseNet) for predicting global pose. The network mainly consists of three parts: feature extractor, feature optimization module and pose regressor.

[0046] Among them, the feature extractor is modified based on the residual network ResNet34, the feature optimization module consists of two cross attention modules and a four-directional long short-term memory module, and the pose regressor consists of a fully connected layer;

[0047] Step 3: Use the publicly available pre-trained model of the ResNet34 network to initialize the parameters of the feature encoder in the Re-PoseNet network, and randomly initialize other network parameters;

[0048] Step 4: For all images I in the training dataset T i Preprocessing is performed to generate training images of size 256×256 with pixel intensities between -1 and 1

[0049] Step 5: Input a set of training images and their pose labels into Re-PoseNet with batch size batch_size=64, obtain the predicted pose [p, q] through the pose regressor, and construct a joint loss function at the same time Calculate the predicted pose [p, q] and its corresponding true value Losses between

[0050] Step 6: Set the initial learning rate of hyperparameters to 5×10 -5 , the number of iterations is 1000, and the Adam optimizer is used to train the network Re-PoseNet. Through iteration, the loss function is continuously converged and reduced, and the network parameters are updated until the number of iterations is reached. The loss function is selected The set of model parameters corresponding to the minimum value is used as the optimal implicit map model for the scenario;

[0051] Step 7: The mobile robot operates in any motion state in the scene, and the image I collected at any time with unknown posture t Perform preprocessing to generate an image of size 256×256 with pixel intensity between -1 and 1 And input it as a query image into the trained Re-PoseNet model with optimal weights;

[0052] Step 8: Get the query image through the feature extractor Feature map Then the optimized feature vector is obtained through the attention module and long short-term memory unit Finally, the pose regressor outputs the predicted position information With rotation information Obtain end-to-end pose estimation results.

[0053] The specific process of constructing the deep neural network (Re-PoseNet) for end-to-end global pose prediction in step 2 is as follows:

[0054] Step 2.1: Modify the classic convolutional neural network ResNet34 and retain its main convolution layer as the feature extractor of Re-PoseNet. Its specific structure is shown in Table 1. There are 33 convolution layers and one maximum pooling layer. Taking convolution layer 1 as an example, its important parameter information includes: convolution kernel size is 7×7, number of channels is 64, and stride is 2. The residual layer is composed of multiple residual units. Each residual unit contains two convolution layers with a convolution kernel of 3×3 and a stride of 1. The number of convolution layer channels of different residual units is adjusted. For any image I input to the feature extractor, its corresponding feature map can be obtained.

[0055] Table 1 Network structure of Re-PoseNet feature extractor

[0056]

[0057] Step 2.2: Use two convolution layers with a convolution kernel of 1×1, a channel of 64, and a stride of 1 to perform convolution operations on the original feature map F, and generate feature maps with the same spatial size but different numbers. and At the same time, a convolution operation is performed on F using a convolution layer with a convolution kernel of 1×1, a channel of 512, and a stride of 1 to generate feature maps of the same size and number.

[0058] Step 2.3: For any position u in the spatial dimension of the feature map F, the global context of the feature at position u is captured by calculating the correlation between the feature at that position and other features in the same row and column:

[0059]

[0060] in, is the vector corresponding to position u in X, is the set Ω u The i-th element in the set The elements in d are composed of vectors corresponding to the same row and column positions as position u in Y, so i = [1, 2, ..., 15], d i，u It is a scalar representing the correlation between features at different positions;

[0061] Step 2.4: Traverse all i and repeat step 2.3 to calculate X uWith the set Ω u The correlation d of all elements in i，u , get a set of weight coefficients corresponding to position u;

[0062] Step 2.5: Traverse all positions u in the feature map F space and repeat steps 2.3 and 2.4 to get It saves the correlation degree corresponding to all position features in the feature map F, that is, d i，u ∈D, and then use the Softmax layer to process D in the channel dimension to obtain the attention map

[0063] Step 2.6: Aggregate the attention map A with the feature map Z, and add the aggregation result to the original feature map F through the residual connection to achieve feature adaptation, adjust the network's attention to different feature information, and filter redundant information to obtain a new feature map. For any position u in the spatial dimension, there is F′ u ∈F′:

[0064]

[0065] Among them, A i，u ∈A, is the set Φ u The i-th element in the set, i = [1, 2, ..., 15] Take the vector at the same row and column as position u in the feature map Z, F u and F′ u Respectively represent the features before and after the attention operation at the spatial position u;

[0066] Step 2.7: Repeat steps 2.2 to 2.6 for the feature map F′ to further capture dense context information and readjust the feature weights to obtain a new feature map

[0067] Step 2.8: Use the average pooling layer to aggregate the information of the feature map F″ on each channel to obtain a feature with a dimension of 512×1×1, and output the feature vector through a fully connected layer with 2048 neurons

[0068] Step 2.9: Reshape the eigenvector V′ into a 32×64 matrix. Then, process it using four LSTMs with 256 neurons, using the up, down, left, and right methods. Then, concatenate the four output vectors into a 1024-dimensional eigenvector V.

[0069] Step 2.10: The pose regressor consists of two fully connected layers, containing 3 and 4 neurons respectively. The feature vector V is input into these two fully connected layers, and the predicted position information can be output. With rotation information

[0070] In step 4, all images I in the training data set T are i The preprocessing process is as follows:

[0071] Step 4.1: Scale all images to a shorter side of 256 pixels, and then randomly crop them to obtain images with a resolution of 256×256.

[0072] Step 4.2: Normalize each image so that its pixel intensity is between -1 and 1:

[0073]

[0074] Among them, μ T and σ T Represent the pixel mean and standard deviation of all images in dataset T respectively.

[0075] Construct the loss function in step 5 Calculate the predicted pose [p, q] and its corresponding true value The process of loss is as follows:

[0076] Step 5.1: Use the 1-norm to calculate the loss between the predicted value and the true value of the position, that is,

[0077] Step 5.2: The rotation information represented by the quaternion is constrained to the hemispherical space by θ = logq to ensure the uniqueness of the value, and the loss between the predicted value and the true value of the rotation is calculated using the 1-norm, that is,

[0078] Step 5.3: Construct a joint loss function to train both losses simultaneously:

[0079]

[0080] Among them, α and γ are learnable hyperparameters. As a kind of homoscedastic uncertainty, they reflect the relative difficulty of different tasks in multi-task joint training. That is, they are used to balance the position and rotation information due to the obvious differences in units and values. The initial values ​​α0 = 0.0 and y0 = -3.0 are set during training.

[0081] In step 7, the query image I i The preprocessing process is as follows:

[0082] Step 7.1: Scale the image so that the shorter side is 256 pixels, and then perform center cropping to obtain a 256×256 resolution image.

[0083] Step 7.2: Normalize the test image in the same way as above so that the pixel intensity of the test image is between -1 and 1:

[0084]

[0085] Beneficial effects:

[0086] The present invention's unmanned system visual self-localization method, based on an end-to-end feature optimization model, utilizes deep learning techniques to construct an end-to-end trainable pose regression model. The proposed feature optimization module helps the network focus on learning scene information and static features that are beneficial for pose estimation. Furthermore, collaborative uncertainty is introduced for joint training to improve network model performance. This method enables real-time self-localization based on single-frame images in dynamic environments over extended periods of time, offering advantages such as high accuracy, strong robustness, minimal storage space requirements, and applicability to a variety of indoor and outdoor scenarios.

[0087] Compared with the existing monocular self-localization method based on the end-to-end model, the present invention is advanced in the following aspects:

[0088] 1. This paper uses a convolutional neural network (ResNet) based on residual units as the feature extractor for Re-PoseNet. Residual connections preserve part of the loss function during backpropagation, increasing network depth while avoiding degradation. This method achieves better performance than the convolutional neural network (GoogleNet) used in the classic PoseNet approach.

[0089] 2. The present invention proposes a feature optimization module that integrates the attention mechanism and the long short-term memory network to redistribute the weights and reconstruct and reduce the dimensionality of the original features learned by the feature extractor. First, two cross-attention modules are used to capture dense global context information, prompting the network to pay more attention to static features and filter out redundant features; then, the features are reconstructed through a four-directional long short-term memory module to retain more information that is beneficial to pose estimation, while reducing the feature dimension and the amount of calculation. Compared with the classic method PoseNet that directly uses the original features for pose estimation, the present invention significantly improves the representativeness and robustness of the features through feature optimization, making the algorithm much more accurate and stable than PoseNet in complex dynamic scenes;

[0090] 3. This paper proposes a joint loss function training strategy based on homoscedastic uncertainty, which jointly learns the position prediction values ​​and orientation prediction values ​​output by the pose regressor. The two balancing factors reflecting the uncertainties of different tasks can automatically find the optimal value during training. The single balancing factor used in the classic method PoseNet requires manual parameter adjustment and is sensitive to different scene types. Therefore, the training strategy of this paper has the advantages of low labor cost, high training efficiency, and better model parameter performance. It is also widely applicable to various indoor and outdoor scenes. BRIEF DESCRIPTION OF THE DRAWINGS

[0091] Figure 1 It is the overall flow chart of the present invention;

[0092] Figure 2 The specific network architecture of the network model Re-PoseNet proposed in this invention;

[0093] Figure 3 Figure 3. Ablation experiment results of the network model Re-PoseNet proposed in this invention on the outdoor dataset Oxford RobotCar (where the black line represents the actual motion trajectory, the gray dots represent the position results calculated using different positioning methods, the network model used by the positioning method in column a uses the modified ResNet34 as the backbone without adding other functional modules, the network model used in column b adds an attention module to the network model in column a, and the network model used in column c adds both an attention module and a long short-term memory module to the network model in column a, i.e., the Re-PoseNet proposed in this invention);

[0094] Figure 4 This is a visualization of the saliency map of image features extracted by the network model Re-PoseNet proposed in this invention and the classic method PoseNet (column a is the original image, column b is the processing effect image using PoseNet, and column c is the processing effect image using Re-PoseNet). DETAILED DESCRIPTION

[0095] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments:

[0096] Example 1:

[0097] The visual self-positioning method for unmanned systems involved in the present invention is based on single-frame images, and the images used in the present invention are all RGB images captured by ordinary cameras. The images contained in the training data set T are collected by a mobile robot equipped with an RGB camera in a specific scene, traversing the scene in arbitrary motion states and trajectories. A global pose label corresponding to each image can be produced using a mature visual odometry or motion recovery algorithm. The network model Re-PoseNet learns an implicit scene map model based on the training data set T. After that, the mobile robot operates in the scene in arbitrary motion states, and can query the global pose of images captured at any time. Taking this as an application example, the present invention provides application effects based on different indoor and outdoor scenes.

[0098] like Figure 1 FIG. 1 is a flow chart of the present invention, which shows a method for visual self-positioning of an unmanned system based on an end-to-end feature optimization model, comprising the following steps:

[0099] Step 1: Use a mobile robot equipped with an RGB camera to randomly traverse a scene and collect a set of images and their global poses as a training dataset T to build an implicit map model of the scene. The global pose corresponding to each frame of the image is used as the true value label for subsequent network training, denoted as in Indicates location information. It is the rotation information represented by quaternion;

[0100] Step 2: Build a trainable end-to-end deep neural network (Re-PoseNet) for predicting global pose. The network mainly consists of three parts: feature extractor, feature optimization module and pose regressor. Its network architecture is as follows: Figure 2 As shown;

[0101] The feature extractor is modified based on the residual network ResNet34. The feature optimization module consists of two cross-attention modules and a four-directional long short-term memory module. The pose regressor consists of a fully connected layer. The specific construction process is as follows:

[0102] 1) The classic convolutional neural network ResNet34 is modified, and its main convolution layer is retained as the feature extractor of Re-PoseNet. Its specific structure is shown in Table 1. There are 33 convolution layers and one maximum pooling layer. Taking convolution layer 1 as an example, its important parameter information includes: convolution kernel size is 7×7, number of channels is 64, and stride is 2. The residual layer is composed of multiple residual units, each of which contains two convolution layers with a convolution kernel of 3×3 and a stride of 1. The number of convolution layer channels of different residual units is adjusted. For any image J input to the feature extractor, its corresponding feature map can be obtained.

[0103] Table 1 Network structure of Re-PoseNet feature extractor

[0104]

[0105] 2) Use two convolution layers with a convolution kernel of 1×1, a channel of 64, and a stride of 1 to perform convolution operations on the original feature map F, and generate feature maps with the same spatial size but different numbers. and At the same time, a convolution operation is performed on F using a convolution layer with a convolution kernel of 1×1, a channel of 512, and a stride of 1 to generate feature maps of the same size and number.

[0106] 3) For any position u in the spatial dimension of the feature map F, the global context information of the position u feature is captured by calculating the correlation between the position feature and other features in the same row and column:

[0107]

[0108] in, is the vector corresponding to position u in X, is the set Ω u The i-th element in the set The elements in d are composed of vectors corresponding to the same row and column positions as position u in Y, so i = [1, 2, ..., 15], d i，u It is a scalar representing the correlation between features at different positions;

[0109] 4) Traverse all i and repeat step 2.3 to calculate X u With the set Ω u The correlation d of all elements in i，u , get a set of weight coefficients corresponding to position u;

[0110] 5) Traverse all positions u in the feature map F space and repeat steps 2.3 and 2.4 to get It saves the correlation degree corresponding to all position features in the feature map F, that is, d i，u ∈D, and then use the Softmax layer to process D in the channel dimension to obtain the attention map

[0111] 6) Aggregate the attention map A with the feature map Z, and add the aggregation result to the original feature map F through the residual connection to achieve feature adaptation, adjust the network's attention to different feature information, and filter redundant information to obtain a new feature map For any position u in the spatial dimension, there is F′ u ∈F′:

[0112]

[0113] Among them, A i，u ∈A, is the set Φ u The i-th element in the set, i = [1, 2, ..., 15] Take the vector at the same row and column as position u in the feature map Z, F u and F′ u Respectively represent the features before and after the attention operation at the spatial position u;

[0114] 7) Repeat steps 2.2 to 2.6 for the feature map F′ to further capture dense context information and readjust the feature weights to obtain a new feature map

[0115] 8) Use the average pooling layer to aggregate the feature map F, and obtain the features of dimension 512×1×1, and output the feature vector through a fully connected layer with 2048 neurons.

[0116] 9) Reshape the eigenvector V′ into a 32×64 matrix, then process it using four long short-term memory units with 256 neurons, using the up, down, left, and right methods respectively. Then, concatenate the four output vectors into a eigenvector V with a dimension of 1024.

[0117] 10) The pose regressor consists of two fully connected layers, containing 3 and 4 neurons respectively. The feature vector V is input into these two fully connected layers respectively, and the predicted position information can be output. With rotation information

[0118] Step 3: Use the publicly available pre-trained model of the ResNet34 network to initialize the parameters of the feature encoder in the Re-PoseNet network, and randomly initialize other network parameters;

[0119] Step 4: For all images I in the training dataset T i Preprocessing is performed to generate training images of size 256×256 with pixel intensities between -1 and 1

[0120] The pre-processing process is as follows:

[0121] 1) All images are scaled to a shorter side of 256 pixels, and then randomly cropped to obtain images with a resolution of 256×256;

[0122] 2) Calculate the pixel average μ of all images in the dataset T T and standard deviation σ T , normalize each image so that its pixel intensity is between -1 and 1:

[0123]

[0124] Step 5: Input a set of training images and their pose labels into Re-PoseNet with batch size batch_size=64, obtain the predicted pose [p, q] through the pose regressor, and construct a joint loss function at the same time Calculate the predicted pose [p, q] and its corresponding true value Losses between

[0125] The construction of the joint loss function The main process is as follows:

[0126] 1) Use the 1-norm to calculate the loss between the predicted value and the true value of the position, that is,

[0127] 2) The rotation information represented by the quaternion is constrained to the hemispherical space through θ = logq to ensure the uniqueness of the value, and the loss between the predicted value and the true value of the rotation is calculated using the 1-norm, that is,

[0128] 3) Construct a joint loss function to train both losses simultaneously:

[0129]

[0130] Among them, α and γ are learnable hyperparameters. As a kind of homoscedastic uncertainty, they reflect the relative difficulty of different tasks in multi-task joint training. That is, they are used to balance the position and rotation information due to the obvious differences in units and values. The initial values ​​α0 = 0.0 and γ0 = -3.0 are set during training.

[0131] Step 6: Set the initial learning rate of hyperparameters to 5×10 -5 , the number of iterations is 1000, and the Adam optimizer is used to train the network Re-PoseNet. Through iteration, the loss function is continuously converged and reduced, and the network parameters are updated until the number of iterations is reached. The loss function is selected The set of model parameters corresponding to the minimum value is used as the optimal implicit map model for the scenario;

[0132] Step 7: The mobile robot operates in any motion state in the scene, and the image I collected at any time with unknown posture t Perform preprocessing to generate an image of size 256×256 with pixel intensity between -1 and 1 And input it as a query image into the trained Re-PoseNet model with optimal weights;

[0133] The pre-processing process is as follows:

[0134] 1) Scale the image to a shorter side of 256 pixels, and then crop the image to a resolution of 256×256 by center cropping.

[0135] 2) The same method as normalizing the test image is used to make the pixel intensity of the test image between -1 and 1:

[0136]

[0137] Step 8: Get the query image through the feature extractor Feature map Then the optimized feature vector is obtained through the attention module and long short-term memory unit Finally, the pose regressor outputs the predicted position information With rotation information Obtain end-to-end pose estimation results.

[0138] The test results and comparative effects of the present invention are shown in Tables 2-5 and Figure 3 、 Figure 4 Here, three public datasets containing various scene types are used to test the present invention, and the median error between the predicted values ​​of all query images in a certain scene and the true values ​​provided by the dataset is used to evaluate the experimental results.

[0139] Table 2 shows the results of a comparative experiment based on the 7Scenes dataset, an indoor dataset captured by a handheld Kinect RGB-D camera in seven different indoor scenes, and the Kinect Fusion algorithm was used to generate the true value of the camera pose. This experiment used only RGB images when using this dataset. Each scene includes multiple sets of image sequences captured at different times using different trajectories and lighting conditions. The training set and query set are taken from different sequences and contain many ambiguous textureless features, which makes pure visual self-localization based on this dataset very challenging. As shown in the table, the method proposed in this paper has higher positioning accuracy in various indoor scenes.

[0140] Table 2 Error comparison between the proposed Re-PoseNet and existing algorithms on indoor datasets

[0141]

[0142] Table 3 shows the results of comparative experiments based on the Cambridge Landmarks dataset, which uses smartphones to capture images in several outdoor urban scenes. The corresponding ground-truth poses are calculated using existing 3D reconstruction methods. The training and query datasets are acquired from different acquisition paths, and these data contain many practical challenges, such as motion blur, varying illumination, varying weather conditions, and dynamic objects (such as pedestrians and vehicles). As shown in the table, the proposed method achieves the best algorithmic performance and robustness in environments with significant variations in conditions and interference from dynamic objects.

[0143] Table 3 Error comparison between the proposed Re-PoseNet and existing algorithms on outdoor datasets

[0144]

[0145] Table 4 shows the results of a comparative test based on the Oxford RobotCar dataset, which is a large outdoor dataset widely used in the field of autonomous driving. This dataset was compiled by Oxford University using the RobotCar car platform to record 1010.46 kilometers of driving in Oxford, England, and the data collection time spanned one year. It consists of more than 100 sequences, including RGB, lidar, GPS / INS and other data. The present invention selects some image sequences and divides them into training sets and query sets, and then names them LOOP1, LOOP2 and FULL according to different acquisition scenarios. These sequences have a large time span and a large scene scale, and there are unstable factors such as weather, lighting, season, pedestrians, and vehicles. As shown in the table, the method proposed in the present invention has obvious advantages in this challenging environment.

[0146] Table 4 Error comparison between the proposed Re-PoseNet and existing algorithms on large-scale dynamic outdoor datasets

[0147]

[0148] Table 5 and Figure 3 Ablation test results for the proposed network model Re-PoseNet on the indoor 7Scenes dataset and the outdoor Oxford RobotCar dataset. RN represents the model after removing the feature optimization module from Re-PoseNet, RN-AM represents the model with only the attention module in the feature optimization module, and RN-AM-LSTM represents the complete Re-PoseNet model. As shown in the table, the proposed feature optimization module can help improve model performance, enhance positioning accuracy, and improve robustness in challenging scenarios. Figure 3 This is the corresponding visualization result. The method proposed in this invention has obvious advantages.

[0149] Table 5 Ablation experiment results of Re-PoseNet on indoor and outdoor datasets

[0150]

[0151]

[0152] Figure 4 Shown are visualizations of image feature extraction using the proposed network model, Re-PoseNet, and the classic method, PoseNet. Sample images are taken from the 7Scenes indoor dataset and the Oxford RobotCar outdoor dataset. Highlighted locations indicate the network's increased focus on relevant areas. Compared to PoseNet, Re-PoseNet prioritizes static objects like buildings and cabinets, which remain stable over time, while ignoring dynamic objects like vehicles and pedestrians. This results in better robustness in dynamic environments.

[0153] According to the above experimental analysis, the algorithm of the present invention is superior to existing algorithms in terms of positioning accuracy and robustness, and is also applicable to various indoor and outdoor scenarios.

[0154] Reference 1: A.Kendall, M.Grimes, and R.Cipolla, "Posenet: Aconvolutional network for real-time 6-dof camera relocalization," in Proceedings of the IEEE international conference on computer vision, 2015, pp.2938-2946.

[0155] The unmanned system visual self-positioning method based on the end-to-end feature optimization model of the present invention can realize real-time self-positioning based on a single-frame image in a long-term dynamic environment. The method first constructs an image feature extractor based on a convolutional neural network to obtain learning features with good conditional invariance; then uses two cross-attention modules and a four-directional long-term and short-term memory module to construct a feature optimization module, redistributes weights of the learning features and reconstructs and reduces the dimensionality, so that the learning features pay more attention to static features and retain more geometric information related to posture, thereby improving the robustness of the learning features for dynamically changing scenes; secondly, homoscedastic uncertainty is introduced to jointly train the position loss and direction loss, which significantly improves the performance of the network model while improving the training efficiency; finally, the six-degree-of-freedom posture estimation result is directly output through the posture regressor, which has the advantages of high precision, good real-time performance, strong robustness, small storage space occupation, and applicability to various indoor and outdoor scenes.

Claims

1. A visual self-localization method for unmanned systems based on an end-to-end feature optimization model, characterized in that: The following steps are involved: Step 1: Construct the model Construct a deep neural network model for end-to-end global pose prediction, namely the Re-PoseNet network, or Re-PoseNet model; Step 2: Model training Train the Re-PoseNet model for a specific scenario; Step 3: Model Application Input the image collected at the specific scene into the Re-PoseNet model and return the end-to-end pose estimation result; The Re-PoseNet model includes a feature extractor, a feature optimization module, and a pose regressor. The feature extractor is based on the residual network ResNet34. The feature optimization module consists of two cross-attention modules and a four-directional long short-term memory module. The pose regressor consists of a fully connected layer. The training set used in step 2 is obtained as follows: A mobile robot equipped with a visual sensor randomly traverses a scene and collects a set of images and their global poses as a training dataset T to build an implicit map model of the scene. The global pose corresponding to each frame of the image is used as the true value label for subsequent network training, denoted as in Indicates location information. It is the rotation information represented by quaternion; The training process is: Step (1): For all images I in the training dataset T i Preprocessing is performed to generate training images of size 256×256 with pixel intensities between -1 and 1 Step (2): Input a set of training images and their pose labels into Re-PoseNet with batch size batch_size=64, obtain the predicted pose [p,q] through the pose regressor, and construct a joint loss function Calculate the predicted pose [p,q] and its corresponding true value Losses between Step (3): Set the initial learning rate of the hyperparameter to 5×10 -5 , the number of iterations is 1000, and the Adam optimizer is used to train the Re-PoseNet network. The loss function is continuously converged and reduced through iterations, and the network parameters are updated until the number of iterations is reached. The loss function is selected The set of model parameters corresponding to the minimum value is used as the optimal implicit map model for the scenario; In step 3, the mobile robot operates in the scene in any motion state, and the image I with unknown posture collected at any time is t Perform preprocessing to generate an image of size 256×256 with pixel intensity between -1 and 1 And input it as a query image into the trained Re-PoseNet model with optimal weights; Get the query image through the feature extractor Feature map Then the optimized feature vector is obtained through the attention module and long short-term memory unit Finally, the pose regressor outputs the predicted position information With rotation information Obtain end-to-end pose estimation results.

2. The unmanned system visual self-positioning method based on the end-to-end feature optimization model according to claim 1 is characterized in that: The specific process of building a deep neural network (Re-PoseNet) for end-to-end global pose prediction is as follows: Step 1.1: Design the specific structure of the feature extractor of Re-PoseNet as shown in Table 1, which has 33 convolutional layers and one maximum pooling layer; Taking convolution layer 1 as an example, its parameter information includes: convolution kernel size is 7×7, number of channels is 64, and stride is 2; the residual layer is composed of multiple residual units, each residual unit contains two convolution layers with convolution kernel size of 3×3 and stride of 1; for any image I input to the feature extractor, its corresponding feature map can be obtained Table 1 Network structure of Re-PoseNet feature extractor Step 1.2: Use two convolution layers with a convolution kernel of 1×1, a channel of 64, and a stride of 1 to perform convolution operations on the original feature map F, and generate feature maps with the same spatial size but different numbers. and At the same time, a convolution operation is performed on F using a convolution layer with a convolution kernel of 1×1, a channel of 512, and a stride of 1 to generate feature maps of the same size and number. Step 1.3: For any position u in the spatial dimension of the feature map F, the global context of the feature at position u is captured by calculating the correlation between the feature at that position and other features in the same row and column: in, is the vector corresponding to position u in X, is the set Ω u The i-th element in the set The elements in d are composed of vectors corresponding to the same row and column positions as position u in Y, so i = [1, 2, ..., 15], d i,u It is a scalar representing the correlation between features at different positions; Step 1.4: Traverse all i and repeat step 1.3 to calculate X u With the set Ω u The correlation d of all elements in i,u , get a set of weight coefficients corresponding to position u; Step 1.5: Traverse all positions u in the feature map F space and repeat steps 1.3 and 1.4 to get It saves the correlation degree corresponding to all position features in the feature map F, that is, d i,u ∈D, and then use the Softmax layer to process D in the channel dimension to obtain the attention map Step 1.6: Aggregate the attention map A with the feature map Z, and add the aggregation result to the original feature map F through the residual connection to achieve feature adaptation, adjust the network's attention to different feature information, and filter out redundant information to obtain a new feature map. For any position u in the spatial dimension, there is F′ u ∈F′: Among them, A i,u ∈A, is the set Φ u The i-th element in the set, i=[1,2,…,15], Take the vector at the same row and column as position u in the feature map Z, F u and F′ u Respectively represent the features before and after the attention operation at the spatial position u; Step 1.7: Repeat steps 1.2 to 1.6 for the feature map F′ to further capture dense context information and readjust the feature weights to obtain a new feature map Step 1.8: Use the average pooling layer to aggregate the information of the feature map F″ on each channel to obtain a feature with a dimension of 512×1×1, and output the feature vector through a fully connected layer with 2048 neurons Step 1.9: Reshape the eigenvector V′ into a 32×64 matrix. Then, process it using four LSTMs with 256 neurons, using the up, down, left, and right methods. Then, concatenate the four output vectors into a 1024-dimensional eigenvector V. Step 1.10: The pose regressor consists of two fully connected layers, containing 3 and 4 neurons respectively. The feature vector V is input into these two fully connected layers, and the predicted position information can be output. With rotation information

Images