Visual positioning method based on sparse Transform

Abstract

The invention belongs to the field of robot visual positioning, and particularly relates to a visual positioning method based on sparse Transform. The objective of the invention is to solve the problem of inaccurate prediction results in tasks such as scene coordinate prediction due to low accuracy of processing intensive regression tasks caused by high complexity of secondary calculation in Transformer in the prior art. The invention provides a feature aggregation network based on a global receptive field, through a sparse Transform mode, points with similar features obtain rich global information with a low calculation amount, and the problems that in the prior art, due to high secondary calculation complexity of Transform, the accuracy rate of processing dense regression tasks is low, and the processing efficiency is low are solved. According to the method, the problem that the prediction result is inaccurate in tasks such as scene coordinate prediction and the like is solved, and the indoor service robot can be started and restarted for self-positioning in a severe indoor scene with illumination variation, low texture, no texture or repeated texture.

Description

Technical Field

[0001] The invention belongs to the field of robot visual positioning, and in particular relates to a visual positioning method based on a sparse Transformer. Background Art

[0002] The present invention is aimed at indoor / outdoor service robots and studies a relocalization technology based on a scene coordinate regression method, aiming to realize a robust and reliable relocalization algorithm.

[0003] With the development of deep learning technology, researchers have tried to apply deep learning technology to the field of visual relocalization and established three mainstream algorithm frameworks:

[0004] That is, the absolute pose regression method, the relative pose regression method, and the scene coordinate regression method.

[0005] Absolute pose regression methods (such as PoseNet, MS-Transformer) use neural networks to directly establish a mapping function between the image and the camera's 6-DOF pose in an end-to-end manner, thereby directly predicting the camera's 6-DOF pose in an end-to-end manner.

[0006] Although highly efficient positioning is achieved, a large number of studies have shown that the absolute pose regression method is essentially similar to the image retrieval method and therefore cannot achieve accurate visual positioning.

[0007] Relative pose regression methods (such as CamNet and DistillPose) first use image retrieval to extract the image most similar to the query image in the database, and then use a neural network to predict the relative pose between the query image and the retrieved image.

[0008] The experimental results show that the relative pose regression method has a certain generalization ability and can achieve a certain positioning accuracy when applied in unknown scenes. However, the relative pose regression method cannot achieve the most accurate positioning effect.

[0009] The scene coordinate regression method (such as KFNet, HSCNet) predicts the 3D coordinates of the landmark points corresponding to the image pixels through random forests or convolutional neural networks, thereby directly establishing the correspondence between 2D pixels and 3D landmark points. Subsequently, the PnP algorithm of RANSAC is used to solve the camera pose.

[0010] In the early stage of research, the non-differentiable nature of the RANSAC algorithm made it impossible to supervise the network using the pose as the true value. To solve this problem,

[0011] DSAC proposed two alternative methods to make RANSAC differentiable, softmax-based RANSAC (differentiable) and DSAC (better effect), realizing an end-to-end trainable camera localization algorithm.

[0012] DSAC uses a convolutional neural network (CNN) to predict the three-dimensional coordinates of an image, obtains a set of model hypotheses from a subset of coordinates through the PNP algorithm, scores the hypothesized models through a scoring network (CNN) based on a scalar function of the reprojection error, selects the hypothesized model based on the idea of ​​probability distribution, and uses finite differences to refine the pose.

[0013] DSAC achieved the highest accuracy of camera localization results at that time, but there was still a problem that the CNN module that scored the model hypothesis was prone to overfitting.

[0014] DSAC++ no longer uses CNN to score the predicted poses, but instead uses the reprojection error to score each pose and select the optimal pose accordingly.

[0015] Specifically, DSAC++ predicts N scene coordinates and constructs T scene coordinate sets, each containing 4 scene coordinates. Based on this, DSAC++ predicts T predicted poses and scores each pose using the reprojection error, selecting the pose with the highest score as the initial pose. Finally, the initial pose is optimized by minimizing the repositioning error to obtain the final pose prediction result.

[0016] DSAC* is an extension of DASC++. It can use RGB images, RGBD images, and RGB images + 3D models as input, which simplifies the training process and achieves higher accuracy.

[0017] In addition, KFNet first introduces temporal information into the scene coordinate regression network, obtains the maximum likelihood prediction of a single image coordinate scene through SCoordNet, uses OFlowNet to generate a prior prediction of the scene coordinates, and finally fuses the two predictions to generate the maximum a posteriori estimate of the final scene coordinates.

[0018] The design of the KFNet network structure and the loss function are based on Kalman filtering, and achieve 2D-3D temporal consistency matching. Considering that there are many pixels in the image (such as points in the sky) that cannot predict the scene coordinates, VS-Net first divides the image into patches and regards the center points of the patches as markers. Subsequently, VS-Net regards the scene coordinate regression problem as a combination of the segmentation problem and the offset prediction.

[0019] Specifically, VS-Net predicts the patch category and 2D offset of each pixel point relative to the marker point. Finally, the 2D pixel coordinates of each marker point are obtained through a voting mechanism, thereby establishing a correspondence between 2D pixel coordinates and 3D scene coordinates, and predicting the camera pose accordingly.

[0020] At present, the scene coordinate regression method based on deep learning has achieved extremely high accuracy in various data sets due to its powerful scene understanding ability. However, this method usually uses a convolutional neural network with a small receptive field to build the model, which leads to mutual interference between the features extracted from similar image blocks, making the network's understanding of the scene confusing.

[0021] Recently, Transformer has shown its superior ability in integrating global features, which provides a solution to the above problem. With the help of global receptive field, Transformer can effectively resolve the ambiguity caused by visual similarity between image patches. Although Transformer alleviates the defects of CNN,

[0022] However, the Transformer in the existing technology has a high secondary calculation complexity, which leads to low accuracy in processing intensive regression tasks, resulting in inaccurate prediction results in tasks such as scene coordinate prediction. Summary of the invention

[0023] The purpose of the present invention is to solve the problem that the Transformer in the prior art has a high secondary computational complexity, resulting in low accuracy in processing intensive regression tasks, resulting in inaccurate prediction results in tasks such as scene coordinate prediction. We propose a visual positioning method based on sparse Transformer, including:

[0024] S1: Build a SACNet network; obtain a trained SACNet network;

[0025] S2: Input the camera image to be identified into the trained SACNet network for visual positioning, and output the predicted scene coordinates of the camera image to be identified;

[0026] S3: Use the PnP algorithm to process the predicted scene coordinates of the camera image to be identified obtained in S2 to obtain the camera pose of the camera image to be identified.

[0027] The beneficial effects of the present invention are:

[0028] The present invention proposes a feature aggregation network based on the global receptive field. Through the sparse Transformer method, points with similar features can obtain rich global information with a lower amount of calculation, thereby improving the accuracy of intensive regression tasks and solving the problem of inaccurate prediction results in tasks such as scene coordinate prediction. Furthermore, the present invention can realize the self-positioning problem of indoor service robots when they are powered on and restarted in harsh indoor scenes with changing lighting and low, no or repeated textures. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 It is a schematic diagram of the overall framework of the SACNet network of the present invention;

[0030] Figure 2 is a flow chart of the network structure of the multi-level sparse attention layer of the present invention;

[0031] Figure 3 It is a specific structural diagram of the filter feedforward network of the present invention. DETAILED DESCRIPTION

[0032] Specific implementation method 1: Combination Figure 1 The invention is described, comprising:

[0033] S1: Build a SACNet network; obtain a trained SACNet network;

[0034] S2: Input the camera image to be identified into the trained SACNet network for visual positioning, and output the predicted scene coordinates of the camera image to be identified;

[0035] S3: Use the PnP algorithm to process the predicted scene coordinates of the camera image to be identified obtained in S2 to obtain the camera pose of the camera image to be identified.

[0036] Specific implementation method 2: The difference between this implementation method and specific implementation method 1 is that:

[0037] In S1, a SACNet network is constructed; a trained SACNet network is obtained; the specific process is as follows:

[0038] S1.1: Obtain historical camera images as training set;

[0039] S1.2: Build a SACNet network; obtain a trained SACNet network.

[0040] The present invention uses a robot platform equipped with a RealSense D435 camera and a VLP-16 laser radar to collect data sets. Specifically, this paper uses the ROS system to collect aligned RGB images and depth maps by subscribing to the / camera / color / image raw and / camera / aligned depth to color / image raw topics, and the corresponding timestamp is t1. Subsequently, this paper collects point cloud data by subscribing to the / velodyne topic provided by the VLP-16 laser radar, and the corresponding timestamp is t2. Subsequently, this paper generates the final RGB-D image and the corresponding point cloud by aligning the timestamps t1 and t2. Finally, this paper uses the radar-based SLAM system A-LOAM to obtain the corresponding pose truth value. The other steps and parameters are the same as those in the first specific implementation method.

[0041] Specific implementation method three: This implementation method is different from the specific implementation method one in that:

[0042] In S1.2, a SACNet network is constructed, wherein the SACNet network includes: a CNN-based backbone network, a multi-layer sparse Transformer network, a feature regulator network, and a regression layer network;

[0043] The trained SACNet network is obtained in S1.2, and the specific process is as follows:

[0044] S1.2.1: Input the images in the training set into the CNN-based backbone network for feature extraction and obtain the first feature F l ;

[0045] S1.2.2: The first feature F obtained in S1.2.1 l Input the multi-layer sparse Transformer network for processing to obtain the second feature F g ;

[0046] S1.2.3: The second feature F obtained in S1.2.2 g With the first feature F l Add, and input the result of the addition into the feature regulator network for regulation to obtain the third feature F f ;

[0047] S1.2.4: The third feature F obtained in S1.2.3 f Input the regression layer network for regression prediction processing to obtain the output result;

[0048] S1.2.5: Calculate the loss function based on the input and output of the SACNet network to obtain the trained SACNet network.

[0049] The present invention proposes a feature aggregation network based on global receptive field, which enables points with similar features to obtain rich global information with lower computational complexity through sparse Transformer, thereby improving the accuracy of dense regression tasks and solving the problem of inaccurate prediction results in tasks such as scene coordinate prediction. The other steps and parameters are the same as those in the first to second embodiments.

[0050] Specific implementation method 4: This implementation method is different from specific implementation methods 1 to 4 in that:

[0051] In S1.2.1, the image in the training set is input into the CNN-based backbone network for feature extraction processing to obtain the first feature F l The specific process is:

[0052] The CNN-based backbone network includes: an initial convolutional layer, a first residual block, a second residual block, a third residual block, and a fourth residual block;

[0053] The images in the training set are processed by the initial convolution layer, the first residual block, the second residual block, the third residual block and the fourth residual block in sequence for feature extraction to obtain the first feature F l ;

[0054] The initial convolution layer includes a convolution layer with a convolution kernel of 7×7 and a step size of 2, a BN layer, and a ReLU activation function layer in sequence;

[0055] The first residual block is composed of two residual units 1; the residual unit 1 includes: a convolution layer 1, a batch normalization layer, a ReLU activation function layer, a convolution layer 1, a batch normalization layer and a ReLU activation function layer in sequence; the convolution layer 1 is a convolution layer with a convolution kernel of 3×3 and a stride of 1;

[0056] The second residual block is composed of two residual units 2; the residual unit 2 includes: a convolution layer 2, a batch normalization layer, a ReLU activation function layer, a convolution layer 2, a batch normalization layer and a ReLU activation function layer in sequence; the convolution layer 2 is a convolution layer with a convolution kernel of 3×3 and a stride of 1;

[0057] The third residual block is composed of two residual units 3; the residual unit 3 includes: a convolution layer 3, a batch normalization layer, a ReLU activation function layer, a convolution layer 3, a batch normalization layer and a ReLU activation function layer in sequence; the convolution layer 3 is a convolution layer with a convolution kernel of 3×3 and a stride of 2;

[0058] The fourth residual block is composed of two residual units 4; the residual unit 4 includes in sequence: a convolution layer 4, a batch normalization layer, a ReLU activation function layer, a convolution layer 4, a batch normalization layer and a ReLU activation function layer; the convolution layer 4 is a convolution layer with a convolution kernel of 3×3 and a step size of 2.

[0059] This makes the final output feature map F l The dimension is R C×H / 8×W / 8

[0060] The other steps and parameters are the same as those in the first to third embodiments.

[0061] Specific implementation method 5: This implementation method is different from specific implementation methods 1 to 4 in that:

[0062] In S1.2.2, the first feature F obtained in S1.2.1 is l Input the multi-layer sparse Transformer network for processing to obtain the second feature F g The specific process is:

[0063] The multi-layer sparse Transformer network includes: LN layer, multi-layer sparse attention layer, LN layer, and filter feedforward network;

[0064] S1.2.2.1: The first feature F l Input an LN layer for processing, and then input the processing result into a multi-level sparse attention layer for processing to obtain a feature sequence M;

[0065] S1.2.2.2: Combine the feature sequence M obtained in S1.2.2.1 with the first feature F l Add together to get the feature sequence M F ;

[0066] S1.2.2.3: The feature sequence M obtained in S1.2.2.2 F Input an LN layer for processing, and the LN layer output features are input into the filtered feedforward network for processing to obtain the global feature F2;

[0067] S1.2.2.4: Combine the global feature F2 obtained in S1.2.2.3 with the feature sequence M F Add together to get the second feature F g ,This network can significantly reduce the computation of the attention mechanism and alleviate the problem of redundant information transmission.,The other steps and parameters are the same as those in the first to fourth embodiments.

[0068] Specific implementation method 6: This implementation method is different from specific implementation methods 1 to 5 in that:

[0069] The first feature F in S1.2.2.1 l Input an LN layer for processing, and then input the processing result into the multi-level sparse attention layer (MSA Layer) for processing. The specific process of obtaining the feature sequence M is as follows:

[0070] S1.2.2.1.1: The first feature F l Input an LN layer to process and obtain the intermediate feature LN (F l );

[0071] In the formula, LN(·) represents normalization processing,

[0072] S1.2.2.1.2: The intermediate feature LN(F l ) Through 1×1 convolution processing, the intermediate feature F is obtained m , and then the intermediate feature F m Input three independent depth-separable convolutional layers with different kernel sizes respectively to obtain three local features. Finally, the three local features are integrated together and input into a 1×1 convolutional layer to obtain the intermediate features. The formula is:

[0073] F m =C(LN(F l ))

[0074]

[0075] Where C(·) refers to a 1×1 convolution operation; D3(·) represents a depth-wise separable convolution operation with a convolution kernel size of 3×3, D5(·) represents a depth-wise separable convolution operation with a convolution kernel size of 5×5, and D7(·) represents a depth-wise separable convolution operation with a convolution kernel size of 7×7. refers to the l2 regularization operation of the spatial dimension, and [·||·||·] indicates integrated processing;

[0076] S1.2.2.1.3: Intermediate features It is divided into three parts, and then the feature graph is converted into a feature sequence operation to obtain the query vector Q, key vector K and value vector V. The formula is expressed as:

[0077]

[0078]

[0079]

[0080] In the formula, I2S(·) refers to the operation of converting the feature map into a feature sequence. Where N = H / 8 × W / 8, Represents a tensor, N represents the number, H represents the image height, W represents the image width, and C represents the number of channels;

[0081] S1.2.2.1.4: Normalize the query vector Q and the key vector K to obtain a standardized query vector and the normalized key vector The formula is:

[0082]

[0083]

[0084] S1.2.2.1.5: According to the standardized query vector and the normalized key vector Calculate the initial attention matrix The specific process is:

[0085]

[0086] In the formula, represents the normalized key vector The transpose of

[0087] S1.2.2.1.6: According to the initial attention matrix M c Calculate the sparse attention matrix M 1 / 2 , sparse attention matrix M 1 / 3 and the sparse attention matrix M 1 / 4 , the specific process is:

[0088] S1.2.2.1.6.1: For the initial attention matrix M c Sort the values ​​of each row in ascending order to get the sorting result;

[0089] S1.2.2.1.6.2: According to the sorting results, the initial attention matrix M c The low values ​​of 1 / 2 in each row are replaced by 0, and the sparse attention matrix M is obtained. 1 / 2 ; (For example, the values ​​of a row of the attention matrix are 12345678, and then 1 / 2 of the lowest value is replaced by 0, the result is 00005678. Replace 1 / 4 of the lowest value with 0, the result is 00345678)

[0090] S1.2.2.1.6.3: According to the sorting results, the initial attention matrix M c Replace 1 / 3 of the low values ​​in each row with 0 to obtain a sparse attention matrix M 1 / 3 ;

[0091] S1.2.2.1.6.4: According to the sorting results, the initial attention matrix M c Replace 1 / 4 of the low values ​​in each row with 0 to obtain a sparse attention matrix M 1 / 4 ;

[0092] S1.2.2.1.7: According to the sparse attention matrix M 1 / 2 , sparse attention matrix M 1 / 3 and the sparse attention matrix M 1 / 4 Calculate the feature sequence M. The specific process is:

[0093] M=C(α1ME 1 / 2 +α2ME 1 / 3 +α3ME 1 / 4 ),

[0094] Among them, α1, α2 and α3 represent learnable weight factors, C(·) represents 1×1 convolution, ME 1 / 2 is 1 / 2 feature representation, ME 1 / 3 1 / 3 feature representation, ME 1 / 4 1 / 3 feature representation

[0095]

[0096]

[0097]

[0098] Among them, Softmax(·) represents the normalization function.

[0099] Although using the attention matrix to retrieve information from the value vector can provide valuable global information and promote feature aggregation, the resulting propagation of redundant information will also affect the positioning accuracy. In particular, feature sequences located in textureless areas (white walls, floors) convey almost no effective feature information and have low similarity weights in the attention matrix. In order to enable the network to adaptively handle scenes with various appearances, the multi-level sparse attention layer constructs three sparse attention matrices to adaptively select different areas of the input attention matrix while minimizing the interference of redundant information; the other steps and parameters are the same as those of the specific implementation methods one to five.

[0100] Specific implementation method 7: This implementation method is different from specific implementation methods 1 to 6 in that:

[0101] In S1.2.2.3, the characteristic sequence M obtained in S1.2.2.2 is F Input an LN layer for processing, and the LN layer output features are input into the filter feedforward network for processing to obtain the global feature F2; the specific process is:

[0102] S1.2.2.3.1: The obtained feature sequence M F Input an LN layer for processing and obtain the intermediate sequence LN(M F );

[0103] S1.2.2.3.2: The intermediate sequence LN(M F ) is sequentially input into the LN layer and the fully connected layer, and then the feature sequence is converted into a feature map to obtain the intermediate feature map M m , the formula is:

[0104] M m =S2I(MLP(LN(LN(M F )))),

[0105] Where S2I(·) refers to the operation of converting feature sequence into feature map, and MLP(·) represents the fully connected layer;

[0106] S1.2.2.3.3: The intermediate feature map M is obtained m Input a 3×3 depth-separable convolutional layer to obtain the intermediate feature map The formula is:

[0107]

[0108] The intermediate feature map Divide into intermediate feature maps through the slice function and the intermediate feature map And according to the intermediate feature map and the intermediate feature map Get the intermediate feature map M 1 , the formula is:

[0109]

[0110] Among them, Φ(·) refers to the GELU nonlinear activation function, and ⊙ refers to the element-by-element product operation;

[0111] S1.2.2.3.4: The intermediate feature map M is obtained m Input a 5×5 depth-separable convolutional layer to obtain the intermediate feature map The formula is:

[0112]

[0113] The intermediate feature map Divide into intermediate feature maps through the slice function and the intermediate feature map And according to the intermediate feature map and the intermediate feature map Get the intermediate feature map M 2 , the formula is:

[0114]

[0115] S1.2.2.3.5: The intermediate feature map M 1 and the intermediate feature map M 2 Concatenate along the channel dimension, and then input the concatenated result into a fully connected layer to obtain the second feature F g ,

[0116] The other steps and parameters are the same as those in Specific Embodiments 1 to 6.

[0117] Specific implementation eight: This implementation differs from specific implementations one to seven in that:

[0118] In S1.2.3, the second feature F obtained in S1.2.2 is g With the first feature F l Add, and input the result of the addition into the feature regulator network for adjustment processing to obtain the third feature F f , the specific process is:

[0119] S1.2.3.1: The second feature F obtained in S1.2.2 g With the first feature F l Add them together to get the fusion features;

[0120] S1.2.3.2: Input the fused features into the local mask extractor to obtain the local soft attention mask F ln ,

[0121] The local mask extractor includes, in sequence: a 1×1 convolution layer, a BN layer, a RELU activation function, a 1×1 convolution and a BN layer;

[0122] S1.2.3.3: Input the fused features into the global mask extractor to obtain the global soft attention mask F gn ,

[0123] The global mask extractor includes, in sequence: a global average pooling layer, a 1×1 convolution layer, a BN layer, a RELU activation function, a 1×1 convolution layer, and a BN layer;

[0124] S1.2.3.4: The local soft attention mask F ln and the global soft attention mask F gnAdd them together and input the result of the addition into the sigmoid activation function to get F g The soft attention mask W g ;

[0125] S1.2.3.5: According to the F g The soft attention mask W g Get F l The soft attention mask W l , the formula is:

[0126] W l =1-W g

[0127] S1.2.3.6: According to the F g The soft attention mask W g and the F l The soft attention mask W l Get the third feature F f , The formula is:

[0128] F f =W l ⊙F l +W g ⊙F g .

[0129] Studies have shown that the inductive bias of convolutional neural networks is crucial for the network to model scene information. In contrast, Transformer has difficulty in acquiring local geometric features. How to fuse local and global features needs further study. To solve this problem, we propose a feature regulator (FC), which can adaptively integrate local and global features to enhance the overall representation ability of the network. The other steps and parameters are the same as those of Specific Implementation Methods 1 to 7.

[0130] Specific implementation method 9: This implementation method is different from specific implementation methods 1 to 8 in that:

[0131] In S1.2.4, the third feature F obtained in S1.2.3 is f Input the regression layer network for regression prediction processing to obtain the prediction result; the specific process is:

[0132] S1.2.4.1: Obtained third feature F f After passing through the first convolution layer, the second convolution layer, the third convolution layer, and the fourth convolution layer in sequence, the intermediate prediction variables are obtained;

[0133] The first convolution layer, the second convolution layer, the third convolution layer, and the fourth convolution layer all include, in sequence: a convolution layer with a convolution kernel of 3x3 and a step size of 1, a BN layer, and a ReLU activation function layer;

[0134] S1.2.4.2: The intermediate prediction variables are passed through the fifth and sixth convolutional layers respectively to obtain N predicted scene coordinates C = {c i =(x i ,y i ,z i )|i=1,2,...,N} and N one-dimensional uncertainties U={u i |i=1,2,...,N}; i represents the number; each predicted coordinate c i Corresponding to a one-dimensional uncertainty u i ;

[0135] The fifth convolutional layer and the sixth convolutional layer each include, in sequence: a convolutional layer with a convolution kernel of 3x3 and a step size of 1, a BN layer, a ReLU activation function layer, and a convolutional layer with a convolution kernel of 3x3 and a step size of 1;

[0136] S1.2.4.3: Discard the predicted scene coordinates corresponding to the uncertainty greater than β, where β is a positive integer, and use the remaining predicted scene coordinates as the prediction results.

[0137] The other steps and parameters are the same as those in Specific Embodiments 1 to 8.

[0138] Specific implementation method 10: This implementation method is different from specific implementation methods 1 to 9 in that:

[0139] The loss function for training the SACNet network model in S1.2.5 is:

[0140]

[0141] Where: Represents the real scene coordinates;

[0142] The other steps and parameters are the same as those in Specific Implementations 1 to 9.

[0143] The above only describes the preferred embodiments of the present invention. It should be understood that the present invention is not limited to the above specific implementation methods. Although the present invention has been disclosed as above in the preferred embodiments, it is not used to limit the present invention. Any technician familiar with the profession can make some changes or modify the technical contents disclosed above into equivalent embodiments with equivalent changes without departing from the scope of the technical solution of the present invention. However, any simple modification, equivalent replacement and improvement made to the above embodiments without departing from the content of the technical solution of the present invention, based on the technical essence of the present invention, within the spirit and principles of the present invention, still fall within the protection scope of the technical solution of the present invention.

Claims

1. A visual positioning method based on sparse Transiormer, characterized in that: The following steps are involved: S1: Build a SACNet network; obtain a trained SACNet network; S2: Input the camera image to be identified into the trained SACNet network for visual positioning, and output the predicted scene coordinates of the camera image to be identified; S3: Use the PnP algorithm to process the predicted scene coordinates of the camera image to be identified obtained in S2 to obtain the camera pose of the camera image to be identified.

2. According to claim 1, a visual positioning method based on sparse Transformer is characterized in that: In S1, a SACNet network is constructed; a trained SACNet network is obtained; the specific process is as follows: S1.1: Obtain historical camera images as training set; S1.2: Build a SACNet network; obtain a trained SACNet network.

3. According to claim 2, a visual positioning method based on sparse Transformer is characterized in that: In S1.2, a SACNet network is constructed, wherein the SACNet network includes: a CNN-based backbone network, a multi-layer sparse Transformer network, a feature regulator network, and a regression layer network; The trained SACNet network is obtained in S1.2, and the specific process is as follows: S1.2.1: Input the images in the training set into the CNN-based backbone network for feature extraction and obtain the first feature F l ; S1.2.2: The first feature F obtained in S1.2.1 l Input the multi-layer sparse Transformer network for processing to obtain the second feature F g ; S1.2.3: The second feature F obtained in S1.2.2 g Add it to the first feature F1, and input the result of the addition into the feature regulator network for regulation to obtain the third feature F f ; S1.2.4: The third feature F obtained in S1.2.3 f Input the regression layer network for regression prediction processing to obtain the output result; S1.2.5: Calculate the loss function based on the input and output of the SACNet network to obtain the trained SACNet network.

4. The visual positioning method based on sparse Transformer according to claim 3, characterized in that: In S1.2.1, the image in the training set is input into the CNN-based backbone network for feature extraction processing to obtain the first feature F l The specific process is: The CNN-based backbone network includes: an initial convolutional layer, a first residual block, a second residual block, a third residual block, and a fourth residual block; The images in the training set are processed by the initial convolution layer, the first residual block, the second residual block, the third residual block and the fourth residual block in sequence for feature extraction to obtain the first feature F l ; The initial convolution layer includes a convolution layer with a convolution kernel of 7×7 and a step size of 2, a BN layer, and a ReLU activation function layer in sequence; The first residual block is composed of two residual units 1; the residual unit 1 includes: a convolution layer 1, a batch normalization layer, a ReLU activation function layer, a convolution layer 1, a batch normalization layer and a ReLU activation function layer in sequence; the convolution layer 1 is a convolution layer with a convolution kernel of 3×3 and a stride of 1; The second residual block is composed of two residual units 2; the residual unit 2 includes: a convolution layer 2, a batch normalization layer, a ReLU activation function layer, a convolution layer 2, a batch normalization layer and a ReLU activation function layer in sequence; the convolution layer 2 is a convolution layer with a convolution kernel of 3×3 and a stride of 1; The third residual block is composed of two residual units 3; the residual unit 3 includes: a convolution layer 3, a batch normalization layer, a ReLU activation function layer, a convolution layer 3, a batch normalization layer and a ReLU activation function layer in sequence; the convolution layer 3 is a convolution layer with a convolution kernel of 3×3 and a step size of 2; The fourth residual block is composed of two residual units 4; the residual unit 4 includes in sequence: a convolution layer 4, a batch normalization layer, a ReLU activation function layer, a convolution layer 4, a batch normalization layer and a ReLU activation function layer; the convolution layer 4 is a convolution layer with a convolution kernel of 3×3 and a step size of 2.

5. The visual positioning method based on sparse Transformer according to claim 4, characterized in that: In S1.2.2, the first feature F1 obtained in S1.2.1 is input into a multi-level sparse Transformer network for processing to obtain a second feature F g The specific process is: The multi-layer sparse Transformer network includes: LN layer, multi-layer sparse attention layer, LN layer, and filter feedforward network; S1.2.2.1: The first feature F l Input an LN layer for processing, and then input the processing result into a multi-level sparse attention layer for processing to obtain a feature sequence M; S1.2.2.2: Combine the feature sequence M obtained in S1.2.2.1 with the first feature F l Add together to get the feature sequence M F ; S1.2.2.3: The feature sequence M obtained in S1.2.2.2 F Input an LN layer for processing, and the LN layer output features are input into the filtered feedforward network for processing to obtain the global feature F2; S1.2.2.4: Combine the global feature F2 obtained in S1.2.2.3 with the feature sequence M F Add together to get the second feature F g .

6. The visual positioning method based on sparse Transformer according to claim 5, characterized in that: The first feature F in S1.2.2.1 l Input an LN layer for processing, and then input the processing result into a multi-level sparse attention layer for processing. The specific process of obtaining the feature sequence M is as follows: S1.2.2.1.1: The first feature F l Input an LN layer to process and obtain the intermediate feature LN (F l ); In the formula, LN(·) represents normalization processing, S1.2.2.1.2: The intermediate feature LN(F l ) Through 1×1 convolution processing, the intermediate feature F is obtained m , and then the intermediate feature F m Input three independent depth-separable convolutional layers with different convolution kernel sizes respectively to obtain three local features. Finally, the three local features are integrated together and input into a 1×1 convolutional layer to obtain the intermediate features. The formula is: F m =C(LN(F l )) Where C(·) refers to a 1×1 convolution operation; D3(·) represents a depth-wise separable convolution operation with a convolution kernel size of 3×3, D5(·) represents a depth-wise separable convolution operation with a convolution kernel size of 5×5, and D7(·) represents a depth-wise separable convolution operation with a convolution kernel size of 7×7. refers to the l2 regularization operation of the spatial dimension, and [·||·||·] indicates integrated processing; S1.2.2.1.3: Intermediate features It is divided into three parts, and then the feature graph is converted into a feature sequence operation to obtain the query vector Q, key vector K and value vector V. The formula is expressed as: In the formula, I2S(·) refers to the operation of converting the feature map into a feature sequence. Where N = H / 8 × W / 8, Represents a tensor, N represents the number, H represents the image height, W represents the image width, and C represents the number of channels; S1.2.2.1.4: Normalize the query vector Q to obtain a standardized query vector Normalize the key vector K to get the standardized key vector And normalize the key vector Transpose to get the normalized key vector after transposition The formula is: S1.2.2.1.5: According to the standardized query vector and the transposed normalized key vector Calculate the initial attention matrix The specific process is: S1.2.2.1.6: According to the initial attention matrix M c Calculate the sparse attention matrix M 1 / 2 , sparse attention matrix M 1 / 3 and the sparse attention matrix M 1 / 4 , the specific process is: S1.2.2.1.6.1: For the initial attention matrix M c Sort the values ​​of each row in ascending order to get the sorting result; S1.2.2.1.6.2: According to the sorting results, the initial attention matrix M c The low values ​​of 1 / 2 in each row are replaced by 0, and the sparse attention matrix M is obtained. 1 / 2 ; S1.2.2.1.6.3: According to the sorting results, the initial attention matrix M c Replace 1 / 3 of the low values ​​in each row with 0 to obtain a sparse attention matrix M 1 / 3 ; S1.2.2.1.6.4: According to the sorting results, the initial attention matrix M c Replace 1 / 4 of the low values ​​in each row with 0 to obtain a sparse attention matrix M 1 / 4 ; S1.2.2.1.7: According to the sparse attention matrix M 1 / 2 , sparse attention matrix M 1 / 3 and the sparse attention matrix M 1 / 4 Calculate the feature sequence M. The specific process is: M=C(α1ME 1 / 2 +α2ME 1 / 3 +α3ME 1 / 4 ), Among them, α1, α2 and α3 represent learnable weight factors, C(·) represents 1×1 convolution, ME 1 / 2 is 1 / 2 feature representation, ME 1 / 3 1 / 3 feature representation, ME 1 / 4 1 / 3 feature representation Among them, Softmax(·) represents the normalization function.

7. The visual positioning method based on sparse Transformer according to claim 6, characterized in that: In S1.2.2.3, the characteristic sequence M obtained in S1.2.2.2 is F Input an LN layer for processing, and the LN layer output features are input into the filter feedforward network for processing to obtain the global feature F2; the specific process is: S1.2.2.3.1: The obtained feature sequence M F Input an LN layer for processing and obtain the intermediate sequence LN(M F ); S1.2.2.3.2: The intermediate sequence LN(M F ) is sequentially input into the LN layer and the fully connected layer, and then the feature sequence is converted into a feature map to obtain the intermediate feature map M m , the formula is: M m =S2I(MLP(LN(LN(M F )))), Where S2I(·) refers to the operation of converting feature sequence into feature map, and MLP(·) represents the fully connected layer; S1.2.2.3.3: The intermediate feature map M is obtained m Input a 3×3 depth-separable convolutional layer to obtain the intermediate feature map The formula is: The intermediate feature map Divide into intermediate feature maps through the slice function and the intermediate feature map And according to the intermediate feature map and the intermediate feature map Get the intermediate feature map M 1 , the formula is: Among them, Φ(·) refers to the GELU nonlinear activation function, and ⊙ refers to the element-by-element product operation; S1.2.2.3.4: The intermediate feature map M is obtained m Input a 5×5 depth-separable convolutional layer to obtain the intermediate feature map The formula is: The intermediate feature map Divide into intermediate feature maps through the slice function and the intermediate feature map And according to the intermediate feature map and the intermediate feature map Get the intermediate feature map M 2 , the formula is: S1.2.2.3.5: The intermediate feature map M 1 and the intermediate feature map M 2 Concatenate along the channel dimension, and then input the concatenated result into a fully connected layer to obtain the second feature F g , 8. The visual positioning method based on sparse Transformer according to claim 7, characterized in that: In S1.2.3, the second feature F obtained in S1.2.2 is g With the first feature F l Add, and input the result of the addition into the feature regulator network for adjustment processing to obtain the third feature F f , the specific process is: S1.2.3.1: The second feature F obtained in S1.2.2 g With the first feature F l Add them together to get the fusion features; S1.2.3.2: Input the fused features into the local mask extractor to obtain the local soft attention mask F ln , The local mask extractor includes, in sequence: a 1×1 convolution layer, a BN layer, a RELU activation function, a 1×1 convolution and a BN layer; S1.2.3.3: Input the fused features into the global mask extractor to obtain the global soft attention mask F gn , The global mask extractor includes, in sequence: a global average pooling layer, a 1×1 convolution layer, a BN layer, a RELU activation function, a 1×1 convolution layer, and a BN layer; S1.2.3.4: The local soft attention mask F ln and the global soft attention mask F gn Add them together and input the result of the addition into the sigmoid activation function to get F g The soft attention mask W g ; S1.2.3.5: According to the F g The soft attention mask W g Get F l The soft attention mask W l , the formula is: IN l =1-W g S1.2.3.6: According to the F g The soft attention mask W g and the F l The soft attention mask W l Get the third feature F f , The formula is: F f =W l ⊙F l +W g ⊙F g 。 9. The visual positioning method based on sparse Transformer according to claim 8, characterized in that: In S1.2.4, the third feature F obtained in S1.2.3 is f Input the regression layer network for regression prediction processing to obtain the output result; the specific process is: S1.2.4.1: Obtained third feature F f After passing through the first convolution layer, the second convolution layer, the third convolution layer, and the fourth convolution layer in sequence, the intermediate prediction variables are obtained; The first convolution layer, the second convolution layer, the third convolution layer, and the fourth convolution layer all include, in sequence: a convolution layer with a convolution kernel of 3x3 and a step size of 1, a BN layer, and a ReLU activation function layer; S1.2.4.2: The intermediate prediction variables are passed through the fifth and sixth convolutional layers respectively to obtain N predicted scene coordinates C = {c i =(x i ,y i , z i )|i=1,2,...,N} and N one-dimensional uncertainties U={u i |i=1,2,...,N};i represents the number;each predicted coordinate c i Corresponding to a one-dimensional uncertainty u i ; The fifth convolutional layer and the sixth convolutional layer each include, in sequence: a convolutional layer with a convolution kernel of 3×3 and a step size of 1, a BN layer, a ReLU activation function layer, and a convolutional layer with a convolution kernel of 3×3 and a step size of 1; S1.2.4.3: Discard the predicted scene coordinates corresponding to the uncertainty greater than β, where β is a positive integer, and use the remaining predicted scene coordinates as the output result.

10. The visual positioning method based on sparse Transformer according to claim 9, characterized in that: The loss function in S1.2.5 is: Where: Represents the real scene coordinates.

Images