Visual positioning methods across multiple indoor scenes

Abstract

This invention presents a visual positioning method applicable to multiple indoor scenes, overcoming the limitation of scene coordinate regression methods for visual positioning only in specific scenarios. It falls under the field of video image capture visual positioning. The invention includes: collecting data in N different scenes and preprocessing the collected data; using RGB images and depth maps from the scenes as input and the scene coordinates corresponding to each pixel in the depth map as output to build a cross-scene visual positioning network; and employing an adaptive parameter sharing strategy when training the cross-scene visual positioning network to determine the convolutional layer weights: selecting the appropriate weights for the cross-scene visual positioning network based on the RGB image to be tested and the corresponding depth image; then using the trained cross-scene visual positioning network to predict the RGB image to be tested, obtaining the scene coordinates of each pixel; and calculating the camera pose based on the obtained scene coordinates and the pixel coordinates in the corresponding depth image.

Description

Technical Field

[0001] This invention relates to a visual positioning method for various indoor scenes, belonging to the field of visual positioning in video image capture. Background Technology

[0002] Given an RGB image, the goal of visual localization is to estimate the 6-DOF camera pose from the camera coordinate system to the world coordinate system. This is a crucial step in robot perception systems, such as Structure from Motion (SFM) and Simultaneous Localization and Mapping (SLAM). Current learning-based state-of-the-art visual localization methods can be categorized into absolute pose regression, relative pose regression, and scene coordinate regression. Among these, scene coordinate regression methods use a trained convolutional neural network (CNN) to directly predict scene coordinates and employ the PnP algorithm to calculate the camera pose.

[0003] As a pioneering work, some scholars used regressive forests to predict 3D coordinates. Subsequent research mainly focused on network architecture design to achieve accurate visual localization. For example, some scholars proposed a hierarchical structure that attaches discrete location labels to each pixel to distinguish similar pixels, thereby achieving higher accuracy. Other scholars described an element-focused approach that effectively fuses multiple features to construct a sparse graph neural network, achieving comprehensive scene parsing. While these methods have yielded excellent results, they also have some drawbacks. The scene coordinate regression of these methods is scene-specific, requiring retraining for new scenes, resulting in storage costs that increase linearly with the number of scenes, making these methods unsustainable under resource-constrained conditions. Summary of the Invention

[0004] To address the limitation that visual positioning scene coordinate regression methods can only be applied to specific scenarios, this invention provides a visual positioning method applicable to multiple indoor scenarios.

[0005] The present invention provides a visual positioning method for multiple indoor scenes, comprising:

[0006] S1. Collect data in N different scenes. The data includes RGB images, depth maps, and the pose of the camera.

[0007] S2. Preprocess the data collected in S1;

[0008] S3. Take the RGB image and depth map in the scene as input, and take the scene coordinates corresponding to each pixel in the depth map as output to build a cross-scene visual localization network. Use the pre-processed RGB images of N scenes to input into the cross-scene visual localization network for training to obtain the trained cross-scene visual localization network of N scenes.

[0009] When training a cross-scene visual localization network, an adaptive parameter sharing strategy is used to determine the weights of the convolutional layers:

[0010] During forward propagation, the learnable scores for each channel of the i-th convolutional layer are determined. This represents the score of the c-th channel, where c = 1, 2, ..., C. in C in Given the number of input channels, calculate the binarized score parameter Θ(s). i ):

[0011]

[0012] Where λ represents the set threshold;

[0013] like If the value is 0, then it continues to determine whether the parameters of each convolutional kernel in the corresponding channel are shared. If the value is 1, then the parameters of each convolutional kernel in the corresponding channel are determined to be task-specific weights.

[0014] Methods for determining whether parameters are shared among convolutional kernels in the same channel:

[0015] Determine the learnable score of each convolutional kernel within the corresponding channel. h is the height of the convolution kernel, and l is the width of the convolution kernel. This represents the score at position (j,k), where j = 1, 2, ..., h, k = 1, 2, ..., l. The score parameters are calculated after binarization of the convolution kernel. like If the value is 0, then the parameters of the corresponding convolutional kernel within the channel are the shared weights w. i ,like If the value is 1, then the parameters of the corresponding convolutional kernel within the channel are task-specific weights.

[0016] The weights of the N trained cross-scene visual localization networks are integrated into a single cross-scene visual localization network.

[0017] S4. Select the weights of the corresponding cross-scene visual localization network based on the RGB image to be tested and the corresponding depth image, and then use the cross-scene visual localization network to predict the RGB image to be tested to obtain the scene coordinates of each pixel. Calculate the camera pose based on the obtained scene coordinates and the pixel coordinates in the corresponding depth image.

[0018] Preferably, the output of the cross-scene visual localization network also includes uncertainty. In S4, the cross-scene visual localization network outputs the scene coordinates and uncertainty of each pixel. The scene coordinates with poor prediction results are removed based on the uncertainty. The camera pose is calculated based on the scene coordinates of the remaining prediction results and the pixel coordinates in the corresponding depth image.

[0019] As a preferred approach, the weights from the N pre-trained cross-scene visual localization networks are integrated into a single cross-scene visual localization network:

[0020] Task-specific weights for the nth scene The gradient is:

[0021]

[0022] Indicates to Gradient calculation operation This represents the training loss for the nth task;

[0023] Task-specific weights after integration into a cross-scene visual localization network The gradient update is as follows:

[0024]

[0025] Shared weights for the nth scene The gradient is:

[0026]

[0027] Indicates to Gradient calculation operation;

[0028] Shared weights after integration into a cross-scene visual localization network The gradient update is as follows:

[0029]

[0030] Share the weights of the nth scene and task-specific weights

[0031] As a preferred option, the training loss for the nth task is:

[0032]

[0033] Where: Q is the number of pixels in the input image, c n,q Let q be the predicted scene coordinates of the q-th pixel. Let u be the real scene coordinate of the q-th pixel. n,q Let q be the uncertainty of the q-th pixel.

[0034] As a preferred approach, during backpropagation, the learnable score s of the nth scene is... n The gradient is:

[0035]

[0036] in, Indicates to The gradient calculation operation.

[0037] As a preferred embodiment, in S3, when training the cross-scene visual localization network, after each epoch, a test is performed using the current cross-scene visual localization network parameters. If the error and accuracy of the test results are both better than the saved optimal network, then the current cross-scene visual localization network parameters are saved as the optimal parameters. When the number of training iterations reaches the set epoch value, training is stopped, and the trained cross-scene visual localization network is obtained.

[0038] Preferably, in S4, the RANSAC and PNP algorithms are used to select the coordinates of Q groups of pixels and the predicted scene points, and the predicted camera pose T is obtained. * for:

[0039]

[0040] Where Q is the number of pixels in the input image, P uq Let P be the pixel coordinate of the q-th pixel. cq Let P be the depth value at the camera coordinates of the q-th pixel, K be the camera intrinsic parameter, T be the camera pose in the world coordinate system, and P be the depth value at the camera coordinates of the q-th pixel. wi Let q be the predicted scene coordinates of the q-th pixel;

[0041] Based on the predicted camera pose T * The reprojection error of each pixel is calculated using the Gauss-Newton method to determine if any outliers exist. If outliers exist, they are removed. The new camera pose T is then obtained using the RANSAC and PNP algorithms based on the remaining pixels. * If there are no exterior points, the current camera pose T * Let T be the spatial pose of the camera.

[0042] Preferably, the cross-scene visual localization network includes a feature extraction layer and a regression layer;

[0043] The feature extraction layer consists of a series of convolutional layers used to encode features in the RGB image. The feature extraction layer uses a modified ResNet34 network as the backbone network. The modified ResNet34 network removes the max pooling layer, average pooling layer and fully connected layer from the ResNet34 network and changes the stride of the four residual blocks to (1,1,2,2).

[0044] The regression layer consists of a series of convolutional layers and is used to predict scene coordinates and uncertainties.

[0045] As a preferred approach, a deep supervised method is employed during training. A regression layer is used to predict scene coordinates and uncertainties, and then the losses between each scene are weighted and summed to obtain the total loss L. total , w represents the training loss for the nth task. n To reconcile the weighting coefficients of the losses from each task, n = 1, 2, ..., N.

[0046] Preferably, in S1, a depth camera is used to acquire a large number of RGB images, depth maps, and the camera pose at the time of each frame of the image being captured in N different scenes. The camera used in different scenes is the same device, and the camera parameters are kept consistent.

[0047] The beneficial effects of this invention are that it designs an adaptive parameter sharing strategy that can automatically determine whether parameters in the channel dimension and single-layer spatial dimension are shared, which is used to effectively optimize the visual positioning feature representation of various scenarios, thereby making the visual positioning method based on scene coordinate regression applicable to a variety of scenarios. Attached Figure Description

[0048] Figure 1 This is a schematic diagram illustrating the principle of a specific embodiment of the present invention;

[0049] Figure 2 This refers to data collected in specific implementation scenarios of the present invention;

[0050] Figure 3 This is a schematic diagram of a cross-scene visual positioning network according to a specific embodiment of the present invention;

[0051] Figure 4 This is a schematic diagram illustrating the principle of the adaptive parameter sharing strategy in a specific embodiment of the present invention. Detailed Implementation

[0052] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0053] It should be noted that, unless otherwise specified, the embodiments and features described in the present invention can be combined with each other.

[0054] The present invention will be further described below with reference to the accompanying drawings and specific embodiments, but this is not intended to limit the scope of the invention.

[0055] Multi-task learning offers a remedy for this problem by treating localization of each scene as a separate task. Compared to learning using independent models, multi-task learning methods achieve more efficient learning with lower overall storage costs by sharing parameters across tasks. Furthermore, research has shown that multi-task learning has the potential to enhance the credibility of learned representations, thus benefiting specific tasks.

[0056] like Figure 1 As shown, the visual positioning method across multiple indoor scenes in this embodiment includes:

[0057] Step 1: Collect data in N different scenes. The data includes RGB images, depth maps, and the pose of the acquisition camera.

[0058] Specifically, this involves acquiring RGB images, depth maps, and camera poses in different scenarios. The acquired scene images must include all representative objects in the scene, and at least 5 sets of data must be acquired for each scene, with each set containing a certain number of images and camera poses.

[0059] Step 2: Preprocess the data collected in S1; normalize the RGB images in the scene.

[0060] Step 3: Take the RGB image and depth map of the scene as input, and take the scene coordinates corresponding to each pixel in the depth map as output to build a cross-scene visual localization network. Use the pre-processed RGB images of N scenes to input into the cross-scene visual localization network for training to obtain the trained cross-scene visual localization network of N scenes.

[0061] When training a cross-scene visual localization network, an adaptive parameter sharing strategy is used to determine the weights of the convolutional layers:

[0062] Typically, a convolutional layer consists of two parameters: weights and biases. Since the biases of batch normalization layers supersede those of convolutional layers, this parameter-sharing strategy only considers the weight parameters. Taking the convolutional kernel of the i-th convolutional layer as an example, its initial weights w... i As shared parameters for the task, two more sets of parameters are designed, namely task-specific weights. and learnable score s i .

[0063] During forward propagation, the learnable scores for each channel of the i-th convolutional layer are determined. This represents the score of the c-th channel, where c = 1, 2, ..., C. in C in Given the number of input channels, calculate the binarized score parameter Θ(s). i ):

[0064]

[0065] Where λ represents the set threshold;

[0066] The metric function Θ(·) is used to determine whether the parameters of the convolutional layer are shared in the current iteration; the adaptive parameter sharing strategy shares parameters from both the channel and kernel dimensions, such as... Figure 4 As shown.

[0067] like If the value is 0, then it continues to determine whether the parameters of each convolutional kernel in the corresponding channel are shared. If the value is 1, then the parameters of each convolutional kernel in the corresponding channel are determined to be task-specific weights.

[0068] For channel-dimension sharing, a learnable score is defined. Whether it is shared can be obtained from the index function Θ(·). Therefore, the task adaptive weights are derived. for:

[0069]

[0070] here This represents element-wise multiplication. Used to generate the output of the current iteration. For the j-th channel, if If the value is greater than the predefined threshold λ, then the task shares the weights. It will be activated and optimized, and vice versa.

[0071] Methods for determining whether parameters are shared among convolutional kernels in the same channel:

[0072] Determine the learnable score of each convolutional kernel within the corresponding channel. h is the height of the convolution kernel, and l is the width of the convolution kernel. This represents the score at position (j,k), where j = 1, 2, ..., h, k = 1, 2, ..., l. The score parameters are calculated after binarization of the convolution kernel. if If the value is greater than the predefined threshold λ, then the task shares the weights. It will be activated and optimized, and vice versa. If If the value is 0, then the parameters of the corresponding convolutional kernel within the channel are the shared weights w. i ,like If the value is 1, then the parameters of the corresponding convolutional kernel within the channel are task-specific weights.

[0073] The weights of the N trained cross-scene visual localization networks are integrated into a single cross-scene visual localization network.

[0074] Step 4: Select the weights of the corresponding cross-scene visual localization network based on the RGB image to be tested and the corresponding depth image. Then use the cross-scene visual localization network to predict the RGB image to be tested and obtain the scene coordinates of each pixel. Calculate the camera pose based on the obtained scene coordinates and the pixel coordinates in the corresponding depth image.

[0075] This implementation uses a multi-task learning approach to collectively optimize visual localization tasks across various scenarios. For multi-task, cross-scene models, the challenge lies in identifying which parameters should be shared between scenes and which should not, thereby achieving effective localization across multiple scenes. To address this issue, this implementation designs an adaptive parameter sharing strategy that automatically determines whether parameters in the channel dimension and single-layer spatial dimension are shared. Compared to traditional scene coordinate regression methods, this implementation adds uncertainty to the output. Uncertainty is used to eliminate scene coordinates with poor prediction performance, and visual localization is achieved based on the more accurate predicted scene coordinates.

[0076] In this embodiment, the output of the cross-scene visual positioning network also includes uncertainty. In S4, the cross-scene visual positioning network outputs the scene coordinates and uncertainty of each pixel. Scene coordinates with poor prediction results are eliminated based on the uncertainty. The camera pose is calculated based on the remaining scene coordinates with good prediction results and the pixel coordinates in the corresponding depth image. Uncertainty is used to evaluate the quality of the scene coordinate prediction results. By setting a threshold for uncertainty, scene point coordinates with uncertainty greater than the threshold are eliminated.

[0077] The specific process of step 1 in this embodiment is as follows:

[0078] A large number of RGB images, depth maps, and camera poses at the time of capture were acquired using a depth camera in different scenes. The same camera was used in all scenes, and the camera parameters remained consistent. The acquired data is as follows: Figure 2 As shown.

[0079] The specific process of step 2 in this embodiment is as follows:

[0080] Calculation of pixel scene coordinates. Specifically, based on the acquired depth map, the camera coordinates P corresponding to each pixel can be obtained. c Combined with camera pose T cw It can calculate the scene coordinates corresponding to a pixel.

[0081] Data augmentation operations are performed on the images in the data, including: random translation of 30 pixels, random scaling from 0.8 to 1.6 times, random rotation from -40° to 40°, and increasing the number of samples in the database. This improves the learning ability of the neural network while effectively preventing overfitting.

[0082] The collected data is randomly divided into a training set and a test set, with the ratio between the training set and the test set approximately satisfying 4:1.

[0083] Normalize the RGB image. Specifically, the method used is: v i ′=(v i / 255)×2-1. Where, v i v is the initial pixel value. i ′ represents the normalized pixel value, which limits the RGB values ​​to the range [-1, 1].

[0084] The specific process of step 3 in this embodiment is as follows:

[0085] Build a cross-scene visual positioning network, the network structure is as follows: Figure 3 As shown. The input consists of multiple RGB images from different scenes, and the output is the scene coordinates and uncertainty corresponding to the pixels in the RGB images. The mini-batch size is set to 4 when inputting the training set images into the network. The ADAM optimizer is used, with hyperparameters set to β1 = 0.9 and β2 = 0.999. A cosine annealing learning rate decay strategy is employed, with an initial learning rate of 1.4 × 10⁻⁶. -3 Using deep supervision techniques, we can simultaneously improve the network's learning ability and optimization speed.

[0086] The cross-scene visual localization network of this embodiment includes two parts: the first part is a feature extraction layer, which performs low-level and high-level feature extraction on images in different scenes and encodes geometric spatial information and semantic information in the environment; the second part is a regression layer, which is used to predict scene coordinates and uncertainty.

[0087] The structure of the feature extraction layer is as follows:

[0088] The feature extraction layer takes a 5×480×640 input tensor, where 5 represents the color value (R, G, B) and pixel coordinates (u, v) of each pixel, and 480 and 640 represent the height and width of the image, respectively. The feature extraction layer consists of a series of convolutional layers used to encode features in the scene image. Considering the dimensionality of the input tensor, a modified ResNet34 is used as the backbone network. Modifications include removing the max-pooling, average-pooling, and fully connected layers, and changing the stride of the four residual blocks to (1, 1, 2, 2).

[0089] The feature extraction layer encodes features in the scene image in five stages, corresponding to Convolutional Layer 1, Residual Block 1, Residual Block 2, Residual Block 3, and Residual Block 4, defined as C1, B1, B2, B3, and B4, respectively. During feature extraction, the first step is the convolutional layer with a kernel size of 7×7 and a stride of 2. Then, residual blocks are used for further feature extraction, obtaining the first, second, and third feature maps. The advantage of using residual blocks is that, through shortening structures, low-dimensional features are appended to high-dimensional features, preventing information loss and network degradation. Simultaneously, the shortening structure further avoids gradient vanishing and gradient exploding. Residual blocks do not significantly increase the number of network parameters but can improve the network's training performance. The feature extraction layer contains a series of task-shared parameters ρ for multiple tasks n. sh and task-specific parameters Each input I n Convert to intermediate representation

[0090] Table 1. Network Module Architecture of Feature Extraction Layer

[0091]

[0092] After each convolution, a batch normalization (BN) layer and an activation function are used: the BN layer smooths the surface of the loss function, helping to improve the training speed of the network. The activation function helps to increase the non-linearity of the network.

[0093] The structure of the regression layer is as follows:

[0094] The regression layer consists of a series of convolutional layers used to predict scene coordinates and uncertainty. The regression layer is divided into six stages, corresponding to convolutional layers 1, 2, 3, 4, and parallel convolutional layers 5 and 6. All convolutional layers have a kernel size of 3×3 and a stride of 1. After passing through convolutional layers 1, 2, 3, and 4, the eleventh feature map is obtained; this eleventh feature map is then passed through convolutional layers 5 and 6 to obtain the scene coordinates and uncertainty. Convolutional layers 1, 2, 3, and 4 include convolution operations, batch normalization, and ReLU activation function processing, while convolutional layers 5 and 6 only contain convolution operations. The regression layer contains a series of task-specific parameters for multiple tasks n. Each intermediate representation F n Transform it into the final output.

[0095] Table 2 Regression Layer Network Module Architecture

[0096] Module Name Convolution parameters Output feature map size Input layer — 512×60×80 First convolutional layer [3×3,512] 512×60×80 Second convolutional layer [3×3,256] 256×60×80 Third convolutional layer [3×3,128] 128×60×80 Fourth convolutional layer [3×3,128] 128×60×80 Fifth convolutional layer [3×3,3] 3×60×80 sixth convolutional layer [3×3,1] 1×60×80

[0097] The RGB images from the training set are input into the constructed cross-scene visual localization network for training. The number of epochs is defined based on the data size. After each epoch, a test is performed using the current model parameters. If the error and accuracy of the test results are both better than the saved optimal model, the current model parameters are saved as the optimal parameters. When the network has reached the set number of epochs, training stops, and the trained cross-scene visual localization network is obtained.

[0098] In this implementation, the weights of the trained N scene-based visual localization networks are integrated into a single scene-based visual localization network:

[0099] Task-specific weights for the nth scene The gradient is:

[0100]

[0101] Indicates to Gradient calculation operation This represents the training loss for the nth task;

[0102] According to the calculation Optimize specific task parameters for each scenario; when using multiple optimization scenarios, the gradient... The weights will be averaged and integrated into a cross-scene visual localization network for task-specific weights. The gradient update is as follows:

[0103]

[0104] Shared weights for the nth scene The gradient is:

[0105]

[0106] Indicates to Gradient calculation operation;

[0107] Shared weights after integration into a cross-scene visual localization network The gradient update is as follows:

[0108]

[0109] Share the weights of the nth scene and task-specific weights

[0110] The learnable score optimization method in this embodiment is as follows:

[0111] During backpropagation, the score s of the nth scene n The gradient can be expressed as:

[0112]

[0113] Furthermore, this invention sets the learnable scores to be task-shared to ensure that the parameters of all tasks can be integrated into a single ensemble model, i.e., s1 = s2 = ... = s N .

[0114] In this implementation, all activation functions use the ReLU activation function. The specific form of the ReLU activation function is as follows:

[0115]

[0116] Where x represents the input and ReLU(x) represents the output.

[0117] The specific process of step 4 in this embodiment is as follows:

[0118] The RGB images from the test set are input into the feature deep aggregation network. During the input process, the mini-batch is set to 1 to obtain the scene coordinates and uncertainty corresponding to each pixel in the RGB image.

[0119] Uncertainty is used to evaluate the quality of scene coordinate prediction results. By setting a threshold for uncertainty, scene point coordinates with uncertainties greater than the threshold are eliminated. Among the scene point coordinates with uncertainties less than the threshold, the RANSAC algorithm is used to select 256 sets of pixels and their predicted scene point coordinates. Subsequently, the Gauss-Newton method is used to optimize the reprojection error.

[0120]

[0121] Where N is the number of pixels in the input image, P ui Let P be the pixel coordinate of the i-th pixel. ci Let P be the depth value of the camera coordinates of the i-th pixel, K be the camera intrinsic parameter, T be the camera pose in the world coordinate system, and P be the depth value of the i-th pixel. wi Let be the predicted scene coordinates of the i-th pixel. Obtain the predicted camera pose T. * Based on this, the reprojection error of each pixel is calculated to determine whether there are outliers. If outliers exist, they are removed. The RANSAC and PNP algorithms are then used to obtain the new camera pose T based on the remaining pixels. * If there are no exterior points, the current camera pose T * Let T be the spatial pose of the camera.

[0122] Considering the accuracy requirements of camera visual positioning in the environment, performance evaluation mainly focuses on two indicators: positional deviation and angular deviation. Smaller deviations indicate higher positioning accuracy. The table below compares the results of the method of this invention with other methods. Compared to other methods, this invention achieves a positional deviation of 0.018m and an angular deviation of 0.640°, which is superior.

[0123] Table 3 Comparative experimental data of different networks

[0124]

[0125]

[0126] Example:

[0127] The goal of this embodiment is to achieve cross-scene visual localization. To ensure the effectiveness of the cross-scene visual localization network, relevant data were collected in various architectural environments.

[0128] RGB images, depth maps, and camera poses were acquired at a rate of 30 FPS. Five datasets were collected for each scene, each containing a certain number of images and camera poses. Four datasets were used for training, and one dataset was used for testing.

[0129] Based on the acquired camera pose and depth map, the scene coordinates corresponding to the pixels in the depth map and RGB image can be calculated.

[0130] During training, overfitting may occur due to issues such as insufficient data or an excessively large network model. This embodiment uses data augmentation operations, including: random translation of 30 pixels, random scaling from 0.8 to 1.6 times, and random rotation from -40° to 40°. Simultaneously, considering the network's training speed, the RGB images were normalized.

[0131] Considering that the image size in this embodiment is only 640×480, the Adam optimizer is used during training, and the mini-batch is set to 4.

[0132] The model converges quickly, achieving convergence within 200,000 iterations on two Tesla V100 GPUs. Ultimately, on the test set, it achieved a positional deviation of 0.020m and an angular deviation of 0.63°.

[0133] While the invention has been described herein with reference to specific embodiments, it should be understood that these embodiments are merely examples of the principles and applications of the invention. Therefore, it should be understood that many modifications can be made to the exemplary embodiments, and other arrangements can be designed without departing from the spirit and scope of the invention as defined by the appended claims. It should be understood that different dependent claims and features described herein can be combined in ways different from those described in the original claims. It is also understood that features described in conjunction with individual embodiments can be used in other described embodiments.

Claims

1. A method of visual localization across multiple indoor scenes, characterized in that, The method comprises: S1, collecting data in N different scenes, the data comprising RGB images, depth maps, and poses of the collection cameras; S2, preprocessing the data collected in S1; S3, building a cross-scene visual positioning network using the RGB images and the depth maps in the scenes as inputs and the scene coordinates corresponding to each pixel point in the depth maps as outputs, inputting the preprocessed RGB images of the N scenes into the cross-scene visual positioning network for training to obtain the trained cross-scene visual positioning network of the N scenes; When training the cross-scene visual positioning network, an adaptive parameter sharing strategy is adopted for determining the weights of the convolutional layers: In forward pass, determine the learnable scores for each channel of the i-th convolutional layer , denote the score of the i-th channel, , is the number of input channels, compute the binarized score parameters :​ wherein, denotes a set threshold value; If is 0, it continues to determine whether the parameters of each convolution kernel in the corresponding channel are shared, if is 1, it determines that the parameters of each convolution kernel in the corresponding channel are task-specific weights ; A method for determining whether the parameters of each convolution kernel in the corresponding channel are shared: determining the learnable score of each convolution kernel in the corresponding channel , is high for the convolution kernel, is wide for the convolution kernel, denotes the score at position (j, k), , , calculating the score parameter of the binarized convolution kernel , if is 0, the parameter of the corresponding convolution kernel in the channel is a shared weight , if is 1, the parameter of the corresponding convolution kernel in the channel is a task-specific weight ; Integrating the weights in the trained cross-scene visual positioning network of the N scenes into one cross-scene visual positioning network; S4, selecting the weights of the corresponding cross-scene visual positioning network according to the to-be-measured RGB image and the corresponding depth image, and then using the cross-scene visual positioning network to predict the to-be-measured RGB image to obtain the scene coordinates of each pixel point, and calculating the camera pose according to the obtained scene coordinates and the pixel coordinates in the corresponding depth image.

2. The method of visual localization across multiple indoor scenes of claim 1, wherein, The output of the cross-scene visual positioning network further comprises an uncertainty, in S4, the cross-scene visual positioning network outputs the scene coordinates and the uncertainty of each pixel point, the scene coordinates with poor prediction effects are removed according to the uncertainty, and the pose of the camera is calculated according to the remaining scene coordinates with good prediction effects and the pixel coordinates in the corresponding depth image.

3. The method of visual localization across multiple indoor scenes of claim 1, wherein, Integrating the weights in the trained cross-scene visual positioning network of the N scenes into one cross-scene visual positioning network: task-specific weight of the nth scene the gradient of the nth scene is: denotes a gradient operation, denotes a gradient operation, denotes a training loss of the nth task; task-specific weights integrated into a cross-scenario visual localization network gradient updates are: shared weight value of the nth scene the gradient of the function representing an operation to gradient operation; shared weights of the integrated cross-scenario visual positioning network gradient update is: 。 4. The method of visual localization across multiple indoor scenes of claim 3, wherein, The training loss of the nth task: wherein: is the number of pixels of the input image, is the true scene coordinate of the pixel, is the predicted scene coordinate of the pixel, is the uncertainty of the pixel.

5. The method of visual localization across multiple indoor scenes of claim 4, wherein, During backpropagation, the gradient of the learnable score for the nth scene is: ​ wherein, represents a gradient operation on .

6. The method of visual positioning across multiple indoor scenes of claim 1, wherein, In S3, when training the cross-scene visual positioning network, once an epoch is completed, a test is performed using the current cross-scene visual positioning network parameters, if the error and the accuracy of the test result are better than those of the saved optimal network, the current cross-scene visual positioning network parameters are saved as the optimal parameters, and when the number of training reaches a set epoch value, the training is stopped to obtain the trained cross-scene visual positioning network.

7. The method of visual positioning across multiple indoor scenes of claim 1, wherein, In S4, RANSAC and PNP algorithm are used to select Group pixel points and predicted scene point coordinates, predicted camera pose are: in, The number of pixels in the input image. For the first Pixel coordinates of 1 pixel For the first The depth value of the camera coordinates in pixels. For camera internal parameters, This represents the camera's pose in the world coordinate system. For the first The predicted scene coordinates for each pixel; According to the predicted camera pose , a re-projection error of each pixel point is calculated using a Gauss-Newton method, it is judged whether there is an outlier, if there is an outlier, the outlier is removed, and a new camera pose is obtained according to the remaining pixel points using a RANSAC and a PNP algorithm , if there is no outlier, the current camera pose is the spatial pose of the camera .

8. The method of visual positioning across multiple indoor scenes of claim 1, wherein, The cross-scene visual positioning network comprises a feature extraction layer and a regression layer; The feature extraction layer is composed of a series of convolutional layers and is used for encoding the features in the RGB images; The feature extraction layer uses a modified ResNet34 network as a backbone network, the modified ResNet34 removes the maximum pooling layer, the average pooling layer, and the fully connected layer in the ResNet34 network, and changes the strides of the four residual blocks to (1, 1, 2, 2); The regression layer is composed of a series of convolutional layers and is used for predicting the scene coordinates and the uncertainty.

9. The method of visual positioning across multiple indoor scenes of claim 8, wherein, During training, a deep supervision method is adopted, using a regression layer to predict the scene coordinates and uncertainty, and then the loss between each scene is weighted and added to obtain the total loss , , represents the training loss of the nth task, is the weight coefficient for adjusting the loss of each task, .

10. The method of visual positioning across multiple indoor scenes of claim 1, wherein, In S1, a depth camera is used to collect a large number of RGB images, depth maps, and the camera poses recorded at the time of shooting each frame of image in N different scenes, the cameras used in different scenes are the same device, and the camera parameters remain consistent.

Images