A visual relocalization method combining graph attention and region classification

Abstract

This invention discloses a visual relocalization method combining graph attention and region classification, relating to the field of visual relocalization in computer vision. This method employs scene region classification to model and hierarchically classify the spatial scene, re-label the data, and uses hierarchical scene coordinate region regression and relative coordinate prediction to transform the relocalization problem from a coarse-to-fine approach into a classification problem. Ultimately, the localization is concentrated within a small region, which helps improve relocalization in large-scale scenes. By utilizing a graph attention mechanism and combining the classification results, pixels in the image can better acquire information from other pixels within the same region, generating superior feature representations. This invention can solve visual relocalization problems in real-world scenes.

Description

Technical Field

[0001] This invention relates to visual relocation techniques in the field of computer vision, and particularly to a visual relocation method based on graph attention and region classification. Background Technology

[0002] Relocalization technology aims to analyze the 6-DoF (degrees of freedom) pose corresponding to the query input in a known scene, achieving precise localization of the shooting orientation and angle. It is a key module in SLAM (Simultaneous Localization and Mapping) systems. In applications such as autonomous robots, self-driving cars, and AR (Augmented Reality) and VR (Virtual Reality) technologies, high reliability and accuracy in localization are required, making relocalization a fundamental and crucial component. Depending on the sensor used, relocalization can be categorized in several ways. Among these, vision-based relocalization, also known as visual relocalization, is widely used and has become a research hotspot in the field of computer vision due to its readily available sensors, rich information, and ability to simultaneously perform multiple visual tasks.

[0003] Visual relocalization methods have evolved from geometric structure methods to image retrieval methods, and then to deep learning methods. The current visual relocalization process can be broadly divided into three steps: scene model construction, image-scene matching, and camera pose calculation. Geometric structure-based methods pre-store a 3D model of the scene and then establish a correspondence by matching the visual features of spatial points in the current query image with those in the 3D scene model. The pose is then solved using the PnP (Perspective-n-Point) algorithm based on RANSAC (Random Sample Consensus). However, this method requires storing a large 3D model and may fail to find keypoints or matching descriptors in certain situations, such as motion blur and textureless regions. Furthermore, manually extracted features have low cross-scene adaptability and high computational cost. Image retrieval-based methods store images with known poses in a database. For a given query image, the database searches for images that are similar to the query image. The pose of the query image can be obtained from the closest image, or a more accurate estimate can be obtained by calculating the relative pose between the two images. Because there are often some differences between the query images and the database images, they are limited in accurately estimating camera pose.

[0004] Absolute pose regression is a relatively direct and simple method for visual relocalization using deep learning, aiming to maintain both efficiency and scalability. It uses known images as a training set, encoding the relationship between image content and camera pose in its parameters to learn this relationship. Upon query, the network directly regresses the 6-DoF pose of the corresponding image. However, current absolute pose regression methods are more similar to image retrieval methods and have failed to surpass them. Scene coordinate regression methods first predict the 3D coordinates of the query image pixels to construct a 2D-3D correspondence, and then, similar to geometry-based methods, calculate the pose using a RANSAC-based PnP algorithm. With the development of deep learning, people have begun to use deep learning to regress scene coordinates, and are gradually using more readily available RGB images as query images. Scene coordinate regression methods offer a better trade-off between accuracy and computational efficiency, becoming a popular class of methods in visual relocalization. However, in large scenes, the blurring of visually similar local image patches cannot be resolved with a limited receptive field. Simply using a larger receptive field size can easily lead to overfitting. Furthermore, the traditional CNN (Convolutional Neural Networks)-based network structure is relatively simple and cannot specifically learn the relationships between nodes, which may lead to a lack of understanding of the global context and the uniqueness of multiple feature clusters.

[0005] In conclusion, it is necessary to propose an effective relocalization method to adapt to scenes of different scales and the concentration of some visual features. Summary of the Invention

[0006] To address the aforementioned problems, the present invention aims to provide a visual relocalization method that combines graph attention and region classification.

[0007] This invention utilizes a coarse-to-fine scene coordinate regression network based on scene region segmentation. After classifying pixels in an image, a graph attention mechanism is applied to pixels of the same category, enabling the network to more fully learn the features between pixels of the same category. The technical solution of this invention includes two processes: network training and inference. The specific technical solution is as follows:

[0008] Step 1: Create a dataset using existing data or by taking pictures, and perform preprocessing such as data labeling and partitioning to obtain the input images used for network training;

[0009] Step 2: Initially extract image features through a series of convolution operations to obtain the features of each pixel in the image;

[0010] Step 3: Obtain the region corresponding to each pixel through a multi-level pixel-level scene region regression module;

[0011] Step 4: Optimize the features corresponding to pixels in the image by performing graph attention-based inter-pixel operations within the same region;

[0012] Step 5: Convert the features corresponding to each pixel into corresponding 3D coordinates through a multi-layer perceptron layer;

[0013] Step 6: Calculate the 6-DoF pose of the input image using the RANSAC-based PnP algorithm.

[0014] Furthermore, the dataset mentioned in step 1 includes images of the corresponding scene obtained by means of shooting, the 6-DoF pose corresponding to the image, and the region number of the pixel in the image corresponding to the pixel in the 3D scene obtained by calculation.

[0015] Furthermore, the training set content in the dataset mentioned in step 1 can be enriched through data augmentation methods such as cropping, rotation, and brightness adjustment to better adapt to small-scale datasets.

[0016] Furthermore, the image feature extraction module mentioned in step 2 is composed of arbitrary convolutional layers. Given an input image, its output feature map is represented as follows:

[0017] X1∈R H×W×C

[0018] Here, H and W represent the height and width of the image, respectively, and C represents the number of channels. The image can also be downsampled before this process to reduce memory requirements. If the above processing is performed, the first two dimensions will change accordingly.

[0019] Furthermore, the multi-level pixel-level scene region regression module mentioned in step 3 has the following specific process:

[0020] Step 3.1: During scene preprocessing, set the total number of layers in the network model to classify scene regions. Then, classify the scene regions from coarse to fine according to the layers. The classification of the next layer is carried out in the sub-regions obtained in the previous layer. There is no overlap between the sub-regions in the same layer. For example, if the total number of layers is selected as 2 and 10 categories are selected in each layer, then the first layer divides the entire scene space into 10 sub-regions. In the second layer, each sub-region obtained in the first layer is further divided into 10 sub-regions. Then there are a total of 10×10 sub-regions in this layer.

[0021] Step 3.2: For each layer in Step 1, set arbitrary convolutional layers and residual connections, and take the output X1 obtained in Step 2 as input. The feature map dimension of the last convolutional layer output is H×W×K, where K represents the number of blocks in the previous layer that it belongs to at this layer, and at the highest layer, it represents the number of blocks that classify the entire scene. Then, using one-hot encoding, the pixel region category in the image at this layer is represented as:

[0022] Y Lm ∈{0,1} H×W×K

[0023] Where m∈{1,2,…,T}, and T is the total number of layers in the network model that classify the scene region. After this step, each pixel can be classified into a category. If this layer is not the highest layer, it is classified into a subcategory under its parent category.

[0024] Step 3.3: Except for the highest level, each subsequent scene region regression module receives information input from the pixel-level region classification regression layer above. During training, the next layer receives the ground truth label input from the previous layer; during inference, the next layer receives the predicted label input from the previous layer. This process is accomplished using a FiLM (Feature-wise Linear Modulation) based adjustment layer, as follows:

[0025] Step 3.3.1: For the training process, the input is the true label of each pixel. For the prediction process, the input is the predicted value of each pixel in the previous level. The feature map w is obtained by passing through any convolutional layer.

[0026] Step 3.3.2: Perform linear adjustment on the current layer input feature map x, specifically expressed as follows:

[0027] f(x,w)=γ(w)⊙x+β(w)

[0028] Here, ⊙ represents the Hadamard product, and γ(w) and β(w) are trained together during network training. One or more FiLM-based conditioning layers can be applied to each layer of the network.

[0029] After each layer in step 3, each pixel in the input image can be mapped to a sub-region in the scene corresponding to that network layer. Specifically, after completing the last layer of step 3, each pixel in the input image has a corresponding category in each layer of the region division from coarse to fine.

[0030] Furthermore, the specific process of the graph attention-based inter-pixel operation in step 4 is as follows:

[0031] Step 4.1: After step 3, each pixel in the input image has obtained the corresponding sub-region category prediction at different levels, which represents the sub-region to which it belongs in the corresponding level after the scene is segmented hierarchically.

[0032] Step 4.2: During the operation between pixels in the same region based on graph attention, the category of each pixel in the input image is based on the true label during training and the predicted label obtained after step 4.1 during inference.

[0033] Step 4.3: After determining the region category label of each pixel in the input image in Step 4.2, pixels of the same category are grouped together according to their category. A threshold K is determined based on the parameter settings when dividing the scene region hierarchy and the size of the input image. min .

[0034] Step 4.4: Count the number of pixels in each group after grouping in 4.3, and keep the number ≥ K. min The groups are represented as follows:

[0035]

[0036] Where i∈{1,2,…,T}, and T is the number of elements ≥K. min The number of groups.

[0037] Step 4.5: Set the number of layers L in the graph attention mechanism, then the graph attention operation is represented as follows:

[0038]

[0039] in, Right now This represents the entire graph attention operation process. This represents the feature representation of all pixels in the same group in step 4.4. This represents each layer of the graph attention mechanism.

[0040] For each group after filtering mentioned in step 4.4, graph attention operation is performed, and the specific process is as follows:

[0041] Step 4.5.1: Select the current group

[0042] Step 4.5.2: For Each layer This can be expressed as: for the current group Each element in the point i Corresponding feature representation (l) di The next layer features are calculated using the method described in the following formula. (l+1) d i :

[0043]

[0044] in, express The aggregated message result of the feature descriptors of all pixels in the graph for point i, where [.||.] represents the concatenation operation. In different graph attention layers... They are not the same. The calculation is performed using the method described in the following formula:

[0045]

[0046] Wherein, attention weight α ij The key-query similarity after Softmax is calculated using the method described below:

[0047]

[0048] Where q, k, and v are the query, key, and value in the graph attention mechanism operation, calculated using the method described below:

[0049]

[0050] The coefficients are not the same in different graph attention layers. Multi-head attention mechanisms can be used to improve the computation of graph attention.

[0051] Step 4.5.3: Perform the calculation for the next group. After all groups have undergone the above calculation, it is considered that one graph attention mechanism module has been completed.

[0052] Furthermore, the conversion of the features corresponding to each pixel into corresponding 3D coordinates mentioned in step 5 is performed by regressing a residual value for each pixel, rather than by directly regressing the 3D coordinates corresponding to that pixel. This residual value represents the difference between the actual 3D coordinates of the pixel and the 3D coordinates of the center position of the finest-grained sub-region regressed at the lowest level in step 3. Therefore, the actual 3D position coordinates corresponding to that pixel should be obtained by combining the results of each layer in step 3 with the aforementioned residual value.

[0053] The beneficial effects of adopting the above technical solution are as follows:

[0054] This invention addresses several existing problems in the visual relocalization domain by proposing a novel and effective method combining graph attention and region classification. The method utilizes scene region segmentation, designing the network from a global to a local perspective. This allows the network to browse the entire scene region in a coarse-to-fine manner, incorporating global information through region classification labels. This achieves more global information at the fine-grained layer, while avoiding ambiguity in local appearances during training and maintaining a large receptive field at the fine-grained layer during inference. A category-based graph attention mechanism is proposed, which, compared to traditional convolutional networks, allows pixels in an image to better acquire information from other points within the same region, resulting in superior feature representations. Attached Figure Description

[0055] Figure 1 This is a flowchart illustrating the network training and predictive inference process of the present invention in a new scenario.

[0056] Figure 2 This is a schematic diagram of the overall network structure of the present invention;

[0057] Figure 3 This is a schematic diagram of the scene area classification module of the present invention;

[0058] Figure 4 This is a schematic diagram of the relative coordinate regression module and graph attention module of the present invention. Detailed Implementation

[0059] The following detailed description of a typical embodiment of the visual relocalization method combining graph attention and region classification of the present invention provides a further specific description of the method.

[0060] In implementing this invention, the applicable relocalization scenario should first be selected, and images of various locations within the scene and the pose corresponding to each image should be acquired using tools such as depth cameras. In this embodiment, the chess scene from the existing dataset 7Scenes is selected. This dataset provides six sets of images, each corresponding to a video sequence captured in that scene. Each set contains 1000 images. Sets 1, 2, 4, and 6 are selected as the training set, and sets 3 and 5 are selected as the test set. Each image provides corresponding depth images and 6-DoF pose information.

[0061] In this embodiment, the total number of layers for scene region classification is set to 2. Specifically, each layer is divided into 25 sub-regions, so the entire scene region is divided into 25 primary sub-regions, and each primary sub-region is further divided into 25 secondary sub-regions, for a total of 625 secondary sub-regions. Then, the image is re-labeled according to the divided regions, so that each pixel corresponds to a primary sub-region label and a secondary sub-region label. The specific method is as follows:

[0062] a.1. Based on the training set images, camera parameters, and pose information, the pixels in each image are projected into the world coordinate system through a transformation from the image coordinate system to the camera coordinate system, and then to the world coordinate system, forming a point cloud representation of the scene. To avoid the large number of points in the point cloud due to the large dataset, which would hinder region segmentation and relabeling, downsampling is performed on the point cloud during this process. Downsampling of the current point cloud model is set to occur every 35 images, with a ratio of 0.02.

[0063] a.2. After obtaining the point cloud representation of the scene, the scene is divided using the K-Means clustering algorithm. First, the K-Means clustering algorithm is performed on the overall point cloud representation of the scene, resulting in 25 sub-classes, corresponding to 25 first-level sub-regions. Then, the K-Means clustering algorithm is performed again on each first-level sub-region, resulting in 25 second-level sub-regions. During the process, the coordinates of the center points of each first-level and second-level sub-region are recorded and saved.

[0064] a.3. For each image, using the process described in a.1 above, obtain the 3D coordinates of each pixel in the world coordinate system. Then, for each pixel, find the center point among the center points of the first-level sub-regions recorded in a.2 that has the closest Euclidean distance to it. The category corresponding to this center point is recorded as the first-level sub-region category of that pixel. Similarly, obtain the second-level sub-region category of that pixel. At this point, the re-annotation process is complete.

[0065] In this embodiment, the dataset images are all 480×640 pixels. To optimize network computation efficiency and reduce memory requirements, a uniform ×1 / 8 downsampling process is performed on the images before they are input to the network. Therefore, the input image size to the network in this embodiment is 60×80 pixels with a dimension of 3. Pixels with a depth information of 0 in the depth image are considered invalid points and are masked and recorded during training data processing.

[0066] The feature extraction module in this embodiment consists of a series of convolutional layers. The final number of channels in the convolutional layer is set to 512. After passing through this module, the input image with dimensions of 60×80×3 is represented as a feature map with dimensions of 60×80×512.

[0067] The scene region classification module in this embodiment consists of a series of convolutional layers with residual connections and FiLM-based adjustment layers. The total number of layers for scene region classification is 2, and the scene coordinate region classification module also has 2 layers. In the first layer, the final convolutional layer has 25 channels, thus outputting a feature map with dimensions of 60×80×25. One-hot encoding yields a form of {0,1}. 60×80×25The first layer represents the sub-regions, classifying each pixel of the input image into a first-level sub-region. The second layer adds a FiLM-based adjustment layer, adjusting the feature map using the equation f(x,w) = γ(w)⊙x + β(w), where w represents the ground truth label y during training. r During the reasoning process Here, γ(·) and β(·) consist of a series of 1×1 convolutions, and two FiLM-based adjustment layers are used in this layer. The final output of the second layer is a feature map with dimensions of 60×80×25. One-hot encoding yields a form of {0,1}. 60×80×25 The sub-region representation maps each pixel of the input image to a secondary sub-region for classification.

[0068] In this embodiment, the relative coordinate regression module's initial part is similar to the second layer of the scene region classification module, consisting of a series of convolutional layers with residual connections and a FiLM-based adjustment layer. The FiLM-based adjustment layer uses two levels of scene region classification information, resulting in a 60×80×64 dimension feature map. This map then enters the graph attention module, where the number of graph attention mechanism layers L is set to 5. Each point in the input image is grouped into one of 625 secondary sub-regions based on its primary and secondary sub-region labels. A minimum threshold K is selected. min For the selected groups, inter-group graph attention is performed using the previously described method to update the corresponding pixel features. Subsequently, a vector with dimensions of 60×80×3 is output through two 1×1 convolutions and an MLP layer. Combined with previous and As the output of the entire network, the scene space coordinates prediction of each masked valid point in the input image is generated by... and The center coordinates of the sub-regions represented by the joint representation are obtained by summing them.

[0069] In this embodiment, the loss function is set as follows:

[0070] L=λ1L c +λ2L r +λ3L rep

[0071] L c The classification loss is represented as follows:

[0072]

[0073] i refers to each valid pixel in the image.

[0074] L r The coordinate regression loss is represented as follows:

[0075]

[0076] i refers to each valid pixel in the image.

[0077] L rep The reprojection loss is expressed as follows:

[0078]

[0079] i refers to each valid pixel in the image. R refers to the rotation matrix in the true pose, t refers to the translation matrix in the true pose, π(.) refers to mapping the point in world coordinates to the image coordinate system, and u refers to the true coordinates of the pixel in the image coordinate system.

[0080] In this embodiment, λ1 = 1, λ2 = 10, and λ3 gradually increases from 0 to 0.1 after the initial 10 rounds of training. c-r =λ c-s =1,λ rce =1,λ ce =0.1.

[0081] In this embodiment, the training device is an NVIDIA RTX3090 GPU (24g). The number of iterations per scene is set to 300K, the Adam optimizer is used, the batch size is set to 1, and the learning rate is set to 1e-4. The training data is augmented by applying random rotation (±30°), random horizontal and vertical shift (20%), random scaling (0.7-1.5), and brightness adjustment (±20).

[0082] The model evaluation method in this embodiment mainly consists of three indicators: translation error (unit: cm), angle error (unit: °), and accuracy. Translation error and angle error refer to the differences in translation and angle between the test sample and the true label. Accuracy refers to the proportion of samples in a set of test samples that simultaneously meet the requirements of translation error ≤ 5 cm and angle error ≤ 5 °.

[0083] The above is a specific implementation example of a visual relocalization method combining graph attention and region classification proposed in this invention, illustrating the principles and main implementation methods of the invention. It should be noted that non-essential improvements and adjustments made to this method by those skilled in the art based on the above description still fall within the scope of protection of this invention.

Claims

1. A visual relocalization method combining graph attention and region classification, characterized in that: (1) Includes the following steps: a. Model the dataset processing, divide the scene region into multi-level sub-region spaces, and relabel the pixels in the dataset images with multi-level sub-region labels; b. Train the network model using the training data to obtain the network parameter data corresponding to this scenario; c. Based on the existing network model, input the image and obtain the spatial coordinate relationship of each pixel in the input image; d. Calculate the 6-DoF pose of the input image using the RANSAC-based PnP algorithm; (2) The network model includes: a. A feature extraction module, consisting of several convolutional layers, represents the input image as a feature map; b. A scene region regression module that predicts the region to which each valid pixel in the input image belongs at each level; c. A relative coordinate regression module uses the above-mentioned region prediction information to perform graph attention mechanism operation on pixels of the same category in the input image, combined with convolution operation, and finally generates the three-dimensional spatial coordinates corresponding to the effective pixels in the image; (3) The processing, modeling, and scene region segmentation of the dataset include: a. Using existing data, image information, camera parameters, and pose parameters, the effective pixels in the corresponding images are projected into three-dimensional space through reprojection to generate a point cloud model of the scene. b. Using the selected classification algorithm, the point cloud space is divided into multiple sub-regions. Then, the segmentation steps are repeated in each sub-region to generate a hierarchical scene division and record the coordinates of the center point of each division. c. For each valid pixel in each image, use the reprojection method described in (3)a to find the region division center that is closest to it in each level, label the category of the pixel in this level as the region, and store the relabeling result; (4) The relative coordinate regression module in the network model includes the following steps: a. Obtain the input feature vector of this module. Where H and W are the height and width of the image, respectively, and C1 represents the number of channels; b. Further feature extraction is performed through several convolutional layers to obtain the output feature vector. Where: C2 represents the number of channels; c. Feed x2 into a multi-level region labeling regression network. In each layer, obtain the region to which each pixel in the image belongs in the region division of that layer, represented by Y using a one-hot code. Lm ∈{0,1} H×W×K Where K represents the number of blocks in the upper layer of the network at this level, and at the highest level it represents the number of blocks in the entire scene. In the multi-layer network architecture, for each layer other than the highest level, the output of the upper-level network architecture is introduced into the lower layer through the FiLM-based adjustment layer. The original feature vector is adjusted according to f(x,w)=γ(w)⊙x+β(w), where: x is the input feature map of the current layer, ⊙ represents the Hadamard product, and γ(w) and β(w) are trained together during the network training process. d. After each of the above layers, each pixel in the input image can be associated with a sub-region in the corresponding scene that corresponds to that network layer. After the last layer is completed, each pixel in the input image has a corresponding category in each layer of the region division from coarse to fine. e. Based on the above steps, and combining the image pixel region labels output by each network layer, determine the region category label for each pixel in the input image. Then, group pixels of the same category according to their category. Finally, determine a threshold K based on the parameter settings for scene region hierarchical division and the size of the input image. min Count the number of pixels in each group after grouping, and keep the number ≥ K. min The groups are represented as follows: f. Set the number of graph attention mechanism layers L, and perform graph attention mechanism operations using the following formula: in, Right now This represents the entire graph attention operation process. This represents the feature representation of all pixels in the same group in step (4)e. Represent each layer of the graph attention mechanism; For each group after filtering in (4)e, for each pixel of each group, a graph attention mechanism operation is performed hierarchically, as follows: in, express The aggregated message result of the feature descriptors of all pixels in the graph for point i, where [.||.] represents the concatenation operation, in different graph attention layers. They are not the same. The calculation is performed using the method described in the following formula: Wherein, attention weight α ij The key-query similarity after Softmax is calculated using the method described below: Where q, k, and v are the query, key, and value in the graph attention mechanism operation, calculated using the method described below: The coefficients are not the same in different graph attention layers; h. After all groups have performed the above operations, summarize the features of all valid pixels to obtain the feature vector representation. Where C3 represents the number of channels, and through multiple convolutions, a predicted three-dimensional coordinate is finally output for each effective feature point; (5) The loss functions used in network training include: L=λ1L c +λ2L r +λ3L rep L c The classification loss is represented as follows: i refers to each valid pixel in the image; L r The coordinate regression loss is represented as follows: i refers to each valid pixel in the image; L rep The reprojection loss is expressed as follows: i refers to each valid pixel in the image, R refers to the rotation matrix in the true pose, t refers to the translation matrix in the true pose, π(.) refers to mapping the point in the world coordinates to the image coordinate system, and u refers to the true coordinates of the pixel in the image coordinate system. Where, λ1, λ2, λ3, λ c-r , λ c-s , λ rce , λ ce As parameters, in the early stages of training, there is a bias towards L. c During the middle and later stages, L was gradually increased. r With L rep The proportion of.

Images