Visual relocation method combining graph attention and region classification

Abstract

The invention discloses a visual relocation method combining graph attention and region classification, and relates to the field of visual relocation in computer vision. According to the method, a scene region classification method is adopted, modeling and hierarchical classification are carried out on a space scene, data are re-labeled, hierarchical scene coordinate region regression and relative coordinate prediction methods are adopted, a re-positioning problem is converted into a classification problem from coarse to fine, and the re-positioning problem is classified. Finally, positioning is concentrated in a small area, and the repositioning problem under a large scene scale can be improved; by applying a graph attention mechanism and combining a classification result, the pixel points in the image can better acquire information of other pixel points in the same area, and more excellent feature representation is generated. According to the invention, the problem of visual relocation in a real scene can be solved.

Description

Technical Field

[0001] The present invention relates to a visual relocalization technology in the field of computer vision, and in particular to a visual relocalization method based on graph attention and region classification. Background Art

[0002] Relocalization technology aims to resolve the 6-DoF (degrees of freedom) pose corresponding to the query input in a known scene, and achieve accurate positioning of the shooting direction and angle. It is a key module in the SLAM (simultaneous localization and mapping) system. Some applications, such as autonomous motion robots, autonomous driving, and AR (augmented reality) and VR (virtual reality) technologies, have high requirements for positioning reliability and accuracy. Relocalization technology has become a basic and important component of these applications. Depending on the sensor, relocalization can be divided into many categories. Among them, vision-based relocalization, also known as visual relocalization, has been widely used due to its simple and easy-to-obtain sensors, rich information, and can be used for multiple visual tasks at the same time. It has become a research hotspot in the field of computer vision.

[0003] Visual relocalization methods have gone through a development process from geometric structure methods to image retrieval methods and then to deep learning methods. The current visual relocalization process can be roughly divided into three steps: scene model construction, image scene matching, and camera pose solution. The geometric structure-based method pre-stores the 3D (three-dimensional) model of the scene, and then establishes a corresponding relationship by finding the visual feature matching of the spatial points of the current query image and the spatial points in the 3D scene model, so as to solve the pose through the PnP (Perspective-n-Point) algorithm based on RANSAC (Random Sample Consensus). However, this method requires the storage of a large 3D model, and in some cases it may not be able to find key points or matching descriptors, such as motion blur and textureless areas. In addition, the manually extracted features are not highly adaptable across scenes and require a large amount of calculation. The image retrieval-based method stores images of known poses into a database. For a given query image, an image similar to the query image is searched in the known image library. The pose of the query image can be obtained through the closest image, or a more accurate estimate can be obtained by calculating the relative pose between the two on this basis. Since there are usually certain differences between the query image and the database images, they are limited in accurately estimating the camera pose.

[0004] The absolute pose regression method is a relatively straightforward and simple method for visual relocalization using deep learning, which aims to maintain both efficiency and scalability. It uses known images as training sets and encodes the relationship between image content and camera pose in its parameters, thereby learning the relationship between image content and pose. At query time, the network directly regresses the 6-DoF pose corresponding to the image. However, current absolute pose regression methods are more similar to image retrieval methods and have failed to surpass image retrieval-based methods. Scene coordinate regression methods first predict the 3D coordinates corresponding to the query image pixels to construct 2D-3D correspondences, and then calculate the pose through a RANSAC-based PnP algorithm similar to geometry-based methods. With the development of deep learning, people began to use deep learning to regress scene coordinates and gradually used simpler and more readily available RGB images as query images. Scene coordinate regression-based methods provide a better trade-off between accuracy and computational efficiency and have become a more popular class of methods in visual relocalization methods. However, in larger scenes, the ambiguity of visually similar local image patches cannot be resolved by a limited receptive field. Simply using a larger receptive field size is prone to overfitting. In addition, the traditional CNN (Convolutional Neural Networks) based network structure is relatively simple and cannot learn the relationship between nodes in a targeted manner, which may lead to a lack of understanding of the global context and the uniqueness of multiple feature clusters.

[0005] In summary, in order to adapt to scenes of different scales and the concentration of some visual features, it is necessary to propose an effective relocalization method. Summary of the invention

[0006] In view of the above problems, the purpose of the present invention is to provide a visual relocalization method combining image attention and region classification.

[0007] A scene coordinate regression network based on scene area division from coarse to fine is used, and after the pixels in the image are classified, a graph attention mechanism is applied to the pixels of the same category to more fully learn the features between the pixels of the same category. The technical solution to implement the present invention includes two processes: network training and reasoning. The specific technical solution is as follows:

[0008] Step 1: Create a data set using existing or photographic methods, and perform preprocessing such as data annotation and division to obtain input images used for network training;

[0009] Step 2: Preliminarily extract image features through a series of convolution operations to obtain the features of each pixel in the image;

[0010] Step 3: Obtain the region corresponding to each pixel through a multi-level pixel-level scene region regression module;

[0011] Step 4: Optimize the features corresponding to the pixels in the image through operations between pixels in the same region based on graph attention;

[0012] Step 5: Through the multi-layer perceptron layer, the features corresponding to each pixel are converted into corresponding 3D coordinates;

[0013] Step 6: Calculate the 6-DoF pose of the input image using the RANSAC-based PnP algorithm.

[0014] Furthermore, the data set mentioned in step 1 includes an image of the corresponding scene obtained by shooting or the like, the 6-DoF pose corresponding to the image, and the area number of the point in the three-dimensional scene corresponding to the pixel point in the image obtained by calculation.

[0015] Furthermore, the training set content in the data set mentioned in step 1 can be enriched through data enhancement methods such as cropping, rotation, and brightness adjustment to better adapt to small-scale data sets.

[0016] Furthermore, the image feature extraction module mentioned in step 2 is composed of any convolutional layer. Given an input image, its output feature map is expressed as:

[0017] X1∈R H×W×C

[0018] Where H and W are the height and width of the image, respectively, and C is the number of channels. The image can also be downsampled before this process to reduce the memory requirements. If the above process is performed, the first two dimensions will change accordingly.

[0019] Furthermore, the multi-level pixel-level scene area regression module mentioned in step 3 has the following specific process:

[0020] Step 3.1: During the scene preprocessing process, the total number of layers for classifying scene areas in the network model is set, and then the scene areas are classified from coarse to fine levels according to the levels, where the classification of the next level is performed in the sub-area obtained in the previous level, and there is no overlap in all sub-areas of the same level. For example, the total number of levels is selected as 2, and 10 categories are selected in each level. Then, the entire scene space area is divided into 10 sub-areas in the first layer, and each sub-area obtained in the first layer is further divided into 10 sub-areas in the second layer, so there are a total of 10×10 sub-areas in this level.

[0021] Step 3.2: For each layer in step 1, set any convolutional layer and residual connection, and use the output X1 obtained in step 2 as input. The dimension of the feature map output by the last convolutional layer is H×W×K, where K represents the number of blocks that classify the area of ​​the previous layer at this level. At the highest level, it represents the number of blocks that classify the entire scene. Then, use one-hot encoding to represent the pixel area category in the image at this level as:

[0022] Y Lm ∈{0, 1} H×W×K

[0023] Where m∈{1, 2, ..., T}, T is the total number of layers for classifying scene areas in the network model. After this step, each pixel can be classified into a category. If this level is not the highest level, it is classified into a subcategory under its parent category.

[0024] Step 3.3: Except for the highest level, each layer of scene region regression module receives information input from the pixel-level region classification regression of the previous layer. For the training process, the next layer receives the real value label input from the previous layer, and for the inference process, the next layer receives the predicted value label input from the previous layer. This process is completed using the adjustment layer based on FiLM (Feature-wise Linear Modulation), and the specific method is as follows:

[0025] Step 3.3.1: For the training process, the input is the true label of each pixel. For the prediction process, the input is the predicted value of each pixel at the previous level. Through any convolutional layer, the feature map w is obtained.

[0026] Step 3.3.2: Perform linear adjustment on the input feature map x of the current layer, which is specifically expressed as follows:

[0027] f(x,w)=γ(w)⊙x+β(w)

[0028] Where ⊙ represents the Hadamard product, γ(w) and β(w) are trained together during the network training process. In each layer of the network, one or more FiLM-based adjustment layers can be applied.

[0029] After passing through each layer in step 3, for each pixel in the input image, it can correspond to a sub-region in the corresponding scene corresponding to the network level. In particular, after completing the last layer of step 3, for each pixel in the input image, there is a corresponding category in each layer of the area division from coarse to fine.

[0030] Furthermore, the specific process of the operation between pixels in the same region based on graph attention mentioned in step 4 is as follows:

[0031] Step 4.1: After step 3, each pixel in the input image has obtained the corresponding sub-region category prediction at different levels, indicating the sub-region to which it belongs in the corresponding level after the scene is segmented into levels.

[0032] Step 4.2: In the process of pixel-to-pixel operations in the same region based on graph attention, the category of each pixel in the input image is based on the true label during the training process, and the predicted label obtained after step 4.1 during the inference process.

[0033] Step 4.3: After determining the regional category label of each pixel in the input image through step 4.2, group the pixels of the same category into a group according to the category described, and determine a threshold K based on the parameter settings when dividing the scene region hierarchy and the size of the input image. min .

[0034] Step 4.4: Count the number of pixels in each group after grouping in 4.3, and keep the number ≥ K min The group is expressed as:

[0035]

[0036] Where i∈{1, 2, ..., T}, T is the number of elements ≥ K min The number of groups.

[0037] Step 4.5: Set the number of graph attention mechanism layers L, then the described graph attention operation is expressed as follows:

[0038]

[0039] in, Right now Represents the entire graph attention operation process, represents the feature representation corresponding to all pixels in the same group in step 4.4, Represents each layer of the graph attention mechanism.

[0040] For each group after screening mentioned in step 4.4, perform graph attention operation. The specific process is as follows:

[0041] Step 4.5.1: Select the current group

[0042] Step 4.5.2: For Each layer in It can be expressed as: For the current group Each element point in i The corresponding feature representation (l)d i , the next layer of features is calculated by the following method (l+1) d i :

[0043]

[0044] in, express The aggregated message result of the feature descriptors of all pixels in for point i, [.||.] represents the concatenation operation. Not the same. The calculation is performed by the following method:

[0045]

[0046] Among them, the attention weight α ij is the key-query similarity after Softmax, which is calculated by the following formula:

[0047]

[0048] Among them, q, k, v are the query, key, and value in the graph attention mechanism operation, which are calculated by the following method:

[0049]

[0050] The coefficients are not the same in different graph attention layers. In the process of calculating graph attention, a multi-head attention mechanism can be used to improve the calculation.

[0051] Step 4.5.3: Calculate the next group. After all groups have completed the above calculation, the graph attention mechanism module is considered to have completed one time.

[0052] Furthermore, the conversion of the features corresponding to each pixel point into the corresponding 3D coordinates mentioned in step 5 is performed by regressing a residual value for each pixel point, rather than directly regressing the 3D coordinates corresponding to the pixel point. The residual value represents the residual between the actual 3D coordinates of the pixel point and the 3D coordinates of the center position of the finest sub-region regressed at the bottom layer of step 3. Therefore, the actual 3D position coordinates corresponding to the pixel point should be obtained by the results of each layer in step 3 and the above residual value.

[0053] The beneficial effects brought by adopting the above technical solution are:

[0054] The present invention solves some existing problems in the field by proposing a novel and effective visual relocalization method that combines graph attention and region classification. The method in the present invention utilizes the scene region segmentation method to design the network from global to local, so that the network can browse the entire scene area from coarse to fine, and contains global information through regional classification labels, so that more global information can be included in the fine-grained layer, while avoiding the ambiguity of local appearance during training, and having a larger receptive field at the fine-grained layer during reasoning. A category-based graph attention mechanism is proposed, which enables the pixels in the image to better obtain information about other points in a similar area to it, and produce better feature representations compared to the traditional use of convolutional networks. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] Figure 1 A flowchart of the present invention performing network training and predictive reasoning in a new scenario;

[0056] Figure 2 It is a schematic diagram of the overall network structure of the present invention;

[0057] Figure 3 This is a schematic diagram of a scene area classification module of the present invention;

[0058] Figure 4 Schematic diagram of the relative coordinate regression module and graph attention module of the present invention. DETAILED DESCRIPTION

[0059] A typical embodiment of a visual relocalization method combining graph attention and region classification of the present invention is described in detail below to provide a further specific description of the method.

[0060] In the implementation process of the present invention, the applicable scene for relocation should be selected first, and the images of various places inside the scene and the corresponding pose of each image should be obtained through tools such as depth cameras. In this embodiment, the chess scene in the existing data set 7Scenes is selected for use. The data set provides six groups of images, each group of images corresponds to a video sequence shot in the scene. The number of images in each group is 1000. Groups 1, 2, 4, and 6 are selected as training sets, and groups 3 and 5 are selected as test sets. Each image provides a corresponding depth image and 6-DoF pose information.

[0061] In this embodiment, the total number of levels of scene area classification is set to 2. Specifically, each level is set to be divided into 25 sub-areas, so the entire scene area is divided into 25 first-level sub-areas, each first-level sub-area is divided into 25 second-level sub-areas, a total of 625 second-level sub-areas. Then, the image is re-labeled according to the divided areas so that each pixel corresponds to a first-level sub-area label and a second-level sub-area label. The specific method is as follows:

[0062] a.1. Based on the training set images, camera parameters and pose information, the pixel points in each image are projected into the world coordinate system in turn through the conversion from the image coordinate system to the camera coordinate system and then to the world coordinate system to form a point cloud representation of the scene. In order to avoid the point cloud having too many points due to the large number of data sets, which makes it inconvenient to divide and re-label the area, the point cloud is downsampled in this process. It is set to downsample the current point cloud model once every 35 images, and the ratio is set to 0.02.

[0063] a.2. After obtaining the point cloud representation of the scene, the scene is divided using the K-Means clustering algorithm. First, the K-Means clustering algorithm is executed on the point cloud representation of the entire scene to obtain 25 subclasses, corresponding to 25 first-level sub-regions. Then, the K-Means clustering algorithm is executed again for each first-level sub-region to obtain 25 second-level sub-regions. In the process, the coordinates of the center points of each first-level sub-region and second-level sub-region are recorded and saved.

[0064] a.3. For each image, the three-dimensional coordinates of each pixel in the world coordinate system are obtained through the process in a.1 above. Then, for each pixel, find the center point with the closest Euclidean distance to the center point of the first-level sub-region recorded in a.2. The category corresponding to the center point is recorded as the first-level sub-region category of the pixel. Similarly, obtain the second-level sub-region category of the pixel. At this point, the re-labeling process is completed.

[0065] The image size of the dataset selected in this embodiment is 480×640. In order to optimize the network operation efficiency and reduce the memory requirement, the image is uniformly downsampled by ×1 / 8 before it is input into the network. Therefore, the input image size of the network end in this embodiment is 60×80, and the dimension is 3. The pixel points whose corresponding depth information is 0 in the depth image are regarded as invalid points, and a mask is set for them during the training data processing and recorded.

[0066] The feature extraction module in this embodiment is composed of a series of convolutional layers, and the number of channels of the final convolutional layer is set to 512. After passing through this module, the input image with a dimension of 60×80×3 is represented as a feature map with a dimension of 60×80×512.

[0067] The scene region classification module in this embodiment is composed of a series of convolutional layers with residual connections and a FiLM-based adjustment layer. The total number of layers for scene region classification is 2, and the number of layers for scene coordinate region classification module is also 2. In the first layer, the number of channels of the final convolutional layer is set to 25, thereby outputting a feature map with a dimension of 60×80×25. Through one-hot encoding, we get the form {0, 1} 60×80×25The sub-region representation corresponds to each pixel of the input image to a first-level sub-region classification. In the second layer, an adjustment layer based on FiLM is added to adjust the feature map with the formula f(x, w) = γ(w)⊙x+β(w). In the training process, w is the real label y r , in the reasoning process The γ(·) and β(·) are composed of a series of 1×1 convolutions. In this layer, two FiLM-based adjustment layers are applied. The final second layer outputs a feature map with a dimension of 60×80×25. Through one-hot encoding, we get the form {0, 1} 60×80×25 The sub-region representation corresponds to each pixel of the input image to a secondary sub-region classification.

[0068] The front part of the relative coordinate regression module in this embodiment is similar to the second layer of the scene area classification module, which consists of a series of convolutional layers containing residual connections and FiLM-based adjustment layers. The FiLM-based adjustment layer uses two levels of scene area classification information. A feature map with a dimension of 60×80×64 is obtained, and then enters the graph attention module. In this module, the number of graph attention mechanism layers L is selected as 5. Each point in the input image corresponds to one of the 625 secondary sub-regions according to the labels of its primary sub-region and secondary sub-region, and is grouped. Select the minimum number threshold K min For the selected groups, the inter-group graph attention operation is performed according to the method described above to update the corresponding pixel features. After that, two 1×1 convolutions and an MLP layer are used to output a vector with a dimension of 60×80×3. Combined with previous and As the output of the entire network, the scene space coordinates of each valid point behind the mask in the input image are predicted by and The center coordinates of the sub-regions represented by the joint are summed up and obtained.

[0069] In this embodiment, the loss function is set as follows:

[0070] L=λ1L c +λ2L r +λ s L rep

[0071] L c Represents the classification loss, which is specifically expressed as follows:

[0072]

[0073]

[0074] i refers to each valid pixel in the image.

[0075] L r Represents the coordinate regression loss, which is specifically expressed as follows:

[0076]

[0077] i refers to each valid pixel in the image.

[0078] L rep Represents the reprojection loss, which is specifically expressed as follows:

[0079]

[0080] i refers to each valid pixel in the image. R refers to the rotation matrix in the regressed pose, t refers to the translation matrix in the regressed pose, π(.) refers to mapping the point in the world coordinate system to the image coordinate system, and u refers to the real coordinate of the pixel in the image coordinate system.

[0081] In this embodiment, λ1=1, λ2=10, and λ3 gradually increases from 0 to 0.1 after the initial 10 rounds of training. c-r =λ c-s =1,λ rce =1,λ ce =0.1.

[0082] During the training process of this embodiment, the training device is NVIDIA RTX3090 GPU (24g). The number of iterations of a single scene is set to 300K, the Adam optimizer is used, the batch-size is set to 1, and the learning rate is set to 1e-4. The training data is expanded by applying random rotation (±30°), random horizontal and vertical shifts (20%), random scaling (0.7-1.5), and brightness adjustment (±20).

[0083] The model evaluation method of this embodiment is mainly composed of three indicators: translation error (unit: cm), angle error (unit: °), and accuracy. Translation error and angle error refer to the difference between the test sample and the true label in translation and angle. Accuracy refers to the proportion of samples in a group of test samples that simultaneously meet the translation error ≤ 5 cm and the angle error ≤ 5 °.

[0084] The above is a specific implementation example of a visual relocalization method combining image attention and region classification proposed by the present invention, which explains the principle and main implementation methods of the present invention. It is necessary to point out that the technical personnel in this field may make some non-essential improvements and adjustments to the method based on the above content, which still fall within the scope of protection of the present invention.

Claims

1. A visual relocalization method combining graph attention and region classification, characterized in that: The following steps are involved: a. Model the dataset processing, divide the scene area into multi-level sub-region spaces, and re-label the pixels in the dataset image with multi-level sub-region labels; b. Use the training data to train the network model and obtain the corresponding network parameter data for the scenario; c. According to the existing network model, input the image and obtain the spatial coordinate relationship corresponding to each pixel in the input image; d. Calculate the 6-DoF pose of the input image using the RANSAC-based PnP algorithm.

2. A visual relocalization method combining graph attention and region classification according to claim 1, characterized in that: The network model includes: a. A feature extraction module, consisting of several convolutional layers, represents the input image as a feature map; b. A scene region regression module that predicts each valid pixel in the input image to its region at each level; c. A relative coordinate regression module uses the above-mentioned regional prediction information to perform a graph attention mechanism operation on pixels of the same category in the input image, combined with a convolution operation, and finally generates the three-dimensional spatial coordinates corresponding to the valid pixels in the image.

3. A visual relocalization method combining graph attention and region classification according to claim 1, characterized in that: The processing modeling and scene area segmentation of the data set include: a. For the existing data, use its image information, camera parameters, posture parameters and other information to project the effective pixels in the corresponding image into three-dimensional space through reprojection to generate a point cloud model of the scene; b. Using the selected classification algorithm, the point cloud space is divided into multiple sub-regions, and then the segmentation steps are repeated in each sub-region to generate a hierarchical scene partition, and the coordinates of each partition center point are recorded; c. For each valid pixel of each image, use the reprojection method described in 3.a to find the closest region division center in each level, label the category of the pixel at this level as the region, and store the relabeling result.

4. A visual relocalization method combining graph attention and region classification according to claims 1-2, characterized in that: The relative coordinate regression module comprises: a. Get the input feature vector of the module b. Through several convolutional layers, further extract features and obtain the output feature vector c. Through the FiLM-based adjustment layer, the original feature vector is adjusted according to f(x,w)=γ(w)ηx+β(w) to obtain the output feature vector d. Construct the encoding layer based on the modules described in 4.b and 4.c above to generate feature representation for the graph attention module e. According to 2.b above, after determining the regional category label of each pixel in the input image, the pixels of the same category are grouped according to the category, and a threshold K is determined according to the parameter settings when dividing the scene area hierarchy and the size of the input image. min , count the number of pixels in each group after grouping, and keep the number ≥ K min The group is expressed as: f. Set the number of graph attention mechanism layers L and use the following formula to perform graph attention mechanism operation: in, Right now Represents the entire graph attention operation process, represents the feature representation corresponding to all pixels in the same group in step 4.e, Represents each layer of the graph attention mechanism; Furthermore, for each group mentioned in 4.e, further, for each pixel point of each group, the graph attention mechanism operation is performed hierarchically, which is expressed as follows: in, express The aggregated message result of the feature descriptors of all pixels in point i, [.||.] represents the concatenation operation, in different graph attention layers are not the same. The calculation is performed by the following method: Among them, the attention weight α ij is the key-query similarity after Softmax, which is calculated by the following formula: Among them, q, k, v are the query, key, and value in the graph attention mechanism operation, which are calculated by the following method: The coefficients are not the same in different graph attention layers; h. After all groups have performed the above operations, all valid pixel features are summarized to obtain the feature vector representation Then, a decoding layer is constructed by combining the modules described in 4.b and 4.c above, and finally a predicted three-dimensional coordinate is output for each valid feature point.

5. A visual relocalization method combining graph attention and region classification according to claims 1-2, characterized in that: The loss functions used in the network training process include: L=λ1L c +λ2L r +λ3L rep L c Represents the classification loss, which is specifically expressed as follows: i refers to each valid pixel in the image; L r Represents the coordinate regression loss, which is specifically expressed as follows: i refers to each valid pixel in the image; L rep Represents the reprojection loss, which is specifically expressed as follows: i refers to each valid pixel in the image, R refers to the rotation matrix in the regressed pose, t refers to the translation matrix in the regressed pose, π(.) refers to mapping the point in the world coordinate system to the image coordinate system, and u refers to the real coordinate of the pixel in the image coordinate system; Among them, λ1, λ2, λ3, λ c-r ,λ c-s ,λ rce ,λ ce is a parameter, and in the early stage of the training process, it tends to be L c , in the middle and late stages, gradually increase L r With L rep proportion.

Images