Intelligent visual positioning method

Abstract

The invention discloses an intelligent visual positioning method which comprises the following steps: acquiring a color image feature sequence and a depth image feature sequence through a feature sequence extraction module, and taking the color image feature sequence and the depth image feature sequence as input; the multistage depth embedding Transform module is composed of a plurality of depth embedding units and a Transform layer, and each depth embedding unit takes a feature sequence as input and aims at outputting a feature sequence with enhanced spatial perception; the scene feature representation obtained by the multi-level depth embedded Transform module is sent to a prediction head to obtain a scene coordinate prediction result, a horizontal gradient operator and a vertical gradient operator act on the scene coordinate prediction result respectively, and a horizontal gradient and a vertical gradient of the scene coordinate prediction result are generated; the horizontal gradient operator and the vertical gradient operator also act on the depth image respectively to generate a horizontal gradient and a vertical gradient of the depth image; and training an intelligent visual positioning network by using depth-guided smooth constraint, regression loss and reprojection loss, and constructing a pose solver to perform pose sampling and pose refining.

Description

Technical Field

[0001] The present invention relates to the field of visual positioning, and in particular to an intelligent visual positioning method. Background Art

[0002] As one of the basic tasks of computer vision, visual positioning has been widely used in autonomous driving, simultaneous positioning and mapping, virtual reality and other fields. Visual positioning aims to estimate the 6-DoF pose of the camera in a known scene based on the query image, that is, the three-dimensional position coordinates and three-dimensional angle deflection of the camera in the world coordinate system. In recent years, relevant scholars have conducted in-depth research on visual positioning in indoor or outdoor street scenes and achieved excellent performance. However, there is little research on visual positioning methods for aerial scenes, which seriously limits the development of aerial systems that rely on navigation. Therefore, exploring an intelligent visual positioning method to achieve accurate positioning of the camera in aerial scenes has important research significance and application value.

[0003] Traditional visual positioning algorithms usually estimate camera pose based on manually designed features, which can be mainly divided into geometric structure-based methods and image retrieval-based methods. Among them, the geometric structure-based method first extracts feature points in the query image, then matches the extracted 2D feature points with the 3D coordinate points in the scene model, and finally solves the camera pose based on the obtained 2D-3D matching relationship. The image retrieval-based method first needs to retrieve the nearest neighbor image of the query image by matching the global features of the image, then match the 2D feature points of the two images, and finally solve the camera pose based on the obtained 2D-2D matching relationship. Although traditional methods have made great progress, they are limited by problems such as poor robustness and low generalization. In some complex scenes with drastic lighting changes and motion blur, positioning failures may occur.

[0004] In recent years, visual localization algorithms based on deep learning have gradually shown superior performance over traditional methods. Kendall et al. proposed a visual localization algorithm based on convolutional neural networks, which achieved visual localization by learning the mapping relationship from query images to camera poses through convolutional neural networks. However, this method only achieved good performance in indoor or outdoor street scenes, and the accuracy was poor when directly applied to complex aerial scenes. Recently, Yan et al. proposed a scalable aerial visual localization algorithm based on multimodal synthetic data, which improved the accuracy of visual localization in aerial scenes by learning cross-modal visual representations. However, existing methods only use convolutional neural networks to extract features of aerial images, and do not effectively explore the global contextual information of the scene. In addition, existing methods only use color images to extract features and lack clear spatial information. Therefore, they are not robust enough when dealing with visual artifacts that are widely present in aerial images, and it is difficult to meet the needs of practical applications. Summary of the invention

[0005] The present invention provides an intelligent visual positioning method, which aims to effectively explore the global context information of aerial images and improve the robustness of the network to the visual artifacts widely present in aerial images by mining the spatial information contained in the depth image, thereby achieving effective visual positioning of aerial scenes. See the following description for details:

[0006] An intelligent visual positioning method, the method comprising:

[0007] The feature sequence extraction module is used to obtain the color image feature sequence and the depth image feature sequence, and these are used as the input of the multi-level deep embedding Transformer module.

[0008] The multi-level deep embedding Transformer module is composed of multiple deep embedding units and Transformer layers, each deep embedding unit takes a feature sequence as input and aims to output a feature sequence with enhanced spatial perception;

[0009] The scene feature representation obtained by the multi-level depth embedding Transformer module is sent to the prediction head to obtain the scene coordinate prediction result. The horizontal gradient operator and the vertical gradient operator act on the scene coordinate prediction result respectively to generate the horizontal gradient and vertical gradient of the scene coordinate prediction result; the horizontal gradient operator and the vertical gradient operator also act on the depth image respectively to generate the horizontal gradient and vertical gradient of the depth image;

[0010] An intelligent visual localization network is trained using depth-guided smoothness constraints, regression loss, and reprojection loss, and a pose solver is constructed for pose sampling and pose refinement.

[0011] Each of the deep embedding units takes a feature sequence as input and aims to output a feature sequence with enhanced spatial perception as follows:

[0012] The input feature sequence and After being processed by convolutional layers, low-dimensional potential embeddings are generated. and

[0013] Embed the potential After a λ-smoothed spatial Softmax layer, a mask similar to spatial attention is generated, and the generated mask g fovea With latent embedding Dot product to generate spatially enhanced embedding

[0014] Potential Embedding and spatially enhanced embedding After weighted fusion, the feature sequence with enhanced spatial perception is obtained through the convolution layer. The calculation formula is expressed as:

[0015]

[0016] Among them, g 3 (·) represents a convolutional layer, and α and β represent learnable weight parameters.

[0017] The depth-guided smoothness constraint is: and Apply L1 penalty and use and The edge-aware term of is weighted to achieve this; as follows:

[0018]

[0019] Among them, d ij represents the depth value of the query image at position (i, j), s ij Represents the scene coordinate value predicted by the network at the query image at position (i, j).

[0020] Among them, the low-dimensional potential embedding and for:

[0021]

[0022] Among them, g 1 (·) and g 2 (·) are composed of one convolutional layer.

[0023] Wherein, the spatial enhancement embedding for:

[0024]

[0025] in, express The eigenvector at position (i, j) in l Represents a learnable weight parameter, representing a dot product operation.

[0026] The beneficial effects of the technical solution provided by the present invention are:

[0027] 1. The present invention uses the ability of Transformer to model long-distance dependencies to mine the global context information of aerial images. At the same time, considering that depth images have the characteristic of describing the spatial position of objects, the network is introduced with depth cues to explicitly perceive spatial information, thereby improving the robustness of the network to visual artifacts and further improving the performance of visual positioning of aerial scenes.

[0028] 2. The present invention designs a multi-level deep embedding Transformer module, which enhances the network's perception of the spatial structure of the aerial scene by adaptively fusing the deep image features with the color image features, thereby improving the network's robustness to visual artifacts. In addition, a depth-guided smoothing loss is designed, which, under the guidance of depth information, encourages the network to learn the geometric characteristics of piecewise smoothness of scene coordinates, thereby improving the prediction accuracy of scene coordinates.

[0029] 3. The present invention is experimentally verified on two aerial scene visual positioning data sets and can achieve performance that is superior to existing aerial scene visual positioning methods. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 The figure is a flow chart of an intelligent visual positioning method. DETAILED DESCRIPTION

[0031] In order to make the objectives, technical solutions and advantages of the present invention more clear, the embodiments of the present invention are described in further detail below.

[0032] The specific implementation of the intelligent visual positioning method in the present invention is described below by way of examples.

[0033] 1. Constructing feature sequence extraction module

[0034] For an input color image and its corresponding depth image, the embodiment of the present invention constructs a feature sequence extraction module to obtain a color image feature sequence T rgb and the deep image feature sequence T depth Specifically, the input color image and depth image are firstly processed through the convolutional layer to obtain the color image features F r and deep image features F d Then, the color image feature F r and deep image features F d The color image features and depth image features with position information are transformed into feature sequences T by flattening operations. rgb and T depth .

[0035] The above process is expressed as follows:

[0036] T rgb =[P 1 +F r,1 ,P 2 +F r,2 ,...,P i +F r,i ]

[0037] T depth =[P 1 +F d,1 ,P 2 +F d,2 ,...,P j +F d,j ]

[0038] Wherein, i represents the number of pixels in the color image feature map, j represents the number of pixels in the depth image feature map, preferably i=j=1350.

[0039] 2. Designing a Multi-Level Deep Embedding Transformer Module

[0040] Considering that color images tend to describe the visual appearance of objects, while depth images can provide explicit spatial information of the entire scene, the embodiment of the present invention designs a multi-level deep embedding Transformer module to fuse the information of depth images and color images and improve the robustness of the network to visual artifacts. Specifically, this module transforms the color image feature sequence T rgb and the deep image feature sequence T depth As input, deep image features are adaptively embedded into multiple levels of Transformer to enhance the network's perception of the spatial structure of aerial scenes.

[0041] The designed multi-level deep embedding Transformer module consists of multiple deep embedding units and Transformer layers. l Feature Sequence and As input, it aims to output a feature sequence with enhanced spatial perception Transformer layer E l Feature Sequence and The sum is taken as input, aiming to output a globally perceptually enhanced feature sequence

[0042] Embed the unit M with the lth depth l For example, first input the feature sequence and After being processed by convolutional layers, low-dimensional potential embeddings are generated. and The calculation formula is expressed as:

[0043]

[0044] Among them, g 1 (·) and g 2(·) are composed of one convolutional layer.

[0045] Then, embed the potential After a λ-smoothed spatial Softmax layer, a mask g similar to spatial attention is generated fovea Then, the generated mask g fovea With latent embedding Dot product to generate spatially enhanced embedding The above process can be expressed as:

[0046]

[0047] in, express The eigenvector at position (i, j) in l Represents a learnable weight parameter, representing a dot product operation.

[0048] Finally, to adaptively fuse the depth information with the spatially enhanced embedding, the latent embedding and spatially enhanced embedding After weighted fusion, a convolutional layer is used to obtain a feature sequence with enhanced spatial perception. The calculation formula is expressed as:

[0049]

[0050] Among them, g 3 (·) represents a convolutional layer, and α and β represent learnable weight parameters. Preferably, the multi-level deep embedding Transformer module consists of 12 deep embedding units and 12 Transformer layers.

[0051] Take the lth Transformer layer E l For example, its input is the output of the lth deep embedding unit and the output of the l-1th Transformer layer The sum of the ... The calculation formula is as follows:

[0052]

[0053] Among them, LN(·) represents layer normalization, MSA(·) represents multi-head self-attention, and MLP(·) represents multi-layer perceptron, which are used to enhance the expressiveness of the model.

[0054] 3. Designing Depth-Guided Smooth Constraints

[0055] The embodiment of the present invention designs a depth-guided smoothness constraint, which guides the network to obtain piecewise-smooth scene coordinate prediction results by utilizing the piecewise-smooth geometric characteristics of the depth image, thereby improving the prediction accuracy of the scene coordinates.

[0056] First, the scene features obtained by multi-level deep embedding Transformer module Indicates that the scene coordinate prediction result s is sent to the prediction head pred , the calculation formula is expressed as:

[0057]

[0058] Here, H(·) represents the prediction head, which consists of 4 convolutional layers and a rectified linear unit.

[0059] Then, the horizontal gradient operator and the vertical gradient operator act on the scene coordinate prediction result s respectively. pred , generating the horizontal gradient of the scene coordinate prediction result and vertical gradient At the same time, the horizontal gradient operator and the vertical gradient operator also act on the depth image to generate the horizontal gradient of the depth image. and vertical gradient The designed depth-guided smoothness constraint is obtained by and Apply L1 penalty and use and This is achieved by weighting the penalty with an edge-aware term.

[0060] The above process is expressed by the following formula:

[0061]

[0062] Among them, d ij represents the depth value of the query image at position (i, j), s ij Represents the scene coordinate value predicted by the network at the query image at position (i, j).

[0063] 4. Training Intelligent Visual Positioning Network

[0064] In order to obtain accurate scene coordinate prediction results, the embodiment of the present invention uses multiple loss functions to train the network. Specifically, a depth-guided smoothness constraint is designed to encourage the network to learn the geometric properties of piecewise smoothness of scene coordinates. At the same time, in order to effectively supervise the intelligent visual positioning network, the embodiment of the present invention adopts a regression loss L regress , the formula is as follows:

[0065]

[0066] in, represents the true value of the scene coordinates corresponding to the query image at position (i, j), s ij represents the scene coordinate value predicted by the network at the query image at position (i, j), u ij Represents the uncertainty of the scene coordinate prediction result corresponding to the query image at position (i, j).

[0067] In addition, the embodiment of the present invention also adopts the reprojection loss L reproject The prediction accuracy of the constrained scene coordinates is expressed as follows:

[0068]

[0069] Among them, C represents the intrinsic parameter matrix of the camera, h represents the true value of the camera pose, and P s represents the set of points in the predicted scene coordinates whose reprojection error is less than the set threshold, P t represents the set of points in the predicted scene coordinates whose reprojection error is greater than the set threshold, p ij Represents the position coordinates of the query image at (i, j).

[0070] Finally, the combined depth-guided smoothness constraint L smooth , regression loss L regress and the reprojection loss L reproject The total loss L total The representation is as follows:

[0071] L total =L regress +L reproject +λ s L smooth

[0072] Among them, λ s The weight representing the depth-guided smoothness constraint is set to 0.1 in the embodiment of the present invention.

[0073] The embodiment of the present invention uses depth-guided smoothness constraints, regression loss, and reprojection loss to train an intelligent visual positioning network, which is divided into two stages. In the first stage, the neural network is trained for 150 epochs using synthetic data to obtain an initial intelligent visual positioning model; in the second stage, the initial model is fine-tuned to four scenes using real-synthetic data, and finally an intelligent visual positioning model that can test the corresponding four different scenes is obtained.

[0074] 5. Build a pose solver

[0075] After the intelligent visual positioning network is trained, in order to further obtain the camera's pose, the embodiment of the present invention constructs a pose solver for pose sampling and pose refinement. Specifically, the pose solver consists of a PnP operator, which optimizes the camera's pose by minimizing the reprojection error of the scene coordinates.

[0076] In the embodiment of the present invention, the scene coordinates output by the intelligent visual positioning network and the camera's intrinsic parameter matrix are input into the PnP operator to obtain the camera's 6-DoF pose. The formula for the reprojection error is as follows:

[0077]

[0078] Where r(·) represents the reprojection error, p i represents the two-dimensional image coordinates associated with pixel i, K represents the intrinsic parameter matrix of the camera, represents the camera pose hypothesis, y i represents the scene coordinates associated with pixel i.

[0079] Those skilled in the art will appreciate that the accompanying drawing is only a schematic diagram of a preferred embodiment, and the serial numbers of the embodiments of the present invention are only for description and do not represent the advantages or disadvantages of the embodiments.

[0080] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. An intelligent visual positioning method, characterized in that: The method comprises: The feature sequence extraction module is used to obtain the color image feature sequence and the depth image feature sequence, and these are used as the input of the multi-level deep embedding Transformer module. The multi-level deep embedding Transformer module is composed of multiple deep embedding units and Transformer layers, each deep embedding unit takes a feature sequence as input and aims to output a feature sequence with enhanced spatial perception; The scene feature representation obtained by the multi-level depth embedding Transformer module is sent to the prediction head to obtain the scene coordinate prediction result. The horizontal gradient operator and the vertical gradient operator act on the scene coordinate prediction result respectively to generate the horizontal gradient and vertical gradient of the scene coordinate prediction result; the horizontal gradient operator and the vertical gradient operator also act on the depth image respectively to generate the horizontal gradient and vertical gradient of the depth image; An intelligent visual localization network is trained using depth-guided smoothness constraints, regression loss, and reprojection loss, and a pose solver is constructed for pose sampling and pose refinement.

2. The intelligent visual positioning method according to claim 1, characterized in that: Each of the deep embedding units takes a feature sequence as input and aims to output a feature sequence with spatial perception enhancement as: The input feature sequence and After being processed by convolutional layers, low-dimensional potential embeddings are generated. and Embed the potential After a λ-smoothed spatial Softmax layer, a mask similar to spatial attention is generated, and the generated mask g fovea With latent embedding Dot product to generate spatially enhanced embedding Potential Embedding and spatially enhanced embedding After weighted fusion, the feature sequence with enhanced spatial perception is obtained through the convolution layer. The calculation formula is expressed as: Among them, g3(·) represents a convolutional layer, and α and β represent learnable weight parameters.

3. The intelligent visual positioning method according to claim 1, characterized in that: The depth-guided smoothness constraint is: and Apply L1 penalty and use and The edge-aware term of is weighted to achieve this; as follows: Among them, d ij represents the depth value of the query image at position (i, j), s ij Represents the scene coordinate value predicted by the network at the query image at position (i, j).

4. The intelligent visual positioning method according to claim 2, characterized in that: The low-dimensional latent embedding and for: Among them, g1(·) and g2(·) are both composed of a convolutional layer.

5. The intelligent visual positioning method according to claim 2, characterized in that: The spatially enhanced embedding for: in, express The eigenvector at position (i, j) in l Represents a learnable weight parameter, representing a dot product operation.

Images