Visual inertial positioning method and device using graph attention network to match points and lines

Abstract

The present invention proposes a visual-inertial positioning method and device that uses a graph attention network to match points and lines, belonging to the field of positioning technology. The method includes: using a monocular camera in a visual-inertial positioning system to capture the current moment image and extract the corresponding point features and line features, and then using a preset point-line matching network containing a graph attention mechanism to obtain the matching relationship between the point features and line features of the current moment image and the previous moment image; detecting the classification results of the main structural direction and line features of the current moment image, combining the measurement results of the inertial measurement unit, optimizing the position of the monocular camera, and obtaining the current positioning result of the visual-inertial positioning system. The present invention uses a graph attention network to perform point and line feature matching, which can improve the accuracy and robustness of visual-inertial positioning in situations of changing lighting, rapid motion, and lack of texture, and has good engineering application value.

Description

Technical Field

[0001] The present invention belongs to the field of positioning technology, and specifically relates to a visual inertial positioning method and device using a graph attention network to match points and lines. Background Art

[0002] Visual-inertial positioning is an autonomous positioning technology based on cameras and inertial measurement units (IMUs). By fusing visual information provided by cameras with motion information provided by IMUs, this technology can accurately and quickly estimate the device's position. This technology aims to solve the problem of estimating the position of unmanned mobile platforms in unknown environments when external positioning technologies such as GPS and UWB (ultra-wideband) are unavailable. This technology has been widely used in fields such as autonomous driving, drones, and augmented reality, achieving excellent positioning results in various indoor and outdoor scenarios.

[0003] Compared to other positioning solutions that don't rely on external positioning devices, visual-inertial positioning technology offers unique advantages in sensor selection. Compared to lidar-based positioning solutions, the combination of an IMU and a camera is low-cost, lightweight, and easy to deploy on small mobile platforms. Compared to wheel tachometer-based positioning solutions, the addition of a camera allows the positioning system to detect previously visited areas and correct positioning errors. It can also be used on non-ground wheeled robot platforms, such as AR / VR (augmented reality / virtual reality) devices and drones.

[0004] Although visual-inertial positioning technology can achieve good positioning results in conditions with good lighting, rich textures, and low-speed motion, it still faces large positioning errors and poor robustness in scenes with fluctuating lighting, poor textures, and rapid motion. This can even lead to diverging positioning results. To improve the robustness and accuracy of visual-inertial positioning, many research efforts have focused on the use of line features, combining point and line features to improve positioning accuracy in scenes with weak textures. In 2019, Zou Danping et al. from Shanghai Jiao Tong University proposed a point-line visual-inertial positioning method using the Atlanta World Hypothesis. They tested this method in indoor environments and demonstrated high positioning accuracy in scenes with weak textures. However, this method still uses traditional optical flow tracking for point-line feature matching, failing to consider the impact of lighting variations and rapid motion on the algorithm's positioning accuracy. In the face of significant inter-frame lighting and viewpoint variations, the robustness of traditional point-line feature matching algorithms needs to be further improved to further enhance the positioning accuracy and robustness of visual-inertial positioning technology.

[0005] In recent years, deep learning technology has made rapid progress in the field of computer vision, demonstrating enormous application potential in many fields, including object detection, action recognition, and semantic segmentation. A growing number of researchers are also using deep learning to solve visual-inertial localization problems. In 2020, researchers including Paul-Edouard Sarlin of the Swiss Federal Institute of Technology in Zurich used a graph attention network to match point features between two images. This method can accurately match point features under large differences in lighting and viewing angles. However, this method is not suitable for matching line features and does not integrate well with existing visual-inertial localization algorithms based on point and line features. Further research and solutions are needed to better integrate deep learning technology with visual-inertial localization algorithms. Summary of the Invention

[0006] This invention aims to overcome the shortcomings of existing technologies by proposing a visual-inertial localization method and device using a graph attention network to match points and lines. By using a graph attention network to perform point and line feature matching, this invention improves the accuracy and robustness of visual-inertial localization in situations with varying lighting, rapid motion, and texture scarcity, and has promising engineering applications.

[0007] The first embodiment of the present invention proposes a visual inertial positioning method for matching points and lines using a graph attention network, comprising:

[0008] During the operation of the visual inertial positioning system, the monocular camera in the system is used to collect the image at the current moment;

[0009] Acquire point features and line features of the image at the current moment;

[0010] Based on the point features and line features of the current moment image, a preset point-line matching network including a graph attention mechanism is used to obtain a matching relationship between the point features and line features of the current moment image and the previous moment image;

[0011] When the total number of acquired images is greater than the preset number of sliding window frames, detecting the classification results of the structural main direction and line features of the image at the current moment based on the line features of the image at the current moment;

[0012] Based on the matching relationship between the point features and line features of the current moment image and the previous moment image, the classification result of the structural main direction and line features of the current moment image, and the measurement result of the inertial measurement unit in the visual inertial positioning system, the position and posture of the monocular camera are optimized to obtain the current positioning result of the visual inertial positioning system.

[0013] In a specific embodiment of the present invention, obtaining the point features and line features of the image at the current moment includes:

[0014] The current moment image I t Input the pre-trained convolutional neural network SuperPoint to obtain the dense descriptor graph corresponding to the image, the location of the point feature, the descriptor of the point feature, and the saliency of the point feature;

[0015] Use LSD line segment detection algorithm to process image I t , get the position of the line feature corresponding to the image;

[0016] Image I t The corresponding dense descriptor graph and the position of the line feature are input into the preset NeXtVLAD feature aggregation network to obtain the descriptor of the line feature corresponding to the image.

[0017] In a specific embodiment of the present invention, the point-line matching network comprises a sequentially connected position embedding subnetwork, a graph attention subnetwork, and a bidirectional Softmax matching operation module;

[0018] The position embedding subnetwork includes two parallel multi-layer perceptron networks; for the current moment image and the previous moment image, the first multi-layer perceptron network takes the position of the point feature of each frame image, the descriptor of the point feature, and the saliency of the point feature as input, and outputs the position embedding vector of the point feature; the second multi-layer perceptron network takes the length of the line feature of each frame image, the direction of the line feature, and the midpoint position of the line feature as input, and outputs the position embedding vector of the line feature;

[0019] The graph attention sub-network takes the descriptors of the point features and the position embedding vectors of the point features, the descriptors of the line features and the position embedding vectors of the line features of the two frames of images as input, and outputs the updated descriptors of the point features and the line features of the two frames of images;

[0020] The bidirectional Softmax matching operation module takes the descriptors of the point features and the line features of the two frames of images output by the graph attention subnetwork as input, and outputs the matching relationship between the point features and the line features of the two frames of images.

[0021] In a specific embodiment of the present invention, the graph attention subnetwork includes nine layers of attention layers connected in sequence, and each layer of attention layer contains six attention units, namely: same-frame point-point attention unit, same-frame line-line attention unit, same-frame point-line attention unit, same-frame line-point attention unit, different-frame point-point attention unit, different-frame line-line attention unit; wherein, in each layer of attention layer, the same-frame point-point attention unit takes the position embedding vector of the point feature of the two frames of images and the descriptor of the point feature as input, outputs the updated point feature descriptor and sends it to the same-frame point-line attention unit; the same-frame line-line attention unit takes the position embedding vector of the line feature of the two frames of images and the descriptor of the line feature as input, outputs the updated line feature descriptor and sends it to the same-frame point-line attention unit; the same-frame point-line attention unit takes the position embedding vector of the line feature of the two frames of images and the descriptor of the line feature as input, outputs the updated line feature descriptor and sends it to the same-frame point-line attention unit; The descriptors of the point features and line features of the two frames are taken as input, and the updated descriptors of the point features and the unupdated line features are output and sent to the same-frame line-point attention unit; the same-frame line-point attention unit takes the position embedding vectors of the point features and line features of the two frames of images and the received descriptors of the point features and line features as input, outputs the unupdated descriptors of the point features and sends them to the different-frame point-point attention unit, and at the same time outputs the updated descriptors of the line features and sends them to the different-frame line-line attention unit; the different-frame point-point attention unit takes the position embedding vectors of the point features of the two frames of images and the received descriptors of the point features as input, and outputs the descriptors of the point features that are finally updated after passing through this layer of attention layer; the different-frame line-line attention unit takes the position embedding vectors of the line features of the two frames of images and the received descriptors of the line features as input, and outputs the descriptors of the line features that are finally updated after passing through this layer of attention layer.

[0022] In a specific embodiment of the present invention, the attention unit uses the attention operation, which is expressed as follows:

[0023]

[0024] Among them, i and j are the subscripts of the features, taken from two subscript sets A and B respectively, and Γ contains all the feature subscript combinations constructed by these two subscript sets; f i is the descriptor of the i-th feature, and MLP is a i and attention vector a(i,Γ) as input multi-layer perceptron network, Softmax j is the softmax operation for subscript j, ← is the update operation; q i 、k j 、v j They are the query vector corresponding to the i-th feature, the key vector and the value vector corresponding to the j-th feature; the feature descriptor, position embedding vector, query vector, key vector and value vector have the same number of channels, which is D.

[0025] In a specific embodiment of the present invention, before obtaining the point features and line features of the image at the current moment, the method further includes:

[0026] Jointly training the NeXtVLAD feature aggregation network and the point-line matching network;

[0027] The joint training of the NeXtVLAD feature aggregation network and the point-line matching network includes:

[0028] First, using an image dataset, virtual image pairs are constructed by adding homography transformations, and the matching relationships of point and line features within the image pairs are recovered from the homography transformations. The matching relationships of the point and line features are used as the matching truth values ​​to supervise the joint training of the network. Secondly, after the training converges, the network weights trained in the previous stage are used as the initial weights. A dataset with real camera poses, camera intrinsic parameters, images, and dense depths is used. Image pairs with larger cospicuous areas in the dataset are selected. The correct matching relationships of point and line features are obtained based on the camera poses, camera intrinsic parameters, and dense depths provided by the dataset. The correct matching relationships of point and line features are used as the matching truth values ​​for the second stage of joint training of the network. After the second stage of training converges, the obtained network weights are used as the final network weights of the NeXtVLAD feature aggregation network and the point and line matching network.

[0029] In a specific embodiment of the present invention, the method further comprises:

[0030] If the total number of images currently collected is less than the preset number of sliding window frames, then when the next image collection moment arrives, continue to collect a new current moment image and obtain the matching relationship between the point features and line features of the new current moment image and the previous moment image;

[0031] If the total number of images currently collected is equal to the preset number of sliding window frames, the initial value of the camera pose at the real scale is obtained, including:

[0032] First, the SfM algorithm is used to obtain the scale-free camera pose corresponding to N frames of images. Second, the inertial measurement unit (IMU) measurement results are aligned with the camera pose obtained by the SfM algorithm to solve the scale factor, gravity direction, gyroscope bias, and camera velocity of N frames. Finally, the scale factor and the scale-free camera pose are used to obtain the camera pose corresponding to N frames of images at the real scale; where N is the preset number of sliding window frames.

[0033] In a specific embodiment of the present invention, the detecting of the classification result of the structural main direction and line features of the image at the current moment includes:

[0034] For image I tAll line features of the image are detected according to the direction of gravity. Among them, based on the normal vector of the line feature, the line features with the normal vector perpendicular to the gravity direction are detected and regarded as the structural lines distributed along the vertical direction. The remaining line features are clustered into horizontal line clusters to obtain image I. t All horizontal structure main directions of image I t All line features are divided into structural lines distributed along the vertical direction or the horizontal direction and non-structural lines distributed along other directions.

[0035] In a specific embodiment of the present invention, the optimizing the posture of the monocular camera includes:

[0036] 1) Estimate the camera pose of the last N frames through sliding window optimization;

[0037] Among them, the camera pose and velocity of the latest N frames, IMU bias, the spatial position of the point landmarks and line landmarks observed in the latest N frames, and the main direction of the structure are used as optimization variables to construct an optimization problem; the optimization problem takes the cost function L as the optimization target, and uses the Dogleg algorithm to minimize the cost function L to obtain the camera pose estimation result of the latest N frames; the cost function L is the sum of the reprojection error of the point feature, the reprojection error of the line feature, the IMU pre-integration error, the structural constraint error and the main direction observation error of the structure; the reprojection error of the point feature is the distance between the projection position of the point landmark on the image and the point feature position actually observed on the frame image, the reprojection error of the line feature is the distance between the projection position of the line landmark on the image and the end point of the line feature actually observed on the frame image, the structural constraint error is the cosine value of the angle between the main direction of the structure and the gravity direction, and the main direction observation error of the structure is the cosine value of the angle between the main direction of the structure and the normal vector of the line feature;

[0038] 2) Detect whether there is a loop in the image at the current moment;

[0039] Among them, for each frame image I t , using the DBoW algorithm with point feature descriptors as input to detect whether there is a loop: if there is a loop, the loop frame image I is returned old , then go to step 3); if there is no loop, the camera pose estimation result of step 1) is used as the current positioning result;

[0040] 3) Optimize the camera pose corresponding to the image trajectory through loop correction;

[0041] Wherein, based on the result of step 7-2), I old As I t-1 , calculate image I old and image I tThen, the PnP algorithm with RANSAC is used to determine the matching relationship between the point features and line features of image I old Relative Image I t Then, the pose value is used as the initial pose value of the loop frame, and the pose of the loop frame is inserted into the optimization variable of the optimization problem in step 1) as an additional optimization variable. old and image I t The reprojection error of the point features and line features constructed by the matching relationship of the point features and line features is added to the cost function of the optimization problem constructed in step 1), and the updated optimization problem is solved to obtain the image I old Relative Image I t The camera pose estimation result is added as a constraint item to the pose graph, and the pose graph optimization algorithm is used to optimize the pose of the image I. old To Image I t The camera pose estimation on the entire trajectory is corrected to obtain the current positioning result.

[0042] The second embodiment of the present invention provides a visual inertial positioning device for matching points and lines using a graph attention network, comprising:

[0043] An image acquisition module is used to acquire the current image using the monocular camera in the visual inertial positioning system during its operation;

[0044] A feature acquisition module, configured to acquire point features and line features of the image at the current moment;

[0045] a feature matching module for obtaining a matching relationship between the point features and line features of the current image and the previous image using a preset point-line matching network including a graph attention mechanism based on the point features and line features of the current image;

[0046] a structure detection module, configured to detect the main structural direction and classification results of the line features of the image at the current moment based on the line features of the image at the current moment when the total number of acquired images is greater than a preset number of sliding window frames;

[0047] A positioning module is configured to optimize the position and posture of the monocular camera based on the matching relationship between the point features and line features of the current image and the previous image, the classification result of the main structural direction and line features of the current image, and the measurement result of the inertial measurement unit in the visual-inertial positioning system to obtain the current positioning result of the visual-inertial positioning system.

[0048] A third embodiment of the present invention provides an electronic device, including:

[0049] at least one processor; and a memory communicatively coupled to the at least one processor;

[0050] The memory stores instructions that can be executed by the at least one processor, and the instructions are configured to execute the above-mentioned visual inertial positioning method for matching points and lines using a graph attention network.

[0051] A fourth aspect of the present invention provides a computer-readable storage medium storing computer instructions for enabling the computer to execute the above-mentioned visual-inertial positioning method for matching points and lines using a graph attention network.

[0052] The characteristics and beneficial effects of the present invention are:

[0053] The present invention demonstrates excellent positioning accuracy and robustness in situations involving changing lighting, rapid motion, and texture scarcity. Using a graph attention network, the present invention matches point and line features between two image frames. This robustness is achieved even when the brightness and viewing angles of the two images differ significantly. During pose estimation, the present invention integrates point and line feature observations, structural principal direction observations, and IMU measurements, maximizing the use of available sensor measurement information and the structural characteristics of the environment for positioning, effectively improving positioning accuracy in weakly textured scenes. The present invention is primarily used for positioning small mobile devices in indoor environments, capable of providing relative positioning information when GPS information is unavailable. The present invention also combines three advantages: it eliminates the need for complex environmental mapping and can be used in unfamiliar environments; the system sensors require only a monocular camera and an IMU, eliminating complex equipment requirements; and it exhibits good adaptability to various operating environments. These advantages make the present invention relatively easy to deploy on small mobile systems at a relatively low cost. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] Figure 1 This is an overall flow chart of a visual-inertial positioning method for matching points and lines using a graph attention network according to an embodiment of the present invention.

[0055] Figure 2 This is a schematic diagram of the graph attention sub-network in a specific embodiment of the present invention. DETAILED DESCRIPTION

[0056] The present invention proposes a visual inertial positioning method and device using a graph attention network to match points and lines. The present invention is described in detail below with reference to the accompanying drawings and specific embodiments.

[0057] The first embodiment of the present invention proposes a visual inertial positioning method for matching points and lines using a graph attention network, comprising:

[0058] During the operation of the visual inertial positioning system, the monocular camera in the system is used to collect the image at the current moment;

[0059] Acquire point features and line features of the image at the current moment;

[0060] Based on the point features and line features of the current moment image, a preset point-line matching network including a graph attention mechanism is used to obtain a matching relationship between the point features and line features of the current moment image and the previous moment image;

[0061] When the total number of acquired images is greater than the preset number of sliding window frames, detecting the classification results of the structural main direction and line features of the image at the current moment based on the line features of the image at the current moment;

[0062] Based on the matching relationship between the point features and line features of the current moment image and the previous moment image, the classification result of the structural main direction and line features of the current moment image, and the measurement result of the inertial measurement unit in the visual inertial positioning system, the position and posture of the monocular camera are optimized to obtain the current positioning result of the visual inertial positioning system.

[0063] In a specific embodiment of the present invention, the visual inertial positioning method using graph attention network to match points and lines has the following overall process: Figure 1 As shown, the following steps are included:

[0064] 1) Build a visual-inertial positioning system, including a monocular camera, an inertial measurement unit (IMU), and an onboard computer. The monocular camera and IMU need to be fixed to the same bracket, with a camera frame rate of 10 to 30 Hz (20 Hz in one embodiment of the present invention) and an IMU frequency of 100 to 400 Hz (200 Hz in this embodiment). All components of the visual-inertial positioning system described in this embodiment can be conventional models.

[0065] 2) The current moment is recorded as time t, and the image of the current moment is collected by the monocular camera in the visual inertial positioning system built in step 1) and recorded as I t .

[0066] 3) Image I obtained in step 2) t Processing is performed to obtain the matching relationship between the point features and line features of the current moment image and the previous moment image; the specific steps are as follows:

[0067] 3-1) For image I t Perform point and line feature extraction.

[0068] Specifically, the pre-trained convolutional neural network SuperPoint is used to process the image I t, get the dense descriptor graph, point feature location, point feature descriptor and point feature saliency corresponding to the image; then use LSD (line segment detection) algorithm to process image I t , get the position of the line feature corresponding to the image; then, image I t The corresponding dense descriptor graph and the position of the line feature are input into the preset NeXtVLAD feature aggregation network to obtain the descriptor of the line feature corresponding to the image; the image I t The position of the point feature, the descriptor of the point feature, the saliency of the point feature, the position of the line feature and the descriptor of the line feature are saved as the point and line feature information of the image.

[0069] 3-2) Judgment:

[0070] If t is the initial time, then when the next image acquisition time arrives, return to step 2); otherwise, go to step 3-3).

[0071] 3-3) Using Image I t Compared with the previous moment image I t-1 The point and line feature information of the two images is used to calculate the matching relationship between the point features and line features of the two images.

[0072] Specifically, a preset point-line matching network is used to obtain the image I t and the previous moment image I t-1 The position of the point feature, the descriptor of the point feature, the saliency of the point feature, the position of the line feature and the descriptor of the line feature are taken as input to obtain the matching relationship between the point features and line features of the two frames of images.

[0073] In this embodiment, the point-line matching network includes three parts: a sequentially connected position embedding subnetwork, a graph attention subnetwork, and a bidirectional Softmax matching operation module.

[0074] The position embedding subnetwork includes two parallel multi-layer perceptron networks; the first multi-layer perceptron network (in this embodiment, the input layer dimension is 4, the output layer dimension is 256, and the hidden layer dimensions are 32, 64, and 128 respectively) takes the position of the point feature of each frame image, the descriptor of the point feature, and the saliency of the point feature as input, and outputs the position embedding vector of the point feature; the second multi-layer perceptron network (in this embodiment, the input layer dimension is 5, the output layer dimension is 256, and the hidden layer dimensions are 32, 64, and 128 respectively) takes the length of the line feature of each frame image, the direction of the line feature, and the midpoint position of the line feature as input, and outputs the position embedding vector of the line feature.

[0075] The principle of the graph attention sub-network is as follows Figure 2As shown, the graph attention sub-network takes the descriptors of the point features and the position embedding vectors of the point features, the descriptors of the line features and the position embedding vectors of the line features of the two frames of images as input, and outputs the descriptors of the point features and the line features after being updated in the two frames of images. Specifically, in this embodiment, the graph attention sub-network includes nine layers of attention layers connected in sequence, and each layer of attention layer contains six attention units, namely: same-frame point-to-point attention unit, same-frame line-to-line attention unit, same-frame point-to-line attention unit, same-frame line-to-point attention unit, different-frame point-to-point attention unit, and different-frame line-to-line attention unit. In the same attention layer, the connection relationship between the attention units is as follows: Figure 2As shown, specifically, the outputs of the same-frame point-point attention unit and the same-frame line-line attention unit are respectively connected to the input of the same-frame point-line attention unit; the output of the same-frame point-line attention unit is connected to the input of the same-frame line-point attention unit; the output of the same-frame line-point attention unit is respectively connected to the input of the different-frame point-point attention unit and the input of the different-frame line-line attention unit, and the output of the different-frame point-point attention unit and the different-frame line-line attention unit is the output of the attention layer. Among them, in each attention layer, the point-to-point attention unit of the same frame takes the position embedding vector of the point feature of the two frames of images and the descriptor of the point feature as input, outputs the updated point feature descriptor and sends it to the point-line attention unit of the same frame; the line-line attention unit of the same frame takes the position embedding vector of the line feature of the two frames of images and the descriptor of the line feature as input, outputs the updated line feature descriptor and sends it to the point-line attention unit of the same frame; the point-line attention unit of the same frame takes the position embedding vector of the point feature and line feature of the two frames of images and the descriptor of the received point feature and line feature as input, outputs the updated point feature descriptor and the unupdated line feature descriptor and sends them to the line-point attention unit of the same frame; The point attention unit takes the position embedding vectors of the point features and line features of the two frames of images and the received descriptors of the point features and line features as input, outputs the unupdated descriptors of the point features and sends them to the different-frame point-point attention unit, and at the same time outputs the updated descriptors of the line features and sends them to the different-frame line-line attention unit; the different-frame point-point attention unit takes the position embedding vectors of the point features of the two frames of images and the received descriptors of the point features as input, and outputs the descriptors of the point features that have been finally updated by the attention layer; the different-frame line-line attention unit takes the position embedding vectors of the line features of the two frames of images and the received descriptors of the line features as input, and outputs the descriptors of the line features that have been finally updated by the attention layer. It should be noted that the position embedding vectors input to the attention units of all attention layers come from the position embedding subnetwork; the descriptors of the point features input to the same-frame point-point attention unit and the descriptors of the line features input to the same-frame line-line attention unit of the first attention layer come from the corresponding descriptors obtained in step 3-1). The heterogeneous point attention unit of the ninth attention layer outputs the descriptor of the point feature after the final update of the graph attention sub-network, and the heterogeneous line attention unit of the ninth attention layer outputs the descriptor of the line feature after the final update of the graph attention sub-network.

[0076] Furthermore, each attention unit uses the attention operation to update the descriptors of the point features and the line features. The following formula describes the process of the attention operation:

[0077]

[0078] Here, i and j are feature subscripts, taken from two subscript sets A and B, respectively. Γ encompasses all feature subscript combinations constructed from these two subscript sets. fi is the descriptor of the i-th feature (a point feature or line feature). The MLP is a multilayer perceptron network that takes fi and the attention vector a(i,Γ) as inputs. Softmaxj is the softmax operation performed on subscript j, and ← is an update operation. qi, kj, and vj are the query vector corresponding to the i-th feature, the key vector corresponding to the j-th feature, and the value vector, respectively. They are generated by a multilayer perceptron network that takes the feature descriptor and position embedding vector as input. The feature descriptor, position embedding vector, query vector, key vector, and value vector all have the same number of channels, which is D. In particular, different sublayers operate between different feature index sets A and B. Specifically, A and B of the same-frame point-point attention unit are all point features in the same image, A and B of the same-frame line-line attention unit are all line features in the same image, A and B of the same-frame point-line attention unit are all point features and all line features in the same image, respectively, A and B of the same-frame line-point attention unit are all line features and all point features in the same image, respectively, A and B of the different-frame point-point attention unit are all point features from different images, and A and B of the different-frame line-line attention unit are all line features from different images. The descriptor of the feature output by the ninth attention layer is the final output of the graph attention subnetwork.

[0079] The bidirectional softmax matching module takes the descriptors of the point features and line features of the two frames of images output by the graph attention subnetwork as input and outputs the matching relationships of all the point features and line features in the two frames of images. Specifically, this embodiment adopts the bidirectional softmax matching method used by the SuperGlue network, calculating the matching score of the features by taking the dot product of the descriptors and using a bidirectional softmax operation to obtain the matching probability of the features. Finally, the feature pair with the highest mutual matching probability is selected as the matching feature, thus obtaining the matching relationship of the point features and line features in the two frames of images.

[0080] 4) Determination: If the total number of images currently collected is less than N frames, then when the next image collection time arrives, return to step 2); if the total number of images currently collected is equal to N frames (N is 10 in this embodiment), proceed to step 5); if the total number of images currently collected is greater than N frames, proceed to step 6); where N is the preset number of sliding window frames.

[0081] 5) Obtain the initial value of the camera pose at the real scale.

[0082] Specifically, the initialization method of VINS-Mono is used to initialize the gravity direction, gyroscope bias, camera pose and velocity of N frames.

[0083] In this embodiment, first, the SfM algorithm (Structure from Motion) is used to obtain the scale-free camera pose corresponding to N frames of images. Second, the IMU measurement results are used to align the camera pose obtained by the SfM algorithm to solve the scale factor, gravity direction, gyroscope bias, and camera velocity of N frames. Finally, the scale factor and the scale-free camera pose are used to obtain the camera pose corresponding to the N frames of images at the real scale.

[0084] The quantities solved in this step include: gravity direction, gyroscope bias, and camera pose and velocity for N frames. These quantities are saved as initial values ​​for structural feature detection and sliding window optimization in subsequent steps 6) and 7).

[0085] 6) Based on the line features of the image at the current moment, the main structural direction of the image at the current moment and the classification results of the line features are detected.

[0086] In this embodiment, the image I t All line features, when the direction of gravity is known, detect the structural line features that conform to the Atlanta world hypothesis. Specifically: First, based on the normal vector of the line feature, the line features whose normal vector is perpendicular to the direction of gravity are detected and used as the structural lines distributed along the vertical direction. Then, for the remaining line features, the MnS (mined-and-stabbed) algorithm proposed in the paper S-VIO: Exploiting Structural Constraints for RGB-D Visual Inertial Odometry is used to cluster the horizontal line clusters. For each line feature, the algorithm will calculate the horizontal subinterval to which it may belong based on its normal vector, and extract the subinterval with a large number of subordinate line features as the main direction of the detected horizontal structure. The line features involved in these subintervals are the structural lines distributed along a certain horizontal direction. In this way, image I is obtained. t All horizontal structure main directions observed, and image I t All line features are divided into structural lines distributed along the vertical direction or along a certain horizontal direction and non-structural lines distributed along other directions.

[0087] 7) Based on the point-line feature matching relationship obtained in step 3), the main structural direction detection result obtained in step 5) (i.e., the number and direction of the main horizontal structural directions), and the line feature classification result (i.e., whether it is a structural line, and if so, whether it is distributed along the vertical direction or along which horizontal main structural direction it is distributed), combined with the IMU measurement results, the camera pose is determined.

[0088] Specifically, this step draws on the VINS-Mono algorithm to perform pose estimation. The specific steps are as follows:

[0089] 7-1) Estimate the camera pose of the last N frames through sliding window optimization.

[0090] In this embodiment, the camera pose and velocity of the latest N frames, the IMU bias, the spatial position of the point landmarks and line landmarks observed in the latest N frames (their initial values ​​are obtained by the triangulation method, the specific method is the same as that used in VINS-Mono), and the main direction of the structure are used as optimization variables to construct an optimization problem. This optimization problem takes the cost function L as the optimization target and uses the Dogleg algorithm to minimize the cost function L to obtain the camera pose estimation results of the latest N frames. The cost function L is the sum of five parts, namely the reprojection error of the point feature (defined as the distance between the projection position of the point landmark on the image and the actual observed point feature position on the frame image), the reprojection error of the line feature (defined as the distance between the projection position of the line landmark on the image and the actual observed line feature endpoint on the frame image), the IMU pre-integration error, the structural constraint error (defined as the cosine of the angle between the main direction of the structure and the direction of gravity), and the main direction observation error of the structure (defined as the cosine of the angle between the main direction of the structure and the normal vector of the line feature).

[0091] 7-2) Check whether there is a loop in the current image.

[0092] In this embodiment, for each frame image I t , using the DBoW (Distributed Bag of Words) algorithm with point feature descriptors as input to detect whether there is a loop: if there is a loop, the loop frame image I is returned old , then go to step 7-3); if there is no loop, the camera pose estimation result of step 7-1) is used as the current positioning result.

[0093] 7-3) Through loop correction, the camera pose corresponding to the image trajectory is optimized.

[0094] In this embodiment, based on the result of step 7-2), I old As I t-1 , repeat steps 3-3) to calculate image I old and image I t The matching relationship between point features and line features.

[0095] After getting the image I old and image I t After the matching relationship between the point features and line features, the PnP (Perspective-n-Points) algorithm with RANSAC is used to determine the image I old Relative Image I tThen, the pose value is used as the initial pose value of the loop frame, and the pose of the loop frame is inserted into the optimization variable list of the optimization problem in step 7-1) as an additional optimization variable. old and image I t The reprojection error of the point features and line features constructed by the matching relationship of the point features and line features is added to the cost function of the optimization problem constructed in step 6-1), and the updated optimization problem is solved. The camera pose estimation result of the loop frame is further optimized to obtain image I old Relative Image I t The camera pose estimation result is obtained by adding the camera pose estimation result as a constraint item to the pose graph, and the pose graph optimization algorithm is used to optimize the pose of the image I. old To Image I t The camera pose estimation on the entire trajectory is corrected to obtain the current positioning result.

[0096] It should be noted that the result of step 7-1) is used to provide the initial value for step 7-3), that is, the estimated value of the camera pose for the entire trajectory (from time zero to the current time). In this embodiment, step 7-1) optimizes the camera pose for the most recent N frames, considering only the observation information from the most recent period. In contrast, step 7-3) optimizes the camera pose for the entire trajectory from the loop frame to the current frame, utilizing the visual feature matching information between the loop frame and the current frame when a loop is detected.

[0097] 8) Save the camera pose optimization result obtained in step 7) and use it as the initial pose value for the next positioning. When the next image acquisition time arrives, return to step 2) to achieve continuous positioning of the visual inertial positioning system.

[0098] In particular, in the method described in this embodiment, before step 2), it is necessary to jointly train the NeXtVLAD feature aggregation network and the point-line matching network. The training method is the same as the training method used in the paper GlueStick: Robust Image Matching by Sticking Points and Lines Together. Specifically, the training method is divided into two stages. First, a picture dataset is used (this embodiment uses the revisitop dataset) to construct a virtual picture pair by adding a homography transformation, and the matching relationship of the point-line features in the picture pair is restored from the homography transformation. The matching relationship of the point-line features is used as the matching true value to supervise the network joint training. Secondly, after the model training converges, the network weights trained in the previous stage are used as the initial weights. A data set with real camera pose, camera intrinsic parameters, pictures and dense depth is used (this embodiment uses the megadepth data set). The picture pairs with larger cosight areas in the data set are selected, and the correct matching relationship of the point-line features is obtained based on the camera pose, camera intrinsic parameters and dense depth provided by the data set. The matching relationship of the point-line features is used as the matching true value to carry out the second stage of network joint training. After the second stage of training converges, the obtained network weights are used as the network weights of the NeXtVLAD feature aggregation network and the point-line matching network when applied.

[0099] A specific embodiment of the present invention is deployed on a desktop computer with an Intel Core i9 processor running at 3.6GHz, 32GB of memory, and an Nvidia RTX3060 GPU (with 12GB of video memory) to run the neural network. The method described in this embodiment is implemented in C++ and uses the TensorRT library to accelerate neural network reasoning.

[0100] In order to test the positioning accuracy of the method described in this embodiment in a real environment, this embodiment selected the public TUM-VI dataset and UMA-VI dataset. Both of the above datasets are handheld acquisition, and the sensor data is given in the form of rosbag, including monocular image data with a frame rate of 30fps and IMU data with a frequency of 200Hz. Both the IMU data and the monocular image data are accompanied by timestamps, and the timestamps are located on the same timeline. The two datasets contain a total of 53 test sequences, and the length of each sequence ranges from 4 minutes to 16 minutes. The recording environment includes typical scenes such as corridors, classrooms, stairwells, halls, offices, laboratories, roads, and parking lots. During the test, we used monocular images with a downscaled frame rate of 10fps and IMU data at 200Hz as algorithm inputs. The average time consumption of each algorithm module was 55ms for feature detection and matching, 20ms for sliding window optimization, and 20ms for loop correction: bag-of-words recall took 20ms, and if a loop frame was successfully recalled, feature matching on the loop frame took 45ms. The PnP algorithm took 32ms, and the pose graph optimization time depended on the length of the trajectory to be optimized (40 to 200ms). Since feature detection and matching, sliding window optimization, and loop correction were executed in parallel in three threads, the method described in this embodiment met the real-time requirements of operation as a whole.

[0101] To implement the above embodiment, a second embodiment of the present invention proposes a visual inertial positioning device for matching points and lines using a graph attention network, comprising:

[0102] An image acquisition module is used to acquire the current image using the monocular camera in the visual inertial positioning system during its operation;

[0103] A feature acquisition module, configured to acquire point features and line features of the image at the current moment;

[0104] a feature matching module for obtaining a matching relationship between the point features and line features of the current image and the previous image using a preset point-line matching network including a graph attention mechanism based on the point features and line features of the current image;

[0105] a structure detection module, configured to detect the main structural direction and classification results of the line features of the image at the current moment based on the line features of the image at the current moment when the total number of acquired images is greater than a preset number of sliding window frames;

[0106] A positioning module is configured to optimize the position and posture of the monocular camera based on the matching relationship between the point features and line features of the current image and the previous image, the classification result of the main structural direction and line features of the current image, and the measurement result of the inertial measurement unit in the visual-inertial positioning system to obtain the current positioning result of the visual-inertial positioning system.

[0107] In a specific embodiment of the present invention, obtaining the point features and line features of the image at the current moment includes:

[0108] The current moment image I t Input the pre-trained convolutional neural network SuperPoint to obtain the dense descriptor graph corresponding to the image, the location of the point feature, the descriptor of the point feature, and the saliency of the point feature;

[0109] Use LSD line segment detection algorithm to process image I t , get the position of the line feature corresponding to the image;

[0110] Image I t The corresponding dense descriptor graph and the position of the line feature are input into the preset NeXtVLAD feature aggregation network to obtain the descriptor of the line feature corresponding to the image.

[0111] In a specific embodiment of the present invention, the point-line matching network comprises a sequentially connected position embedding subnetwork, a graph attention subnetwork, and a bidirectional Softmax matching operation module;

[0112] The position embedding subnetwork includes two parallel multi-layer perceptron networks; for the current moment image and the previous moment image, the first multi-layer perceptron network takes the position of the point feature of each frame image, the descriptor of the point feature, and the saliency of the point feature as input, and outputs the position embedding vector of the point feature; the second multi-layer perceptron network takes the length of the line feature of each frame image, the direction of the line feature, and the midpoint position of the line feature as input, and outputs the position embedding vector of the line feature;

[0113] The graph attention sub-network takes the descriptors of the point features and the position embedding vectors of the point features, the descriptors of the line features and the position embedding vectors of the line features of the two frames of images as input, and outputs the updated descriptors of the point features and the line features of the two frames of images;

[0114] The bidirectional Softmax matching operation module takes the descriptors of the point features and the line features of the two frames of images output by the graph attention subnetwork as input, and outputs the matching relationship between the point features and the line features of the two frames of images.

[0115] In a specific embodiment of the present invention, the graph attention subnetwork includes nine layers of attention layers connected in sequence, and each layer of attention layer contains six attention units, namely: same-frame point-point attention unit, same-frame line-line attention unit, same-frame point-line attention unit, same-frame line-point attention unit, different-frame point-point attention unit, different-frame line-line attention unit; wherein, in each layer of attention layer, the same-frame point-point attention unit takes the position embedding vector of the point feature of the two frames of images and the descriptor of the point feature as input, outputs the updated point feature descriptor and sends it to the same-frame point-line attention unit; the same-frame line-line attention unit takes the position embedding vector of the line feature of the two frames of images and the descriptor of the line feature as input, outputs the updated line feature descriptor and sends it to the same-frame point-line attention unit; the same-frame point-line attention unit takes the position embedding vector of the line feature of the two frames of images and the descriptor of the line feature as input, outputs the updated line feature descriptor and sends it to the same-frame point-line attention unit; The descriptors of the point features and line features of the two frames are taken as input, and the updated descriptors of the point features and the unupdated line features are output and sent to the same-frame line-point attention unit; the same-frame line-point attention unit takes the position embedding vectors of the point features and line features of the two frames of images and the received descriptors of the point features and line features as input, outputs the unupdated descriptors of the point features and sends them to the different-frame point-point attention unit, and at the same time outputs the updated descriptors of the line features and sends them to the different-frame line-line attention unit; the different-frame point-point attention unit takes the position embedding vectors of the point features of the two frames of images and the received descriptors of the point features as input, and outputs the descriptors of the point features that are finally updated after passing through this layer of attention layer; the different-frame line-line attention unit takes the position embedding vectors of the line features of the two frames of images and the received descriptors of the line features as input, and outputs the descriptors of the line features that are finally updated after passing through this layer of attention layer.

[0116] In a specific embodiment of the present invention, the attention unit uses the attention operation, which is expressed as follows:

[0117]

[0118] Among them, i and j are the subscripts of the features, taken from two subscript sets A and B respectively, and Γ contains all the feature subscript combinations constructed by these two subscript sets; f i is the descriptor of the i-th feature, and MLP is a i and attention vector a(i,Γ) as input multi-layer perceptron network, Softmax j is the softmax operation for subscript j, ← is the update operation; q i 、k j 、v j They are the query vector corresponding to the i-th feature, the key vector and the value vector corresponding to the j-th feature; the feature descriptor, position embedding vector, query vector, key vector and value vector have the same number of channels, which is D.

[0119] In a specific embodiment of the present invention, before obtaining the point features and line features of the image at the current moment, the method further includes:

[0120] Jointly training the NeXtVLAD feature aggregation network and the point-line matching network;

[0121] The joint training of the NeXtVLAD feature aggregation network and the point-line matching network includes:

[0122] First, using an image dataset, virtual image pairs are constructed by adding homography transformations, and the matching relationships of point and line features within the image pairs are recovered from the homography transformations. The matching relationships of the point and line features are used as the matching truth values ​​to supervise the joint training of the network. Secondly, after the training converges, the network weights trained in the previous stage are used as the initial weights. A dataset with real camera poses, camera intrinsic parameters, images, and dense depths is used. Image pairs with larger cospicuous areas in the dataset are selected. The correct matching relationships of point and line features are obtained based on the camera poses, camera intrinsic parameters, and dense depths provided by the dataset. The correct matching relationships of point and line features are used as the matching truth values ​​for the second stage of joint training of the network. After the second stage of training converges, the obtained network weights are used as the final network weights of the NeXtVLAD feature aggregation network and the point and line matching network.

[0123] In a specific embodiment of the present invention, it also includes:

[0124] If the total number of images currently collected is less than the preset number of sliding window frames, then when the next image collection moment arrives, continue to collect a new current moment image and obtain the matching relationship between the point features and line features of the new current moment image and the previous moment image;

[0125] If the total number of images currently collected is equal to the preset number of sliding window frames, the initial value of the camera pose at the real scale is obtained, including:

[0126] First, the SfM algorithm is used to obtain the scale-free camera pose corresponding to N frames of images. Second, the inertial measurement unit (IMU) measurement results are aligned with the camera pose obtained by the SfM algorithm to solve the scale factor, gravity direction, gyroscope bias, and camera velocity of N frames. Finally, the scale factor and the scale-free camera pose are used to obtain the camera pose corresponding to N frames of images at the real scale; where N is the preset number of sliding window frames.

[0127] In a specific embodiment of the present invention, the detecting of the classification result of the structural main direction and line features of the image at the current moment includes:

[0128] For image I tAll line features of the image are detected according to the direction of gravity. Among them, based on the normal vector of the line feature, the line features with the normal vector perpendicular to the gravity direction are detected and regarded as the structural lines distributed along the vertical direction. The remaining line features are clustered into horizontal line clusters to obtain image I. t All horizontal structure main directions of image I t All line features are divided into structural lines distributed along the vertical direction or the horizontal direction and non-structural lines distributed along other directions.

[0129] In a specific embodiment of the present invention, the optimizing the posture of the monocular camera includes:

[0130] 1) Estimate the camera pose of the last N frames through sliding window optimization;

[0131] Among them, the camera pose and velocity of the latest N frames, IMU bias, the spatial position of the point landmarks and line landmarks observed in the latest N frames, and the main direction of the structure are used as optimization variables to construct an optimization problem; the optimization problem takes the cost function L as the optimization target, and uses the Dogleg algorithm to minimize the cost function L to obtain the camera pose estimation result of the latest N frames; the cost function L is the sum of the reprojection error of the point feature, the reprojection error of the line feature, the IMU pre-integration error, the structural constraint error and the main direction observation error of the structure; the reprojection error of the point feature is the distance between the projection position of the point landmark on the image and the point feature position actually observed on the frame image, the reprojection error of the line feature is the distance between the projection position of the line landmark on the image and the end point of the line feature actually observed on the frame image, the structural constraint error is the cosine value of the angle between the main direction of the structure and the gravity direction, and the main direction observation error of the structure is the cosine value of the angle between the main direction of the structure and the normal vector of the line feature;

[0132] 2) Detect whether there is a loop in the image at the current moment;

[0133] Among them, for each frame image I t , using the DBoW algorithm with point feature descriptors as input to detect whether there is a loop: if there is a loop, the loop frame image I is returned old , then go to step 3); if there is no loop, the camera pose estimation result of step 1) is used as the current positioning result;

[0134] 3) Optimize the camera pose corresponding to the image trajectory through loop correction;

[0135] Wherein, based on the result of step 7-2), I old As I t-1 , calculate image I old and image I tThen, the PnP algorithm with RANSAC is used to determine the matching relationship between the point features and line features of image I old Relative Image I t Then, the pose value is used as the initial pose value of the loop frame, and the pose of the loop frame is inserted into the optimization variable of the optimization problem in step 1) as an additional optimization variable. old and image I t The reprojection error of the point features and line features constructed by the matching relationship of the point features and line features is added to the cost function of the optimization problem constructed in step 1), and the updated optimization problem is solved to obtain the image I old Relative Image I t The camera pose estimation result is added as a constraint item to the pose graph, and the pose graph optimization algorithm is used to optimize the pose of the image I. old To Image I t The camera pose estimation on the entire trajectory is corrected to obtain the current positioning result.

[0136] This enables point-line feature matching through the graph attention network, improving the accuracy and robustness of visual-inertial positioning in situations of lighting changes, rapid motion, and lack of texture, and has good engineering application value.

[0137] To implement the above embodiment, a third aspect of the present invention provides an electronic device, including:

[0138] at least one processor; and a memory communicatively coupled to the at least one processor;

[0139] The memory stores instructions that can be executed by the at least one processor, and the instructions are configured to execute the above-mentioned visual inertial positioning method for matching points and lines using a graph attention network.

[0140] To implement the above-mentioned embodiment, the fourth aspect of the present invention proposes a computer-readable storage medium, which stores computer instructions, and the computer instructions are used to enable the computer to execute the above-mentioned visual-inertial positioning method for matching points and lines using a graph attention network.

[0141] It should be noted that the computer-readable medium mentioned above in the present disclosure may be a computer-readable signal medium or a computer-readable storage medium, or any combination of the two. A computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or component, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, a computer-readable storage medium may be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, device, or component. In the present disclosure, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, which carries computer-readable program code. Such a propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium that can transmit, propagate, or transport a program for use by or in conjunction with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium may be transmitted using any suitable medium, including but not limited to wires, optical cables, RF (radio frequency), etc., or any suitable combination thereof.

[0142] The computer-readable medium may be included in the electronic device or may exist independently without being incorporated into the electronic device. The computer-readable medium carries one or more programs that, when executed by the electronic device, cause the electronic device to perform the visual-inertial localization method using a graph attention network to match points and lines according to the aforementioned embodiment.

[0143] Computer program code for performing the operations of the present disclosure may be written in one or more programming languages, or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on the remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., through the Internet using an Internet service provider).

[0144] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art can combine and combine different embodiments or examples described in this specification and features of different embodiments or examples without contradiction.

[0145] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of the technical features being referred to. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of such features. Throughout the description of this application, "plurality" means at least two, for example, two, three, etc., unless otherwise specifically defined.

[0146] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, segment or portion of code comprising one or more executable instructions for implementing the steps of a specific logical function or process, and the scope of the preferred embodiments of the present application includes alternative implementations in which functions may be performed out of the order shown or discussed, including performing functions in a substantially simultaneous manner or in the reverse order depending on the functions involved, which should be understood by those skilled in the art to which the embodiments of the present application belong.

[0147] The logic and / or steps represented in the flowcharts or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing the logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (e.g., a computer-based system, a system including a processor, or other system that can fetch and execute instructions from an instruction execution system, apparatus, or device). For purposes of this specification, a "computer-readable medium" can be any device that can contain, store, communicate, propagate, or transport a program for use by, or in conjunction with, an instruction execution system, apparatus, or device. More specific examples (a non-exhaustive list) of computer-readable media include the following: an electrical connection with one or more wires (electronic devices), a portable computer disk cartridge (magnetic device), random access memory (RAM), read-only memory (ROM), erasable and programmable read-only memory (EPROM or flash memory), fiber optic devices, and a portable compact disc read-only memory (CDROM). Furthermore, the computer-readable medium may even be paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium and then editing, interpreting or otherwise processing it in a suitable manner if necessary, and then storing it in a computer memory.

[0148] It should be understood that various parts of the present application can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented using software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented using hardware, as in another embodiment, any one of the following technologies known in the art or a combination thereof can be used to implement: a discrete logic circuit having a logic gate circuit for implementing a logic function on a data signal, an application-specific integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0149] Those skilled in the art will understand that all or part of the steps in the method of the above embodiment can be completed by instructing related hardware through a program, and the program can be stored in a computer-readable storage medium. When the program is executed, it includes one or a combination of the steps of the method embodiment.

[0150] In addition, the functional units in the various embodiments of the present application may be integrated into a processing module, or each unit may exist physically separately, or two or more units may be integrated into a module. The above-mentioned integrated module may be implemented in the form of hardware or in the form of a software functional module. If the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it may also be stored in a computer-readable storage medium.

[0151] The storage medium mentioned above may be a read-only memory, a magnetic disk, or an optical disk, etc. Although the embodiments of the present application have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting the present application. Persons skilled in the art may make changes, modifications, substitutions, and variations to the above embodiments within the scope of the present application.

Claims

1. A visual inertial positioning method using graph attention network to match points and lines, characterized by: include: During the operation of the visual inertial positioning system, the monocular camera in the system is used to collect the image at the current moment; Acquire point features and line features of the image at the current moment; Based on the point features and line features of the current moment image, a preset point-line matching network including a graph attention mechanism is used to obtain a matching relationship between the point features and line features of the current moment image and the previous moment image; When the total number of acquired images is greater than the preset number of sliding window frames, detecting the classification results of the structural main direction and line features of the image at the current moment based on the line features of the image at the current moment; Based on the matching relationship between the point features and line features of the current moment image and the previous moment image, the classification result of the structural main direction and line features of the current moment image, and the measurement result of the inertial measurement unit in the visual inertial positioning system, the position and posture of the monocular camera are optimized to obtain the current positioning result of the visual inertial positioning system.

2. The method according to claim 1, characterized in that The acquiring of point features and line features of the image at the current moment includes: The current moment image I t Input the pre-trained convolutional neural network SuperPoint to obtain the dense descriptor graph corresponding to the image, the location of the point feature, the descriptor of the point feature, and the saliency of the point feature; Use LSD line segment detection algorithm to process image I t , get the position of the line feature corresponding to the image; Image I t The corresponding dense descriptor graph and the position of the line feature are input into the preset NeXtVLAD feature aggregation network to obtain the descriptor of the line feature corresponding to the image.

3. The method according to claim 2, characterized in that The point-line matching network includes a sequentially connected position embedding subnetwork, a graph attention subnetwork, and a bidirectional Softmax matching operation module; The position embedding subnetwork includes two parallel multi-layer perceptron networks; for the current moment image and the previous moment image, the first multi-layer perceptron network takes the position of the point feature of each frame image, the descriptor of the point feature, and the saliency of the point feature as input, and outputs the position embedding vector of the point feature; the second multi-layer perceptron network takes the length of the line feature of each frame image, the direction of the line feature, and the midpoint position of the line feature as input, and outputs the position embedding vector of the line feature; The graph attention sub-network takes the descriptors of the point features and the position embedding vectors of the point features, the descriptors of the line features and the position embedding vectors of the line features of the two frames of images as input, and outputs the updated descriptors of the point features and the line features of the two frames of images; The bidirectional Softmax matching operation module takes the descriptors of the point features and the line features of the two frames of images output by the graph attention subnetwork as input, and outputs the matching relationship between the point features and the line features of the two frames of images.

4. The method according to claim 3, characterized in that The graph attention subnetwork includes nine attention layers connected in sequence, and each attention layer contains six attention units, namely: same-frame point-point attention unit, same-frame line-line attention unit, same-frame point-line attention unit, same-frame line-point attention unit, different-frame point-point attention unit, different-frame line-line attention unit; wherein, in each attention layer, the same-frame point-point attention unit takes the position embedding vector of the point feature of the two frames of images and the descriptor of the point feature as input, outputs the updated point feature descriptor and sends it to the same-frame point-line attention unit; the same-frame line-line attention unit takes the position embedding vector of the line feature of the two frames of images and the descriptor of the line feature as input, outputs the updated line feature descriptor and sends it to the same-frame point-line attention unit; the same-frame point-line attention unit takes the position embedding vector of the point feature and the line feature descriptor of the two frames of images as input, outputs the updated line feature descriptor and sends it to the same-frame point-line attention unit; the same-frame point-line attention unit takes the position embedding vector of the point feature and the line feature of the two frames of images and the received point feature and line feature as input. The descriptor of the feature is taken as input, and the updated descriptor of the point feature and the unupdated descriptor of the line feature are output and sent to the same-frame line-point attention unit; the same-frame line-point attention unit takes the position embedding vectors of the point features and line features of the two frames of images and the received descriptors of the point features and line features as input, outputs the unupdated descriptor of the point features and sends it to the different-frame point-point attention unit, and at the same time outputs the updated descriptor of the line features and sends it to the different-frame line-line attention unit; the different-frame point-point attention unit takes the position embedding vectors of the point features of the two frames of images and the received descriptors of the point features as input, and outputs the descriptor of the point features that are finally updated after passing through this layer of attention layer; the different-frame line-line attention unit takes the position embedding vectors of the line features of the two frames of images and the received descriptors of the line features as input, and outputs the descriptor of the line features that are finally updated after passing through this layer of attention layer.

5. The method according to claim 4, characterized in that The attention unit uses the attention operation, which is expressed as follows: Among them, i and j are the subscripts of the features, taken from two subscript sets A and B respectively, and Γ contains all the feature subscript combinations constructed by these two subscript sets; f i is the descriptor of the i-th feature, and MLP is a i and attention vector a(i,Γ) as input multi-layer perceptron network, Softmax j is the softmax operation for subscript j, ← is the update operation; q i 、k j 、v j They are the query vector corresponding to the i-th feature, the key vector and the value vector corresponding to the j-th feature; the feature descriptor, position embedding vector, query vector, key vector and value vector have the same number of channels, which is D.

6. The method according to claim 4, characterized in that Before acquiring the point features and line features of the image at the current moment, the method further includes: Jointly training the NeXtVLAD feature aggregation network and the point-line matching network; The joint training of the NeXtVLAD feature aggregation network and the point-line matching network includes: First, using an image dataset, virtual image pairs are constructed by adding homography transformations, and the matching relationships of point and line features within the image pairs are recovered from the homography transformations. The matching relationships of the point and line features are used as the matching truth values ​​to supervise the joint training of the network. Secondly, after the training converges, the network weights trained in the previous stage are used as the initial weights. A dataset with real camera poses, camera intrinsic parameters, images, and dense depths is used. Image pairs with larger cospicuous areas in the dataset are selected. The correct matching relationships of point and line features are obtained based on the camera poses, camera intrinsic parameters, and dense depths provided by the dataset. The correct matching relationships of point and line features are used as the matching truth values ​​for the second stage of joint training of the network. After the second stage of training converges, the obtained network weights are used as the final network weights of the NeXtVLAD feature aggregation network and the point and line matching network.

7. The method according to claim 4, characterized in that The method further comprises: If the total number of images currently collected is less than the preset number of sliding window frames, then when the next image collection moment arrives, continue to collect a new current moment image and obtain the matching relationship between the point features and line features of the new current moment image and the previous moment image; If the total number of images currently collected is equal to the preset number of sliding window frames, the initial value of the camera pose at the real scale is obtained, including: First, the SfM algorithm is used to obtain the scale-free camera pose corresponding to N frames of images. Second, the inertial measurement unit (IMU) measurement results are aligned with the camera pose obtained by the SfM algorithm to solve the scale factor, gravity direction, gyroscope bias, and camera velocity of N frames. Finally, the scale factor and the scale-free camera pose are used to obtain the camera pose corresponding to N frames of images at the real scale; where N is the preset number of sliding window frames.

8. The method according to claim 7, characterized in that The classification result of detecting the main structural direction and line features of the image at the current moment includes: For image I t All line features of the image are detected according to the direction of gravity. Among them, based on the normal vector of the line feature, the line features with the normal vector perpendicular to the gravity direction are detected and regarded as the structural lines distributed along the vertical direction. The remaining line features are clustered into horizontal line clusters to obtain image I. t All horizontal structure main directions of image I t All line features are divided into structural lines distributed along the vertical direction or the horizontal direction and non-structural lines distributed along other directions.

9. The method according to claim 8, characterized in that Optimizing the position and posture of the monocular camera includes: 1) Estimate the camera pose of the last N frames through sliding window optimization; Among them, the camera pose and velocity of the latest N frames, IMU bias, the spatial position of the point landmarks and line landmarks observed in the latest N frames, and the main direction of the structure are used as optimization variables to construct an optimization problem; the optimization problem takes the cost function L as the optimization target, and uses the Dogleg algorithm to minimize the cost function L to obtain the camera pose estimation result of the latest N frames; the cost function L is the sum of the reprojection error of the point feature, the reprojection error of the line feature, the IMU pre-integration error, the structural constraint error and the main direction observation error of the structure; the reprojection error of the point feature is the distance between the projection position of the point landmark on the image and the point feature position actually observed on the frame image, the reprojection error of the line feature is the distance between the projection position of the line landmark on the image and the end point of the line feature actually observed on the frame image, the structural constraint error is the cosine value of the angle between the main direction of the structure and the gravity direction, and the main direction observation error of the structure is the cosine value of the angle between the main direction of the structure and the normal vector of the line feature; 2) Detect whether there is a loop in the image at the current moment; Among them, for each frame image I t , using the DBoW algorithm with point feature descriptors as input to detect whether there is a loop: if there is a loop, the loop frame image I is returned old , then go to step 3); if there is no loop, the camera pose estimation result of step 1) is used as the current positioning result; 3) Optimize the camera pose corresponding to the image trajectory through loop correction; Wherein, based on the result of step 7-2), I old As I t-1 , calculate image I old and image I t Then, the PnP algorithm with RANSAC is used to determine the matching relationship between the point features and line features of image I old Relative Image I t Then, the pose value is used as the initial pose value of the loop frame, and the pose of the loop frame is inserted into the optimization variable of the optimization problem in step 1) as an additional optimization variable. old and image I t The reprojection error of the point features and line features constructed by the matching relationship of the point features and line features is added to the cost function of the optimization problem constructed in step 1), and the updated optimization problem is solved to obtain the image I old Relative Image I t The camera pose estimation result is added as a constraint item to the pose graph, and the pose graph optimization algorithm is used to optimize the pose of the image I. old To Image I t The camera pose estimation on the entire trajectory is corrected to obtain the current positioning result.

10. A visual inertial positioning device using a graph attention network to match points and lines, characterized in that: include: An image acquisition module is used to acquire the current image using the monocular camera in the visual inertial positioning system during its operation; A feature acquisition module, configured to acquire point features and line features of the image at the current moment; a feature matching module for obtaining a matching relationship between the point features and line features of the current image and the previous image using a preset point-line matching network including a graph attention mechanism based on the point features and line features of the current image; a structure detection module, configured to detect the main structural direction and classification results of the line features of the image at the current moment based on the line features of the image at the current moment when the total number of acquired images is greater than a preset number of sliding window frames; A positioning module is configured to optimize the position and posture of the monocular camera based on the matching relationship between the point features and line features of the current image and the previous image, the classification result of the main structural direction and line features of the current image, and the measurement result of the inertial measurement unit in the visual-inertial positioning system to obtain the current positioning result of the visual-inertial positioning system.

Images