Visual inertial positioning method and device for matching points and lines by using graph attention network

Abstract

The invention provides a visual inertial positioning method and device for matching points and lines by using a graph attention network, and belongs to the technical field of positioning. The method comprises the following steps: acquiring an image at a current moment by using a monocular camera in a visual inertial positioning system, extracting corresponding point features and line features, and further obtaining a matching relationship between the point features and the line features of the image at the current moment and an image at a previous moment by using a preset point-line matching network containing a graph attention mechanism; and detecting the classification result of the main structure direction and the line features of the image at the current moment, and optimizing the pose of the monocular camera in combination with the measurement result of the inertial measurement unit to obtain the current positioning result of the visual inertial positioning system. According to the method, point-line feature matching is carried out through the graph attention network, the accuracy and robustness of visual inertial positioning under the conditions of illumination variation, rapid movement and texture shortage can be improved, and the method has good engineering application value.

Description

Technical Field

[0001] The present invention belongs to the field of positioning technology, and specifically relates to a visual inertial positioning method and device using a graph attention network to match points and lines. Background Art

[0002] Visual inertial positioning is an autonomous positioning technology based on cameras and inertial measurement units (IMUs). This technology can accurately and quickly estimate the position of the device by fusing the visual information provided by the camera and the motion information provided by the IMU. This technology is committed to solving the problem of estimating the position of unmanned mobile platforms in unknown environments when external positioning technologies such as GPS and UWB (ultra-wideband) cannot be used. This technology has been widely used in fields such as autonomous driving, drones, and augmented reality, and has achieved good positioning results in various indoor and outdoor scenarios.

[0003] Compared with other positioning solutions that do not rely on external positioning devices, visual inertial positioning technology has unique advantages in sensor selection. Compared with the positioning solution based on lidar, the combination of IMU and camera is low-cost, light-weight, and easy to deploy on small mobile platforms. Compared with the positioning solution based on wheel speed meter, the addition of camera enables the positioning system to detect previously visited areas and correct positioning errors, and can also be used for non-ground wheeled robot platforms, such as AR / VR (augmented reality / virtual reality) devices and drones.

[0004] Although visual inertial positioning technology can achieve good positioning results under conditions of good lighting, rich textures, and slow motion, it still faces large positioning errors and poor robustness in scenes with changing lighting, lack of texture, and fast motion, and may even cause divergence in positioning results. In order to improve the robustness and accuracy of visual inertial positioning technology, many research works focus on the use of line features, combining point features and line features to improve the positioning accuracy in weak texture scenes. In 2019, Zou Danping et al. from Shanghai Jiao Tong University proposed a point-line visual inertial positioning method using the Atlanta world hypothesis, and tested it in an indoor environment, proving that the method has high positioning accuracy in weak texture scenes. However, this method still uses the traditional optical flow tracking method to match point-line features, and does not consider the impact of lighting changes and rapid motion on the positioning accuracy of the algorithm. In the face of large changes in lighting and perspective between frames, the robustness of the traditional point-line feature matching algorithm still needs to be further improved to further improve the positioning accuracy and robustness of visual inertial positioning technology.

[0005] In recent years, deep learning technology has made rapid progress in the field of computer vision, showing great application potential in many fields such as target detection, action recognition, semantic segmentation, etc. More and more researchers have begun to use deep learning technology to solve the problem of visual inertial positioning. In 2020, Paul-Edouard Sarlin and other researchers from the Swiss Federal Institute of Technology in Zurich used a graph attention network to match point features on two images, which can achieve accurate point feature matching under large differences in lighting and perspective. However, this method is not suitable for line feature matching and cannot be well combined with existing visual inertial positioning algorithms based on point and line features. How to better combine deep learning technology with visual inertial positioning algorithms still requires more research and solutions. Summary of the invention

[0006] The present invention aims to overcome the shortcomings of the existing technology and proposes a visual inertial positioning method and device using a graph attention network to match points and lines. The present invention uses a graph attention network to perform point and line feature matching, which can improve the accuracy and robustness of visual inertial positioning under conditions of lighting changes, rapid motion, and lack of texture, and has good engineering application value.

[0007] The first embodiment of the present invention proposes a visual inertial positioning method using a graph attention network to match points and lines, comprising:

[0008] During the operation of the visual inertial positioning system, the monocular camera in the system is used to collect the image at the current moment;

[0009] Acquire point features and line features of the image at the current moment;

[0010] Based on the point features and line features of the image at the current moment, a matching relationship between the point features and line features of the image at the previous moment is obtained by using a preset point-line matching network including a graph attention mechanism;

[0011] When the total number of acquired images is greater than the preset number of sliding window frames, based on the line features of the image at the current moment, detecting the classification results of the main structural direction and the line features of the image at the current moment;

[0012] Based on the matching relationship between the point features and line features of the current moment image and the previous moment image, the classification result of the structural main direction and line features of the current moment image, and the measurement result of the inertial measurement unit in the visual inertial positioning system, the position and posture of the monocular camera are optimized to obtain the current positioning result of the visual inertial positioning system.

[0013] In a specific embodiment of the present invention, the step of obtaining the point features and line features of the image at the current moment includes:

[0014] The current moment image I t Input the pre-trained convolutional neural network SuperPoint to obtain the dense descriptor graph corresponding to the image, the location of the point feature, the descriptor of the point feature, and the saliency of the point feature;

[0015] Use LSD line segment detection algorithm to process image I t , get the position of the line feature corresponding to the image;

[0016] Image I t The corresponding dense descriptor graph and the position of the line feature are input into the preset NeXtVLAD feature aggregation network to obtain the descriptor of the line feature corresponding to the image.

[0017] In a specific embodiment of the present invention, the point-line matching network comprises a sequentially connected position embedding subnetwork, a graph attention subnetwork, and a bidirectional Softmax matching operation module;

[0018] The position embedding subnetwork includes two parallel multi-layer perceptron networks; for the current moment image and the previous moment image, the first multi-layer perceptron network takes the position of the point feature of each frame image, the descriptor of the point feature and the saliency of the point feature as input, and outputs the position embedding vector of the point feature; the second multi-layer perceptron network takes the length of the line feature of each frame image, the direction of the line feature and the midpoint position of the line feature as input, and outputs the position embedding vector of the line feature;

[0019] The graph attention sub-network takes the descriptors of the point features and the position embedding vectors of the point features, the descriptors of the line features and the position embedding vectors of the line features of the two frames of images as input, and outputs the descriptors of the point features and the descriptors of the line features after being updated in the two frames of images;

[0020] The bidirectional Softmax matching operation module takes the descriptors of the point features and the line features of the two frames of images output by the graph attention subnetwork as input, and outputs the matching relationship between the point features and the line features of the two frames of images.

[0021] In a specific embodiment of the present invention, the graph attention subnetwork includes nine layers of attention layers connected in sequence, and each layer of attention layer contains six attention units, namely: same-frame point-point attention unit, same-frame line-line attention unit, same-frame point-line attention unit, same-frame line-point attention unit, different-frame point-point attention unit, and different-frame line-line attention unit; wherein, in each layer of attention layer, the same-frame point-point attention unit takes the position embedding vector of the point features of the two frames of images and the descriptor of the point features as input, outputs the updated point feature descriptor and sends it to the same-frame point-line attention unit; the same-frame line-line attention unit takes the position embedding vector of the line features of the two frames of images and the descriptor of the line features as input, outputs the updated line feature descriptor and sends it to the same-frame point-line attention unit; the same-frame point-line attention unit takes the position embedding vector of the point features and the line features of the two frames of images and the received position embedding vector of the line features The descriptors of the point features and line features of the two frames are taken as input, and the updated descriptors of the point features and the unupdated line features are output and sent to the same-frame line-point attention unit; the same-frame line-point attention unit takes the position embedding vectors of the point features and line features of the two frames of images and the received descriptors of the point features and line features as input, outputs the unupdated descriptors of the point features and sends them to the different-frame point-point attention unit, and at the same time outputs the updated descriptors of the line features and sends them to the different-frame line-line attention unit; the different-frame point-point attention unit takes the position embedding vectors of the point features of the two frames of images and the received descriptors of the point features as input, and outputs the descriptors of the point features that are finally updated after passing through this layer of attention layer; the different-frame line-line attention unit takes the position embedding vectors of the line features of the two frames of images and the received descriptors of the line features as input, and outputs the descriptors of the line features that are finally updated after passing through this layer of attention layer.

[0022] In a specific embodiment of the present invention, the attention unit uses an attention operation, which is expressed as follows:

[0023]

[0024] Among them, i and j are the feature subscripts, which are taken from two subscript sets A and B respectively, and Γ contains all feature subscript combinations constructed by these two subscript sets; f i is the descriptor of the i-th feature, and MLP is a i and attention vector a(i,Γ) as input multi-layer perceptron network, Softmax j is the softmax operation for index j, ← is the update operation; q i , k j 、v j They are the query vector corresponding to the i-th feature, the key vector and the value vector corresponding to the j-th feature; the feature descriptor, position embedding vector, query vector, key vector and value vector have the same number of channels, which is D.

[0025] In a specific embodiment of the present invention, before acquiring the point features and line features of the image at the current moment, the method further includes:

[0026] Jointly training the NeXtVLAD feature aggregation network and the point-line matching network;

[0027] The joint training of the NeXtVLAD feature aggregation network and the point-line matching network includes:

[0028] Firstly, a picture dataset is used to construct virtual picture pairs by adding homography transformation, and the matching relationship of point and line features in the picture pairs is restored from the homography transformation. The matching relationship of the point and line features is used as the matching truth value to supervise the joint training of the network. Secondly, after the training converges, the network weights completed in the previous stage of training are used as the initial weights. A dataset with real camera pose, camera intrinsic parameters, pictures and dense depth is used. Picture pairs with larger cosight areas in the dataset are selected. The correct matching relationship of point and line features is obtained according to the camera pose, camera intrinsic parameters and dense depth provided by the dataset. The correct matching relationship of point and line features is used as the matching truth value for the second stage of network joint training. After the second stage of training converges, the obtained network weights are used as the final network weights of the NeXtVLAD feature aggregation network and the point and line matching network.

[0029] In a specific embodiment of the present invention, the method further comprises:

[0030] If the total number of images currently collected is less than the preset number of sliding window frames, when the next image collection moment arrives, continue to collect new current moment images and obtain the matching relationship between the point features and line features of the new current moment image and the previous moment image;

[0031] If the total number of images currently collected is equal to the preset number of sliding window frames, the initial value of the camera pose at the real scale is obtained, including:

[0032] First, the SfM algorithm is used to obtain the scale-free camera pose corresponding to N frames of images. Secondly, the inertial measurement unit (IMU) measurement results are used to align with the camera pose obtained by the SfM algorithm to solve the scale factor, gravity direction, gyroscope bias and camera velocity of N frames. Finally, the scale factor and the scale-free camera pose are used to obtain the camera pose corresponding to N frames of images at the real scale. N is the preset number of sliding window frames.

[0033] In a specific embodiment of the present invention, the classification result of detecting the structural main direction and line features of the image at the current moment includes:

[0034] For image I tAll line features of the image are detected according to the direction of gravity, and the structural line features that meet the Atlanta world hypothesis are detected; among them, based on the normal vector of the line feature, the line features whose normal vector is perpendicular to the gravity direction are detected and used as the structural lines distributed along the vertical direction; the remaining line features are clustered into horizontal line clusters to obtain image I t All horizontal structure main directions; finally, image I t All line features are divided into structural lines distributed along the vertical direction or the horizontal direction and non-structural lines distributed along other directions.

[0035] In a specific embodiment of the present invention, the optimizing the posture of the monocular camera includes:

[0036] 1) Estimate the camera pose of the most recent N frames through sliding window optimization;

[0037] Among them, the camera pose and speed of the latest N frames, IMU bias, the spatial position of the point landmarks and line landmarks observed in the latest N frames, and the main direction of the structure are used as optimization variables to construct an optimization problem; the optimization problem takes the cost function L as the optimization target, and uses the Dogleg algorithm to minimize the cost function L to obtain the camera pose estimation results of the latest N frames; the cost function L is the sum of the reprojection error of the point feature, the reprojection error of the line feature, the IMU pre-integration error, the structural constraint error and the main direction observation error of the structure; the reprojection error of the point feature is the distance between the projection position of the point landmark on the image and the point feature position actually observed on the frame image, the reprojection error of the line feature is the distance between the projection position of the line landmark on the image and the line feature endpoint actually observed on the frame image, the structural constraint error is the cosine value of the angle between the main direction of the structure and the gravity direction, and the main direction observation error of the structure is the cosine value of the angle between the main direction of the structure and the normal vector of the line feature;

[0038] 2) Detect whether there is a loop in the image at the current moment;

[0039] Among them, for each frame image I t , using the DBoW algorithm with point feature descriptors as input to detect whether there is a loop: If there is a loop, the loop frame image I is returned old , then go to step 3); if there is no loop, the camera pose estimation result of step 1) is used as the current positioning result;

[0040] 3) Through loop correction, the camera pose corresponding to the image trajectory is optimized;

[0041] Wherein, based on the result of step 7-2), I old As I t-1 , calculate image I old and image I tThen, the PnP algorithm with RANSAC is used to determine the matching relationship between the point features and line features of image I old Relative Image I t The pose value is then used as the initial pose value of the loop frame, and the pose of the loop frame is inserted into the optimization variable of the optimization problem in step 1) as an additional optimization variable. old and image I t The reprojection error of the point features and line features constructed by the matching relationship of the point features and line features is added to the cost function of the optimization problem constructed in step 1), and the updated optimization problem is solved to obtain image I old Relative Image I t The camera pose estimation result is obtained by adding the estimation result as a constraint item to the pose graph, and the pose graph optimization algorithm is used to optimize the pose graph from image I old To Image I t The camera pose estimation on the entire trajectory is corrected to obtain the current positioning result.

[0042] The second aspect of the present invention provides a visual inertial positioning device for matching points and lines using a graph attention network, comprising:

[0043] An image acquisition module is used to acquire the current moment image using the monocular camera in the visual inertial positioning system during its operation;

[0044] A feature acquisition module, used to acquire point features and line features of the image at the current moment;

[0045] A feature matching module, configured to obtain a matching relationship between the point features and line features of the image at the current moment and the image at the previous moment by using a preset point-line matching network including a graph attention mechanism based on the point features and line features of the image at the current moment;

[0046] A structure detection module, used for detecting the main structural direction and classification result of the line features of the image at the current moment based on the line features of the image at the current moment when the total number of acquired images is greater than the preset number of sliding window frames;

[0047] A positioning module is used to optimize the position and posture of the monocular camera based on the matching relationship between the point features and line features of the image at the current moment and the image at the previous moment, the classification result of the structural main direction and line features of the image at the current moment, and the measurement result of the inertial measurement unit in the visual inertial positioning system to obtain the current positioning result of the visual inertial positioning system.

[0048] A third aspect of the present invention provides an electronic device, including:

[0049] at least one processor; and a memory communicatively coupled to the at least one processor;

[0050] Among them, the memory stores instructions that can be executed by the at least one processor, and the instructions are configured to execute the above-mentioned visual inertial positioning method for matching points and lines using a graph attention network.

[0051] The fourth aspect of the present invention proposes a computer-readable storage medium, which stores computer instructions, and the computer instructions are used to enable the computer to execute the above-mentioned visual inertial positioning method for matching points and lines using a graph attention network.

[0052] The characteristics and beneficial effects of the present invention are:

[0053] The present invention has good positioning accuracy and robustness under the conditions of illumination change, rapid motion and lack of texture. The present invention matches the point and line features of two frames of images through the graph attention network. When there are large differences in the brightness and observation angle of the two images, the matching can still be completed, and it has good robustness. In the pose estimation, the point and line feature observation, the main direction observation of the structure and the IMU measurement are integrated, and the available sensor measurement information and the structural characteristics of the environment can be used to the maximum extent for positioning, which effectively improves the positioning accuracy in weak texture scenes. The present invention is mainly used for the positioning task of small mobile devices in indoor environments, and can provide the positioning information relative to the starting point when GPS information cannot be obtained. The present invention also has three advantages: no complex environment mapping preparation process is required, and it can be used in unfamiliar environments; the system sensor only needs a monocular camera and an IMU, and there are no complex equipment requirements; it has good adaptability to various operating environments. The above advantages make the present invention relatively convenient to deploy on small mobile systems, and the required cost is also relatively low. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] Figure 1 It is an overall flow chart of a visual inertial positioning method using a graph attention network to match points and lines according to an embodiment of the present invention.

[0055] Figure 2 It is a schematic diagram of the middle attention sub-network in a specific embodiment of the present invention. DETAILED DESCRIPTION

[0056] The present invention proposes a visual inertial positioning method and device using a graph attention network to match points and lines. The present invention is described in detail below in conjunction with the accompanying drawings and specific embodiments.

[0057] The first embodiment of the present invention proposes a visual inertial positioning method using a graph attention network to match points and lines, comprising:

[0058] During the operation of the visual inertial positioning system, the monocular camera in the system is used to collect the image at the current moment;

[0059] Acquire point features and line features of the image at the current moment;

[0060] Based on the point features and line features of the image at the current moment, a matching relationship between the point features and line features of the image at the previous moment is obtained by using a preset point-line matching network including a graph attention mechanism;

[0061] When the total number of acquired images is greater than the preset number of sliding window frames, based on the line features of the image at the current moment, detecting the classification results of the main structural direction and the line features of the image at the current moment;

[0062] Based on the matching relationship between the point features and line features of the current moment image and the previous moment image, the classification result of the structural main direction and line features of the current moment image, and the measurement result of the inertial measurement unit in the visual inertial positioning system, the position and posture of the monocular camera are optimized to obtain the current positioning result of the visual inertial positioning system.

[0063] In a specific embodiment of the present invention, the visual inertial positioning method using a graph attention network to match points and lines has the following overall process: Figure 1 As shown, the following steps are included:

[0064] 1) Building a visual inertial positioning system, including: a monocular camera, an inertial measurement unit (IMU) and an onboard computer; wherein the monocular camera and the IMU need to be fixed on the same bracket, the camera frame rate is 10 to 30 Hz (20 Hz in a specific embodiment of the present invention), and the IMU frequency is 100 to 400 Hz (200 Hz in this embodiment); each component of the visual inertial positioning system described in this embodiment can adopt a conventional model.

[0065] 2) The current moment is recorded as time t, and the image collected by the monocular camera in the visual inertial positioning system built in step 1) is recorded as I t .

[0066] 3) Image I obtained in step 2) t Processing is performed to obtain the matching relationship between the point features and line features of the current moment image and the previous moment image; the specific steps are as follows:

[0067] 3-1) For image I t Perform point and line feature extraction.

[0068] Specifically, the pre-trained convolutional neural network SuperPoint is used to process image I t, get the dense description sub-graph corresponding to the image, the location of the point feature, the descriptor of the point feature and the saliency of the point feature; then use the LSD (line segment detection) algorithm to process the image I t , get the position of the line feature corresponding to the image; then, image I t The corresponding dense description sub-graph and the position of the line feature are input into the preset NeXtVLAD feature aggregation network to obtain the descriptor of the line feature corresponding to the image; the image I t The position of the point feature, the descriptor of the point feature, the saliency of the point feature, the position of the line feature and the descriptor of the line feature are saved as the point and line feature information of the image.

[0069] 3-2) Judgment:

[0070] If t is the initial time, then when the next image acquisition time arrives, return to step 2); otherwise, go to step 3-3).

[0071] 3-3) Using Image I t The image I at the previous moment t-1 The point and line feature information of the two images is used to calculate the matching relationship between the point features and line features of the two images.

[0072] Specifically, a preset point-line matching network is used to obtain image I t and the previous moment image I t-1 The position of the point feature, the descriptor of the point feature, the saliency of the point feature, the position of the line feature and the descriptor of the line feature are taken as input to obtain the matching relationship between the point features and the line features of the two frames of images.

[0073] In this embodiment, the point-line matching network includes three parts: a sequentially connected position embedding subnetwork, a graph attention subnetwork, and a bidirectional Softmax matching operation module.

[0074] The position embedding subnetwork includes two parallel multilayer perceptron networks; the first multilayer perceptron network (in this embodiment, the input layer dimension is 4, the output layer dimension is 256, and the hidden layer dimensions are 32, 64, and 128 respectively) takes the position of the point feature of each frame image, the descriptor of the point feature and the saliency of the point feature as input, and outputs the position embedding vector of the point feature; the second multilayer perceptron network (in this embodiment, the input layer dimension is 5, the output layer dimension is 256, and the hidden layer dimensions are 32, 64, and 128 respectively) takes the length of the line feature of each frame image, the direction of the line feature, and the midpoint position of the line feature as input, and outputs the position embedding vector of the line feature.

[0075] The principle of the graph attention sub-network is as follows Figure 2As shown, the graph attention subnetwork takes the descriptors of the point features and the position embedding vectors of the point features, the descriptors of the line features and the position embedding vectors of the line features of the two frames of images as input, and outputs the descriptors of the point features and the line features after update of the two frames of images. Specifically, in this embodiment, the graph attention subnetwork includes nine sequentially connected attention layers, and each attention layer contains six attention units, namely: same-frame point-to-point attention unit, same-frame line-to-line attention unit, same-frame point-to-line attention unit, same-frame line-to-point attention unit, different-frame point-to-point attention unit, and different-frame line-to-line attention unit. In the same attention layer, the connection relationship between the attention units is as follows: Figure 2As shown, specifically, the outputs of the same-frame point-point attention unit and the same-frame line-line attention unit are respectively connected to the input of the same-frame point-line attention unit; the output of the same-frame point-line attention unit is connected to the input of the same-frame line-point attention unit; the output of the same-frame line-point attention unit is respectively connected to the input of the different-frame point-point attention unit and the input of the different-frame line-line attention unit, and the outputs of the different-frame point-point attention unit and the different-frame line-line attention unit are the output of the attention layer. Among them, in each attention layer, the point-to-point attention unit in the same frame takes the position embedding vector of the point features of the two frames of images and the descriptor of the point features as input, outputs the updated descriptor of the point features and sends it to the point-to-line attention unit in the same frame; the line-to-line attention unit in the same frame takes the position embedding vector of the line features of the two frames of images and the descriptor of the line features as input, outputs the updated descriptor of the line features and sends it to the point-to-line attention unit in the same frame; the point-to-line attention unit in the same frame takes the position embedding vector of the point features and line features of the two frames of images and the descriptors of the received point features and line features as input, outputs the updated descriptor of the point features and the descriptor of the line features that have not been updated and sends them to the line-to-point attention unit in the same frame; The point attention unit takes the position embedding vectors of the point features and line features of the two frames of images and the received descriptors of the point features and line features as input, outputs the unupdated descriptors of the point features to be sent to the different frame point attention unit, and outputs the updated descriptors of the line features to be sent to the different frame line attention unit; the different frame point attention unit takes the position embedding vectors of the point features of the two frames of images and the received descriptors of the point features as input, outputs the descriptors of the point features finally updated after the attention layer; the different frame line attention unit takes the position embedding vectors of the line features of the two frames of images and the received descriptors of the line features as input, and outputs the descriptors of the line features finally updated after the attention layer. It should be noted that the position embedding vectors input to the attention units of all attention layers come from the position embedding subnetwork; the descriptors of the point features input to the same frame point attention unit and the line features input to the same frame line attention unit of the first attention layer come from the corresponding descriptors obtained in step 3-1). The heterogeneous point attention unit of the ninth attention layer outputs the descriptor of the point feature after the final update of the graph attention sub-network, and the heterogeneous line attention unit of the ninth attention layer outputs the descriptor of the line feature after the final update of the graph attention sub-network.

[0076] Furthermore, each attention unit uses the attention operation to update the descriptors of the point features and the line features. The following formula describes the process of the attention operation:

[0077]

[0078] Among them, i and j are the subscripts of the feature, taken from two subscript sets A and B respectively, Γ contains all the feature subscript combinations constructed by these two subscript sets, fi is the descriptor of the i-th feature (a point feature or a line feature), MLP is a multi-layer perceptron network with fi and attention vector a(i,Γ) as input, Softmaxj is the softmax operation for subscript j, ← is the update operation; qi, kj, vj are the query vector corresponding to the i-th feature, the key vector corresponding to the j-th feature, and the value vector, which are generated by the multi-layer perceptron network with the descriptor and position embedding vector of the feature as input. The descriptor, position embedding vector, query vector, key vector, and value vector of the feature have the same number of channels, which is D. In particular, different sublayers operate between different feature index sets A and B. Specifically, A and B of the same-frame point-to-point attention unit are all point features in the same image, A and B of the same-frame line-to-line attention unit are all line features in the same image, A and B of the same-frame point-to-line attention unit are all point features and all line features in the same image, A and B of the same-frame line-to-point attention unit are all line features and all point features in the same image, A and B of the different-frame point-to-point attention unit are all point features from different images, A and B of the different-frame line-to-line attention unit are all line features from different images. The descriptor of the feature output by the ninth attention layer is the final output of the graph attention subnetwork.

[0079] The bidirectional Softmax matching operation module takes the descriptors of the point features and the line features of the two frames of images output by the graph attention subnetwork as input, and outputs the matching relationship of all the point features and the matching relationship of the line features on the two frames of images. Specifically, this embodiment adopts the bidirectional Softmax matching method used by the SuperGlue network, calculates the matching score of the feature by the dot product of the descriptor, and uses the bidirectional softmax operation to obtain the matching probability of the feature, and finally takes the feature pair with the largest mutual matching probability as the matching feature, so as to obtain the matching relationship of the point features and line features on the two frames of images.

[0080] 4) Determination: If the total number of images currently collected is less than N frames, then when the next image collection time arrives, return to step 2); if the total number of images currently collected is equal to N frames (N is 10 in this embodiment), then go to step 5); if the total number of images currently collected is greater than N frames, then go to step 6); wherein N is the preset number of sliding window frames.

[0081] 5) Obtain the initial value of the camera pose at the real scale.

[0082] Specifically, the initialization method of VINS-Mono is used to initialize the gravity direction, gyroscope bias, camera pose and velocity of N frames.

[0083] In this embodiment, first, the SfM algorithm (Structure from Motion) is used to obtain the scale-free camera pose corresponding to N frames of images; secondly, the IMU measurement results are used to align with the camera pose obtained by the SfM algorithm to solve the scale factor, gravity direction, gyroscope bias and camera speed of N frames; finally, the scale factor and the scale-free camera pose are used to obtain the camera pose corresponding to N frames of images at the real scale.

[0084] The quantities solved in this step include: gravity direction, gyroscope bias, and camera position and velocity of N frames, which will be saved as initial values ​​for structural characteristic detection and sliding window optimization in subsequent steps 6) and 7).

[0085] 6) Based on the line features of the image at the current moment, detect the main structural direction of the image at the current moment and the classification results of the line features.

[0086] In this embodiment, the image I t All line features are used to detect structural line features that conform to the Atlanta world hypothesis when the direction of gravity is known. Specifically: First, based on the normal vector of the line feature, the line features whose normal vector is perpendicular to the direction of gravity are detected and used as structural lines distributed along the vertical direction. Then, for the remaining line features, the MnS (mined-and-stabbed) algorithm proposed in the paper S-VIO: Exploiting Structural Constraints for RGB-D Visual Inertial Odometry is used to cluster horizontal line clusters. For each line feature, the algorithm calculates the horizontal subinterval to which it may belong based on its normal vector, and extracts the subinterval with a large number of subordinate line features as the main direction of the detected horizontal structure. The line features involved in these subintervals are the structural lines distributed along a certain horizontal direction. In this way, image I is obtained. t All horizontal structure main directions observed, and image I t All line features are divided into structural lines distributed along the vertical direction or along a certain horizontal direction and non-structural lines distributed along other directions.

[0087] 7) Based on the point-line feature matching relationship obtained in step 3), the main structural direction detection result obtained in step 5) (i.e., the number and direction of the main horizontal structural directions) and the classification result of the line feature (i.e., whether it is a structural line, and if so, whether it is distributed along the vertical direction or along which horizontal structural main direction it is distributed), combined with the IMU measurement results, the camera's position and posture are determined.

[0088] Specifically, this step draws on the VINS-Mono algorithm to perform pose estimation. The specific steps are as follows:

[0089] 7-1) Estimate the camera pose of the most recent N frames through sliding window optimization.

[0090] In this embodiment, the camera pose and speed of the latest N frames, IMU bias, the spatial position of the point landmarks and line landmarks observed in the latest N frames (the initial value is obtained by the triangulation method, the specific method is the same as that used in VINS-Mono), and the main direction of the structure are used as optimization variables to construct an optimization problem. This optimization problem takes the cost function L as the optimization target, and uses the Dogleg algorithm to minimize the cost function L to obtain the camera pose estimation results of the latest N frames. The cost function L is the sum of five parts, namely the reprojection error of the point feature (defined as the distance between the projection position of the point landmark on the image and the point feature position actually observed on the frame image), the reprojection error of the line feature (defined as the distance between the projection position of the line landmark on the image and the line feature endpoint actually observed on the frame image), the IMU pre-integration error, the structural constraint error (defined as the cosine value of the angle between the main direction of the structure and the gravity direction) and the main direction observation error of the structure (defined as the cosine value of the angle between the main direction of the structure and the normal vector of the line feature).

[0091] 7-2) Check whether there is a loop in the image at the current moment.

[0092] In this embodiment, for each frame image I t , using the DBoW (Distributed Bag of Words) algorithm with point feature descriptors as input to detect whether there is a loop: if there is a loop, the loop frame image I is returned old , then go to step 7-3); if there is no loop, the camera pose estimation result of step 7-1) is used as the current positioning result.

[0093] 7-3) Through loop correction, the camera pose corresponding to the image trajectory is optimized.

[0094] In this embodiment, based on the result of step 7-2), I old As I t-1 , repeat step 3-3) to calculate image I old and image I t The matching relationship between point features and line features.

[0095] After getting the image I old and image I t After matching the point features and line features, the PnP (Perspective-n-Points) algorithm with RANSAC is used to determine the image I old Relative Image I tThen, the pose value is used as the initial pose value of the loop frame, and the pose of the loop frame is inserted into the optimization variable list of the optimization problem in step 7-1) as an additional optimization variable. old and image I t The reprojection errors of the point features and line features constructed by the matching relationship of the point features and line features are added to the cost function of the optimization problem constructed in step 6-1), and the updated optimization problem is solved. The camera pose estimation result of the loop frame is further optimized to obtain image I old Relative Image I t Finally, the camera pose estimation result is added as a constraint item to the pose graph, and the pose graph optimization algorithm is used to optimize the pose of the image I. old To Image I t The camera pose estimation on the entire trajectory is corrected to obtain the current positioning result.

[0096] It should be noted that the result of step 7-1) is used to provide the initial value of step 7-3), that is, the estimated value of the camera pose on the entire trajectory (from time zero to the current time). In this embodiment, step 7-1) is to optimize the camera pose of the most recent N frames, and only considers the observation information of the most recent period, while step 7-3) is to optimize the camera pose of the entire trajectory from the loop frame to the current frame by using the visual feature matching information between the loop frame and the current frame when a loop is detected.

[0097] 8) Save the camera pose optimization result obtained in step 7) and use it as the initial pose value for the next positioning. When the next image acquisition time arrives, return to step 2) to achieve continuous positioning of the visual inertial positioning system.

[0098] In particular, in the method described in this embodiment, before step 2), it is necessary to jointly train the NeXtVLAD feature aggregation network and the point-line matching network, and the training method is the same as the training method used in the paper GlueStick: Robust Image Matching by Sticking Points and Lines Together. Specifically, the training method is divided into two stages. First, a picture data set is used (the present embodiment uses the revisitop data set), and a virtual picture pair is constructed by adding a homography transformation, and the matching relationship of the point-line features in the picture pair is restored from the homography transformation, and the matching relationship of the point-line features is used as the matching truth value to supervise the network joint training. Secondly, after the model training converges, the network weights completed in the previous stage of training are used as the initial weights, and a data set with real camera pose, camera intrinsic parameters, pictures and dense depth is used (the present embodiment uses the megadepth data set), and the picture pairs with larger common viewing areas in the data set are selected, and the correct matching relationship of the point-line features is obtained according to the camera pose, camera intrinsic parameters, and dense depth provided by the data set, and the matching relationship of the point-line features is used as the matching truth value to carry out the second stage of network joint training. After the second stage of training converges, the obtained network weights are used as the network weights of the NeXtVLAD feature aggregation network and the point-line matching network when they are applied.

[0099] A specific embodiment of the present invention is deployed on a desktop computer with an Intel Core i9 processor, a main frequency of 3.6 GHz, 32 GB of memory, and an Nvidia RTX3060 GPU (with a video memory size of 12 GB) to run the neural network. The method described in this embodiment is implemented based on the C++ language and uses the TensorRT library to accelerate the reasoning of the neural network.

[0100] In order to test the positioning accuracy of the method described in this embodiment in a real environment, this embodiment uses the public TUM-VI dataset and UMA-VI dataset. Both of the above datasets are handheld collection, and the sensor data is given in the form of rosbag, including monocular image data with a frame rate of 30fps and IMU data with a frequency of 200Hz. Both IMU data and monocular image data are accompanied by timestamps, and the timestamps are located on the same timeline. The two datasets contain a total of 53 test sequences, and the length of each sequence ranges from 4 minutes to 16 minutes. The recording environment includes typical scenes such as corridors, classrooms, stairwells, halls, offices, laboratories, roads, and parking lots. During the test, we used monocular images with a frame rate of 10fps and IMU data of 200Hz as algorithm inputs. The average time consumption of each module of the statistical algorithm showed that feature detection and matching took 55ms, sliding window optimization took 20ms, and loop correction: word bag recall took 20ms, if a loop frame was successfully recalled, feature matching on the loop frame took 45ms, PnP algorithm took 32ms, and pose graph optimization time depended on the length of the trajectory to be optimized (40-200ms). Since feature detection and matching, sliding window optimization and loop correction are executed in parallel in three threads, the method described in this embodiment meets the real-time requirements of operation as a whole.

[0101] To implement the above embodiment, the second aspect of the present invention proposes a visual inertial positioning device using a graph attention network to match points and lines, comprising:

[0102] An image acquisition module is used to acquire the current moment image using the monocular camera in the visual inertial positioning system during its operation;

[0103] A feature acquisition module, used to acquire point features and line features of the image at the current moment;

[0104] A feature matching module, configured to obtain a matching relationship between the point features and line features of the image at the current moment and the image at the previous moment by using a preset point-line matching network including a graph attention mechanism based on the point features and line features of the image at the current moment;

[0105] A structure detection module, used for detecting the main structural direction and classification result of the line features of the image at the current moment based on the line features of the image at the current moment when the total number of acquired images is greater than the preset number of sliding window frames;

[0106] A positioning module is used to optimize the position and posture of the monocular camera based on the matching relationship between the point features and line features of the image at the current moment and the image at the previous moment, the classification result of the structural main direction and line features of the image at the current moment, and the measurement result of the inertial measurement unit in the visual inertial positioning system to obtain the current positioning result of the visual inertial positioning system.

[0107] In a specific embodiment of the present invention, the step of obtaining the point features and line features of the image at the current moment includes:

[0108] The current moment image I t Input the pre-trained convolutional neural network SuperPoint to obtain the dense descriptor graph corresponding to the image, the location of the point feature, the descriptor of the point feature, and the saliency of the point feature;

[0109] Use LSD line segment detection algorithm to process image I t , get the position of the line feature corresponding to the image;

[0110] Image I t The corresponding dense descriptor graph and the position of the line feature are input into the preset NeXtVLAD feature aggregation network to obtain the descriptor of the line feature corresponding to the image.

[0111] In a specific embodiment of the present invention, the point-line matching network comprises a sequentially connected position embedding subnetwork, a graph attention subnetwork, and a bidirectional Softmax matching operation module;

[0112] The position embedding subnetwork includes two parallel multi-layer perceptron networks; for the current moment image and the previous moment image, the first multi-layer perceptron network takes the position of the point feature of each frame image, the descriptor of the point feature and the saliency of the point feature as input, and outputs the position embedding vector of the point feature; the second multi-layer perceptron network takes the length of the line feature of each frame image, the direction of the line feature and the midpoint position of the line feature as input, and outputs the position embedding vector of the line feature;

[0113] The graph attention sub-network takes the descriptors of the point features and the position embedding vectors of the point features, the descriptors of the line features and the position embedding vectors of the line features of the two frames of images as input, and outputs the descriptors of the point features and the descriptors of the line features after being updated in the two frames of images;

[0114] The bidirectional Softmax matching operation module takes the descriptors of the point features and the line features of the two frames of images output by the graph attention subnetwork as input, and outputs the matching relationship between the point features and the line features of the two frames of images.

[0115] In a specific embodiment of the present invention, the graph attention subnetwork includes nine layers of attention layers connected in sequence, and each layer of attention layer contains six attention units, namely: same-frame point-point attention unit, same-frame line-line attention unit, same-frame point-line attention unit, same-frame line-point attention unit, different-frame point-point attention unit, and different-frame line-line attention unit; wherein, in each layer of attention layer, the same-frame point-point attention unit takes the position embedding vector of the point features of the two frames of images and the descriptor of the point features as input, outputs the updated point feature descriptor and sends it to the same-frame point-line attention unit; the same-frame line-line attention unit takes the position embedding vector of the line features of the two frames of images and the descriptor of the line features as input, outputs the updated line feature descriptor and sends it to the same-frame point-line attention unit; the same-frame point-line attention unit takes the position embedding vector of the point features and the line features of the two frames of images and the received position embedding vector of the line features The descriptors of the point features and line features of the two frames are taken as input, and the updated descriptors of the point features and the unupdated line features are output and sent to the same-frame line-point attention unit; the same-frame line-point attention unit takes the position embedding vectors of the point features and line features of the two frames of images and the received descriptors of the point features and line features as input, outputs the unupdated descriptors of the point features and sends them to the different-frame point-point attention unit, and at the same time outputs the updated descriptors of the line features and sends them to the different-frame line-line attention unit; the different-frame point-point attention unit takes the position embedding vectors of the point features of the two frames of images and the received descriptors of the point features as input, and outputs the descriptors of the point features that are finally updated after passing through this layer of attention layer; the different-frame line-line attention unit takes the position embedding vectors of the line features of the two frames of images and the received descriptors of the line features as input, and outputs the descriptors of the line features that are finally updated after passing through this layer of attention layer.

[0116] In a specific embodiment of the present invention, the attention unit uses an attention operation, which is expressed as follows:

[0117]

[0118] Among them, i and j are the feature subscripts, which are taken from two subscript sets A and B respectively, and Γ contains all feature subscript combinations constructed by these two subscript sets; f i is the descriptor of the i-th feature, and MLP is a i and attention vector a(i,Γ) as input multi-layer perceptron network, Softmax j is the softmax operation for index j, ← is the update operation; q i , k j 、v j They are the query vector corresponding to the i-th feature, the key vector and the value vector corresponding to the j-th feature; the feature descriptor, position embedding vector, query vector, key vector and value vector have the same number of channels, which is D.

[0119] In a specific embodiment of the present invention, before acquiring the point features and line features of the image at the current moment, the method further includes:

[0120] Jointly training the NeXtVLAD feature aggregation network and the point-line matching network;

[0121] The joint training of the NeXtVLAD feature aggregation network and the point-line matching network includes:

[0122] Firstly, a picture dataset is used to construct virtual picture pairs by adding homography transformation, and the matching relationship of point and line features in the picture pairs is restored from the homography transformation. The matching relationship of the point and line features is used as the matching truth value to supervise the joint training of the network. Secondly, after the training converges, the network weights completed in the previous stage of training are used as the initial weights. A dataset with real camera pose, camera intrinsic parameters, pictures and dense depth is used. Picture pairs with larger cosight areas in the dataset are selected. The correct matching relationship of point and line features is obtained according to the camera pose, camera intrinsic parameters and dense depth provided by the dataset. The correct matching relationship of point and line features is used as the matching truth value for the second stage of network joint training. After the second stage of training converges, the obtained network weights are used as the final network weights of the NeXtVLAD feature aggregation network and the point and line matching network.

[0123] In a specific embodiment of the present invention, it also includes:

[0124] If the total number of images currently collected is less than the preset number of sliding window frames, when the next image collection moment arrives, continue to collect new current moment images and obtain the matching relationship between the point features and line features of the new current moment image and the previous moment image;

[0125] If the total number of images currently collected is equal to the preset number of sliding window frames, the initial value of the camera pose at the real scale is obtained, including:

[0126] First, the SfM algorithm is used to obtain the scale-free camera pose corresponding to N frames of images. Secondly, the inertial measurement unit (IMU) measurement results are used to align with the camera pose obtained by the SfM algorithm to solve the scale factor, gravity direction, gyroscope bias and camera velocity of N frames. Finally, the scale factor and the scale-free camera pose are used to obtain the camera pose corresponding to N frames of images at the real scale. N is the preset number of sliding window frames.

[0127] In a specific embodiment of the present invention, the classification result of detecting the structural main direction and line features of the image at the current moment includes:

[0128] For image I tAll line features of the image are detected according to the direction of gravity, and the structural line features that meet the Atlanta world hypothesis are detected; among them, based on the normal vector of the line feature, the line features whose normal vector is perpendicular to the gravity direction are detected and used as the structural lines distributed along the vertical direction; the remaining line features are clustered into horizontal line clusters to obtain image I t All horizontal structure main directions; finally, image I t All line features are divided into structural lines distributed along the vertical direction or the horizontal direction and non-structural lines distributed along other directions.

[0129] In a specific embodiment of the present invention, the optimizing the posture of the monocular camera includes:

[0130] 1) Estimate the camera pose of the most recent N frames through sliding window optimization;

[0131] Among them, the camera pose and speed of the latest N frames, IMU bias, the spatial position of the point landmarks and line landmarks observed in the latest N frames, and the main direction of the structure are used as optimization variables to construct an optimization problem; the optimization problem takes the cost function L as the optimization target, and uses the Dogleg algorithm to minimize the cost function L to obtain the camera pose estimation results of the latest N frames; the cost function L is the sum of the reprojection error of the point feature, the reprojection error of the line feature, the IMU pre-integration error, the structural constraint error and the main direction observation error of the structure; the reprojection error of the point feature is the distance between the projection position of the point landmark on the image and the point feature position actually observed on the frame image, the reprojection error of the line feature is the distance between the projection position of the line landmark on the image and the line feature endpoint actually observed on the frame image, the structural constraint error is the cosine value of the angle between the main direction of the structure and the gravity direction, and the main direction observation error of the structure is the cosine value of the angle between the main direction of the structure and the normal vector of the line feature;

[0132] 2) Detect whether there is a loop in the image at the current moment;

[0133] Among them, for each frame image I t , using the DBoW algorithm with point feature descriptors as input to detect whether there is a loop: If there is a loop, the loop frame image I is returned old , then go to step 3); if there is no loop, the camera pose estimation result of step 1) is used as the current positioning result;

[0134] 3) Through loop correction, the camera pose corresponding to the image trajectory is optimized;

[0135] Wherein, based on the result of step 7-2), I old As I t-1 , calculate image I old and image I tThen, the PnP algorithm with RANSAC is used to determine the matching relationship between the point features and line features of image I old Relative Image I t The pose value is then used as the initial pose value of the loop frame, and the pose of the loop frame is inserted into the optimization variable of the optimization problem in step 1) as an additional optimization variable. old and image I t The reprojection error of the point features and line features constructed by the matching relationship of the point features and line features is added to the cost function of the optimization problem constructed in step 1), and the updated optimization problem is solved to obtain image I old Relative Image I t The camera pose estimation result is obtained by adding the estimation result as a constraint item to the pose graph, and the pose graph optimization algorithm is used to optimize the pose graph from image I old To Image I t The camera pose estimation on the entire trajectory is corrected to obtain the current positioning result.

[0136] This makes it possible to match point and line features through a graph attention network, improving the accuracy and robustness of visual inertial positioning in situations of lighting changes, rapid motion, and lack of texture, and has good engineering application value.

[0137] To implement the above embodiment, a third aspect of the present invention provides an electronic device, including:

[0138] at least one processor; and a memory communicatively coupled to the at least one processor;

[0139] Among them, the memory stores instructions that can be executed by the at least one processor, and the instructions are configured to execute the above-mentioned visual inertial positioning method for matching points and lines using a graph attention network.

[0140] To implement the above-mentioned embodiments, the fourth aspect of the present invention proposes a computer-readable storage medium, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable the computer to execute the above-mentioned visual-inertial positioning method for matching points and lines using a graph attention network.

[0141] It should be noted that the computer-readable medium disclosed above may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in combination with an instruction execution system, device or device. In the present disclosure, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, in which a computer-readable program code is carried. This propagated data signal may take a variety of forms, including but not limited to an electromagnetic signal, an optical signal, or any suitable combination of the above. The computer readable signal medium may also be any computer readable medium other than a computer readable storage medium, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer readable medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

[0142] The computer-readable medium may be included in the electronic device, or may exist independently without being installed in the electronic device. The computer-readable medium carries one or more programs. When the one or more programs are executed by the electronic device, the electronic device executes a visual-inertial positioning method for matching points and lines using a graph attention network according to the above embodiment.

[0143] Computer program code for performing the operations of the present disclosure may be written in one or more programming languages, or a combination thereof, including object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., through the Internet using an Internet service provider).

[0144] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, without contradiction.

[0145] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In the description of this application, the meaning of "plurality" is at least two, such as two, three, etc., unless otherwise clearly and specifically defined.

[0146] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, segment or portion of code that includes one or more executable instructions for implementing the steps of a specific logical function or process, and the scope of the preferred embodiments of the present application includes alternative implementations in which functions may not be performed in the order shown or discussed, including performing functions in a substantially simultaneous manner or in the reverse order depending on the functions involved, which should be understood by technicians in the technical field to which the embodiments of the present application belong.

[0147] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute the instructions), or in combination with these instruction execution systems, devices or apparatuses. For the purpose of this specification, "computer-readable medium" can be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in combination with these instruction execution systems, devices or apparatuses. More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection with one or more wires (electronic device), a portable computer disk box (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), a fiber optic device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium and then editing, interpreting or otherwise processing in a suitable manner if necessary, and then stored in a computer memory.

[0148] It should be understood that the various parts of the present application can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, multiple steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0149] A person skilled in the art may understand that all or part of the steps in the method for implementing the above-mentioned embodiment may be completed by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, which, when executed, includes one or a combination of the steps of the method embodiment.

[0150] In addition, each functional unit in each embodiment of the present application may be integrated into a processing module, or each unit may exist physically separately, or two or more units may be integrated into one module. The above-mentioned integrated module may be implemented in the form of hardware or in the form of a software functional module. If the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it may also be stored in a computer-readable storage medium.

[0151] The storage medium mentioned above may be a read-only memory, a magnetic disk or an optical disk, etc. Although the embodiments of the present application have been shown and described above, it can be understood that the above embodiments are exemplary and cannot be understood as limiting the present application. A person of ordinary skill in the art may change, modify, replace and modify the above embodiments within the scope of the present application.

Claims

1. A visual inertial positioning method using a graph attention network to match points and lines, characterized in that: include: During the operation of the visual inertial positioning system, the monocular camera in the system is used to collect the image at the current moment; Acquire point features and line features of the image at the current moment; Based on the point features and line features of the image at the current moment, a matching relationship between the point features and line features of the image at the previous moment is obtained by using a preset point-line matching network including a graph attention mechanism; When the total number of acquired images is greater than the preset number of sliding window frames, based on the line features of the image at the current moment, detecting the classification results of the main structural direction and the line features of the image at the current moment; Based on the matching relationship between the point features and line features of the current moment image and the previous moment image, the classification result of the structural main direction and line features of the current moment image, and the measurement result of the inertial measurement unit in the visual inertial positioning system, the position and posture of the monocular camera are optimized to obtain the current positioning result of the visual inertial positioning system.

2. The method according to claim 1, characterized in that The step of obtaining the point features and line features of the image at the current moment includes: The current moment image I t Input the pre-trained convolutional neural network SuperPoint to obtain the dense descriptor graph corresponding to the image, the location of the point feature, the descriptor of the point feature, and the saliency of the point feature; Use LSD line segment detection algorithm to process image I t , get the position of the line feature corresponding to the image; Image I t The corresponding dense descriptor graph and the position of the line feature are input into the preset NeXtVLAD feature aggregation network to obtain the descriptor of the line feature corresponding to the image.

3. The method according to claim 2, characterized in that The point-line matching network includes a sequentially connected position embedding subnetwork, a graph attention subnetwork, and a bidirectional Softmax matching operation module; The position embedding subnetwork includes two parallel multi-layer perceptron networks; for the current moment image and the previous moment image, the first multi-layer perceptron network takes the position of the point feature of each frame image, the descriptor of the point feature and the saliency of the point feature as input, and outputs the position embedding vector of the point feature; the second multi-layer perceptron network takes the length of the line feature of each frame image, the direction of the line feature and the midpoint position of the line feature as input, and outputs the position embedding vector of the line feature; The graph attention sub-network takes the descriptors of the point features and the position embedding vectors of the point features, the descriptors of the line features and the position embedding vectors of the line features of the two frames of images as input, and outputs the descriptors of the point features and the descriptors of the line features after being updated in the two frames of images; The bidirectional Softmax matching operation module takes the descriptors of the point features and the line features of the two frames of images output by the graph attention subnetwork as input, and outputs the matching relationship between the point features and the line features of the two frames of images.

4. The method according to claim 3, characterized in that The graph attention subnetwork includes nine layers of attention layers connected in sequence, and each layer of attention layer contains six attention units, namely: same-frame point-point attention unit, same-frame line-line attention unit, same-frame point-line attention unit, same-frame line-point attention unit, different-frame point-point attention unit, and different-frame line-line attention unit; wherein, in each layer of attention layer, the same-frame point-point attention unit takes the position embedding vector of the point features of the two frames of images and the descriptor of the point features as input, outputs the updated point feature descriptor and sends it to the same-frame point-line attention unit; the same-frame line-line attention unit takes the position embedding vector of the line features of the two frames of images and the descriptor of the line features as input, outputs the updated line feature descriptor and sends it to the same-frame point-line attention unit; the same-frame point-line attention unit takes the position embedding vector of the point features and line features of the two frames of images and the received point features and line features as input. The descriptor of the feature is taken as input, and the updated descriptor of the point feature and the unupdated descriptor of the line feature are output and sent to the same-frame line-point attention unit; the same-frame line-point attention unit takes the position embedding vectors of the point features and line features of the two frames of images and the received descriptors of the point features and line features as input, outputs the unupdated descriptor of the point features and sends them to the different-frame point-point attention unit, and at the same time outputs the updated descriptor of the line features and sends them to the different-frame line-line attention unit; the different-frame point-point attention unit takes the position embedding vectors of the point features of the two frames of images and the received descriptors of the point features as input, and outputs the descriptor of the point features that is finally updated after passing through this layer of attention layer; the different-frame line-line attention unit takes the position embedding vectors of the line features of the two frames of images and the received descriptors of the line features as input, and outputs the descriptor of the line features that is finally updated after passing through this layer of attention layer.

5. The method according to claim 4, characterized in that The attention unit uses the attention operation, which is expressed as follows: Among them, i and j are the feature subscripts, which are taken from two subscript sets A and B respectively, and Γ contains all feature subscript combinations constructed by these two subscript sets; f i is the descriptor of the i-th feature, and MLP is a i and attention vector a(i,Γ) as input multi-layer perceptron network, Softmax j is the softmax operation for index j, ← is the update operation; q i , k j 、v j They are the query vector corresponding to the i-th feature, the key vector and the value vector corresponding to the j-th feature; the feature descriptor, position embedding vector, query vector, key vector and value vector have the same number of channels, which is D.

6. The method according to claim 4, characterized in that Before acquiring the point features and line features of the image at the current moment, the method further includes: Jointly training the NeXtVLAD feature aggregation network and the point-line matching network; The joint training of the NeXtVLAD feature aggregation network and the point-line matching network includes: Firstly, a picture dataset is used to construct virtual picture pairs by adding homography transformation, and the matching relationship of point and line features in the picture pairs is restored from the homography transformation. The matching relationship of the point and line features is used as the matching truth value to supervise the joint training of the network. Secondly, after the training converges, the network weights completed in the previous stage of training are used as the initial weights. A dataset with real camera pose, camera intrinsic parameters, pictures and dense depth is used. Picture pairs with larger cosight areas in the dataset are selected. The correct matching relationship of point and line features is obtained according to the camera pose, camera intrinsic parameters and dense depth provided by the dataset. The correct matching relationship of point and line features is used as the matching truth value for the second stage of network joint training. After the second stage of training converges, the obtained network weights are used as the final network weights of the NeXtVLAD feature aggregation network and the point and line matching network.

7. The method according to claim 4, characterized in that The method further comprises: If the total number of images currently collected is less than the preset number of sliding window frames, when the next image collection time arrives, continue to collect new current moment images and obtain the matching relationship between the point features and line features of the new current moment image and the previous moment image; If the total number of images currently collected is equal to the preset number of sliding window frames, the initial value of the camera pose at the real scale is obtained, including: First, the SfM algorithm is used to obtain the scale-free camera pose corresponding to N frames of images. Secondly, the inertial measurement unit (IMU) measurement results are used to align with the camera pose obtained by the SfM algorithm to solve the scale factor, gravity direction, gyroscope bias and camera velocity of N frames. Finally, the scale factor and the scale-free camera pose are used to obtain the camera pose corresponding to N frames of images at the real scale. N is the preset number of sliding window frames.

8. The method according to claim 7, characterized in that The classification result of detecting the main structural direction and line features of the image at the current moment includes: For image I t All line features of the image are detected according to the direction of gravity, and the structural line features that meet the Atlanta world hypothesis are detected; among them, based on the normal vector of the line feature, the line features whose normal vector is perpendicular to the gravity direction are detected and used as the structural lines distributed along the vertical direction; the remaining line features are clustered into horizontal line clusters to obtain image I t All horizontal structure main directions; finally, image I t All line features are divided into structural lines distributed along the vertical direction or the horizontal direction and non-structural lines distributed along other directions.

9. The method according to claim 8, characterized in that The optimizing the posture of the monocular camera includes: 1) Estimate the camera pose of the most recent N frames through sliding window optimization; Among them, the camera pose and speed of the latest N frames, IMU bias, the spatial position of the point landmarks and line landmarks observed in the latest N frames, and the main direction of the structure are used as optimization variables to construct an optimization problem; the optimization problem takes the cost function L as the optimization target, and uses the Dogleg algorithm to minimize the cost function L to obtain the camera pose estimation results of the latest N frames; the cost function L is the sum of the reprojection error of the point feature, the reprojection error of the line feature, the IMU pre-integration error, the structural constraint error and the main direction observation error of the structure; the reprojection error of the point feature is the distance between the projection position of the point landmark on the image and the point feature position actually observed on the frame image, the reprojection error of the line feature is the distance between the projection position of the line landmark on the image and the line feature endpoint actually observed on the frame image, the structural constraint error is the cosine value of the angle between the main direction of the structure and the gravity direction, and the main direction observation error of the structure is the cosine value of the angle between the main direction of the structure and the normal vector of the line feature; 2) Detect whether there is a loop in the image at the current moment; Among them, for each frame image I t , using the DBoW algorithm with point feature descriptors as input to detect whether there is a loop: If there is a loop, the loop frame image I is returned old , then go to step 3); if there is no loop, the camera pose estimation result of step 1) is used as the current positioning result; 3) Through loop correction, the camera pose corresponding to the image trajectory is optimized; Wherein, based on the result of step 7-2), I old As I t-1 , calculate image I old and image I t Then, the PnP algorithm with RANSAC is used to determine the matching relationship between the point features and line features of image I old Relative Image I t The pose value is then used as the initial pose value of the loop frame, and the pose of the loop frame is inserted into the optimization variable of the optimization problem in step 1) as an additional optimization variable. old and image I t The reprojection error of the point features and line features constructed by the matching relationship of the point features and line features is added to the cost function of the optimization problem constructed in step 1), and the updated optimization problem is solved to obtain image I old Relative Image I t The camera pose estimation result is obtained by adding the estimation result as a constraint item to the pose graph, and the pose graph optimization algorithm is used to optimize the pose graph from image I old To Image I t The camera pose estimation on the entire trajectory is corrected to obtain the current positioning result.

10. A visual inertial positioning device using a graph attention network to match points and lines, characterized in that: include: An image acquisition module is used to acquire the current moment image using the monocular camera in the visual inertial positioning system during its operation; A feature acquisition module, used to acquire point features and line features of the image at the current moment; A feature matching module, configured to obtain a matching relationship between the point features and line features of the image at the current moment and the image at the previous moment by using a preset point-line matching network including a graph attention mechanism based on the point features and line features of the image at the current moment; A structure detection module, used for detecting the main structural direction and classification result of the line features of the image at the current moment based on the line features of the image at the current moment when the total number of acquired images is greater than the preset number of sliding window frames; A positioning module is used to optimize the position and posture of the monocular camera based on the matching relationship between the point features and line features of the image at the current moment and the image at the previous moment, the classification result of the structural main direction and line features of the image at the current moment, and the measurement result of the inertial measurement unit in the visual inertial positioning system to obtain the current positioning result of the visual inertial positioning system.

Images