Underwater visual positioning method based on deep learning visual inertial odometer

Abstract

The invention discloses an underwater visual positioning method based on a deep learning visual inertial odometer. The method comprises the following steps: constructing a system comprising a deep learning visual odometer model and a nonlinear optimization rear end; the method comprises the following steps: acquiring underwater video frames through an underwater robot provided with a monocular camera, training a model, processing the underwater video frames to be positioned through the trained model, outputting successfully associated feature point information, issuing the feature point information through a communication interface, and carrying out nonlinear optimization on node data of a rear-end subscription communication interface. And the pose of the monocular camera is output after processing, and underwater visual positioning is realized. According to the method, deep learning is introduced into feature point extraction and matching, so that sufficient matching feature point position information can be obtained in an underwater weak texture and unstable illumination environment, a nonlinear optimization loss function can be converged more quickly, and more accurate camera pose information is obtained; and the better visual positioning capability of the underwater mobile robot can be realized.

Description

Technical Field

[0001] The present invention relates to an underwater visual positioning method, and relates to the fields of computer vision, underwater visual positioning technology and marine technology, and in particular to an underwater visual positioning method based on deep learning visual inertial odometer. Background Art

[0002] There are abundant energy, minerals and biological resources under the water, which have great economic and strategic value. The exploration and acquisition of underwater resources cannot be separated from advanced marine equipment. As a new type of marine equipment, underwater robots can replace operators to perform underwater tasks for a long time, over a large range and at a low cost in complex sea conditions by carrying various sensors and operating tools, thereby minimizing the risk of casualties. When underwater robots perform underwater operations, their own positioning and posture estimation are the basis, which leads to tasks such as autonomous navigation, obstacle avoidance and path tracking. Higher-level marine environment detection, seabed resource exploration, and maintenance of marine pipelines and cables all require accurate and reliable underwater positioning technology support. However, due to the lack of global positioning system support similar to GPS (Global Positioning System) underwater, related tasks are difficult to carry out. Real-time positioning technology for underwater mobile robots has become an important technical component for going deep blue.

[0003] Although traditional underwater acoustic positioning technologies such as long baseline, short baseline and ultra-short baseline can obtain the robot's position information more accurately, they also have defects such as the need to deploy base stations in advance, high cost of equipment installation and maintenance, difficulty in obtaining the underwater robot's motion posture, and lack of detection target texture information. Underwater visual positioning technology is an important supplement to underwater acoustic positioning. Unlike underwater acoustic positioning, the advantages of underwater visual positioning are that the imaging is intuitive and subtle (the camera can obtain richer information), the optical detection speed is faster (the transmission speed of light in water is on the order of 109m / s), it is suitable for small-scale precise detection, does not actively emit signals outward, has strong concealment, and has low camera cost. Visual SLAM (Simultaneous Localization and Mapping) in visual positioning technology (using visual methods for simultaneous positioning and mapping) can measure data through sensors such as cameras installed on mobile devices to achieve self-positioning in unknown environments. The research on visual odometers is the key to this visual positioning technology. Underwater visual SLAM algorithm is an important part of underwater visual positioning technology, among which the research on underwater visual odometer calculation method is also the key content of underwater visual positioning technology.

[0004] At present, the Zeno autonomous underwater vehicle AUV (Autonomous Underwater Vehicle) has been used to test the visual odometry using the scale-invariant feature transform SIFT (Scale-Invariant Feature Transform) feature extraction algorithm and the accelerated robust feature SURF (Speeded-Up Robust Features) feature extraction algorithm in the sea area. In a simple trajectory tracking experiment, the scale-invariant feature transform SIFT feature extraction method is more in line with the real trajectory. The complex underwater lighting environment and the turbidity of seawater make underwater grayscale images generally have problems of uneven exposure and low contrast, which makes the extraction of image feature points more difficult than in the atmospheric environment. Therefore, it is undoubtedly a solution to first enhance and repair the underwater image and then send it to the feature extraction algorithm. Some scholars first used the histogram equalization algorithm to enhance the underwater images of the AQYACOL underwater dataset, and then sent the enhanced underwater images to the visual odometry, and used the feature extraction ORB (Oriented FAST and Rotated BRIEF) algorithm for feature extraction, which increased the number of extracted feature points and made the visual odometry initialization and pose estimation more stable; some scholars also adopted the means of underwater image enhancement. To address the problems of low resolution and color distortion of underwater images, they proposed an image enhancement Retinex method based on the guided filter in the HSV (Hue Saturation Value) color space, and used an autonomous underwater vehicle AUV to collect laboratory pool images for verification. The enhanced images were also subjected to feature extraction using the feature extraction ORB algorithm, which significantly increased the number of feature points.

[0005] The underwater visual SLAM algorithm based on the image enhancement strategy can increase the feature points that can be extracted from low-resolution and color-distorted underwater images to a certain extent, providing a good foundation for subsequent optimization and loop detection. However, these methods are only verified in a single water environment, and the universality of different sea areas has not been verified. In addition, the image enhancement methods do not give the running time on the hardware platform, and the real-time performance cannot be guaranteed. Summary of the invention

[0006] In order to solve the problems existing in the background technology, the present invention provides an underwater visual positioning method based on deep learning visual inertial odometer. The present invention is a method that uses an underwater mobile device as a mounting platform, fuses underwater image data with inertial measurement unit IMU (Inertial Measurement Unit) data, obtains the position and attitude information of the camera mounted on the underwater mobile device, and thus realizes the positioning technology of the underwater mobile device.

[0007] The technical solution adopted by the present invention is: The underwater visual positioning method based on deep learning visual inertial odometer of the present invention comprises: S1: Build an underwater monocular visual-inertial odometry system, which includes a deep learning visual odometry model based on a descriptor loss function with a scale scaling factor and a nonlinear optimization backend.

[0008] S2: Acquire several underwater videos through an underwater robot equipped with a monocular camera and extract underwater video frames of continuous frames. Each two underwater video frames are combined into an image pair, and each group of image pairs is constructed as a training set; use the training set to train the deep learning visual odometry model, so as to obtain a trained underwater deep learning visual odometry model.

[0009] S3: The underwater video to be located is obtained through an underwater robot equipped with a monocular camera and continuous frames of underwater video frames are extracted, which are then input into the underwater deep learning visual odometer model. After processing, the feature point information of successful association is output and published through the communication interface of the robot operating system ROS (Robot Operating System).

[0010] S4: The nonlinear optimization backend subscribes to the node data of the robot operating system ROS communication interface, and outputs the position and pose of the monocular camera after processing to achieve underwater visual positioning.

[0011] In step S1, the deep learning visual odometer model includes a feature extraction layer and a feature matching layer connected in sequence.

[0012] The deep learning model of the present invention is applied to underwater environments with unstable lighting and weak texture, which can significantly increase the number of reliable matching feature points, making the position and posture estimation of the underwater camera more accurate and stable.

[0013] In step S1, the descriptor loss function of the scale scaling factor is introduced L d as follows: L d (D,D ' , S )=(∑ h,w ∑ h',w' ( l d ( d hw , d' h'w' ; s hwh'w' ))) / ( H cW c ) 2 l d ( d hw , d' h'w' ; s hwh'w' )= λ d × s hwh'w' × max (0, m p -( d hw T d' h'w' ) / ( d k ) 1 / 2 )+(1- s hwh'w' )× max (0,( d hw T d' h'w' ) / ( d k ) 1 / 2 - m n ) h , h' =1, 2, …, H c w , w' =1, 2, …, W c Among them, D and D ' They represent the original image and its homography relationship matrix input into the decoder in the feature extraction layer of the deep learning visual odometry model. S The output image at the same position after transformation, d hw and d' h'w' They represent the original input image D and the transformed output image D respectively. ' The eigenvectors in s hwh'w' Represents the homography relationship matrix S Elements in l d ( ) represents descriptor loss; T represents transpose;H c and W c Denote the input original image D and the output image D respectively ' height and width; λ d , m p and m n Denote the descriptor loss respectively. l d The preset first, second and third hyperparameters of ; d k Represents the scale factor.

[0014] In the step S2, for each image pair, two underwater video frames in the image pair are input into the feature extraction layer for processing, and the feature extraction layer extracts each feature point and its descriptor in the two underwater video frames respectively.

[0015] In the step S2, for each image pair, the descriptors of each feature point in the two underwater video frames output by the feature extraction layer are input into the feature matching layer for processing. The feature matching layer matches each descriptor one by one, thereby associating each feature point between the two underwater video frames in the image pair, and publishing the successfully associated feature point information through the robot operating system ROS communication interface.

[0016] In the step S2, the deep learning visual odometry model is trained, that is, the feature extraction layer is trained. First, the underwater video frame is subjected to random affine transformation and then input into the feature extraction layer, so as to perform transfer learning on the feature extraction layer to generate pseudo-true value labels, obtain a trained underwater feature extraction layer, and finally obtain a trained underwater deep learning visual odometry model.

[0017] In the step S4, an inertial measurement unit IMU is also installed on the underwater robot. After the nonlinear optimization backend subscribes to the node data of the robot operating system ROS communication interface, it is processed through a sliding window optimization algorithm to minimize the weighted sum of the feature point reprojection error and the inertial measurement unit IMU pre-integration error, and finally outputs the monocular camera pose of the camera frequency and the inertial measurement unit IMU frequency.

[0018] In the step S4, the deep learning visual odometry model of the underwater monocular visual inertial odometry system runs on a graphics processing unit (GPU) device, and the nonlinear optimization backend runs on a central processing unit (CPU) device.

[0019] The method of the present invention introduces a deep learning method into the image feature extraction and matching part of the underwater visual odometer, decouples this part from the nonlinear optimization backend, and runs them on a graphics processing unit (GPU) device and a central processing unit (CPU) device respectively, making full use of hardware computing resources.

[0020] The electronic device of the present invention comprises: a memory and a processor coupled to each other, wherein the memory stores program data, and the processor calls the program data to execute the method as described above.

[0021] The computer-readable storage medium of the present invention stores program data thereon, and when the program data is executed by a processor, the method described above is implemented.

[0022] The method of the present invention is aimed at the task requirement of an underwater robot to achieve autonomous underwater visual positioning in a real operating environment. Even in harsh underwater visual environments such as dynamic lighting and weak texture scenes, stable and reliable visual positioning can still be achieved.

[0023] The beneficial effects of the present invention are: Compared with other visual inertial odometers, the present invention introduces a deep learning method in feature point extraction and matching, so that the visual inertial odometer can obtain sufficient matching feature point position information in underwater weak texture and unstable lighting environments, provide a good initial value for the nonlinear optimization backend, and enable the nonlinear optimization loss function to converge more quickly, thereby obtaining more accurate camera pose information and achieving better visual positioning capability of underwater mobile robots. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 It is a working mode diagram of the method of the present invention; Figure 2 This is a comparison chart of the image matching experiment in an underwater unstable lighting environment, where: Figure 2 (a) is the result of feature point extraction and matching using the sparse optical flow method KLT (Kanade-Lucas-Tomasi) in an underwater unstable lighting environment image. Figure 2 (b) is the feature point extraction and matching result of the underwater unstable lighting environment image using the feature extraction ORB method. Figure 2 (c) is a feature point extraction and matching result diagram of an underwater unstable lighting environment image using the method of the present invention; Figure 3 This is a comparison chart of the underwater weak texture environment image matching experiment, where: Figure 3 (a) is the result of feature point extraction and matching using the sparse optical flow method KLT for underwater weak texture environment images. Figure 3(b) is the feature point extraction and matching result of underwater weak texture environment image using feature extraction ORB method. Figure 3 (c) is a feature point extraction and matching result diagram of an underwater weak texture environment image using the method of the present invention; Figure 4 A flow chart of model training for transfer learning of the present invention; Figure 5 Schematic diagram of image expansion for homographic adaptation; Figure 6 The precision and recall curves recorded for model training are as follows: Figure 6 (a) is the accuracy curve of the model training record. Figure 6 (b) is the recall rate curve of the model training record. DETAILED DESCRIPTION

[0025] The present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0026] The underwater visual positioning method based on deep learning visual inertial odometer of the present invention is specifically as follows: like Figure 1 As shown in the figure, an underwater monocular visual inertial odometry system is first constructed. The underwater monocular visual inertial odometry system includes a deep learning visual odometry model based on a descriptor loss function with a scale scaling factor and a nonlinear optimization backend; the deep learning visual odometry model includes a feature extraction layer and a feature matching layer connected in sequence. The feature extraction layer uses a pre-trained feature point extraction MagicPoint model, and the visual geometry VGG structure in the feature point extraction MagicPoint model is replaced with the structure of the first three layers of the residual network ResNet50 model; the feature matching layer uses the LightGlue (Sparse Lightweight Graph Matching with Uncertainty Estimation) model. The nonlinear optimization backend uses a nonlinear optimization backend based on simultaneous positioning and mapping SLAM.

[0027] Then, an underwater robot equipped with a monocular camera is used to obtain several underwater videos and extract underwater video frames of continuous frames. Each two underwater video frames form an image pair, and each group of image pairs is constructed as a training set; the training set is used to train the deep learning visual odometry model, that is, the feature extraction layer is trained. First, the underwater video frame is randomly affine transformed and then input into the feature extraction layer, so as to perform transfer learning on the feature extraction layer to generate pseudo-true value labels, obtain the trained underwater feature extraction layer, and finally obtain the trained underwater deep learning visual odometry model; in specific implementation, the parameters of the first three layers of the residual network ResNet50 model in the SuperPoint model remain unchanged, and the parameters of other structures are optimized with training. During the training process, the feature extraction workflow in the deep learning visual odometry model is as follows: First, the width is W Gao Wei H The first grayscale image is input into the original SuperPoint model, which first undergoes an encoder module consisting of several convolutional layers, pooling layers, and activation layers, and then reduces the image width and height to 1 / 8 of the original. That is, the width of the first grayscale image becomes W c = W / 8, high becomes H c = H / 8, and at the same time increase the number of channels in the output feature layer to obtain high-dimensional feature information of the image, and use the last feature layer as the input of the subsequent feature point position decoder and feature point descriptor decoder. However, the encoder of the original model uses the visual geometry VGG (Visual Geometry Group) network structure. In the process of increasing the number of network layers, low-dimensional features will gradually be lost and the network weights will be difficult to update. Therefore, the visual geometry VGG structure of the original model is replaced with the structure of the first three layers of the residual network ResNet50 model, which not only retains the original 1 / 8 ratio of the output feature layer of the backbone network to the length and width of the input image, but also introduces residual connections, which greatly alleviates the defects of the original model encoder. The feature point position decoder is used to generate the probability of whether each pixel position on the image is a feature point. The input feature layer size of this module is W c × H c ×65, where X is obtained by adjusting the width, height and channel dimension of the feature layer after the output feature layer of the decoder passes through the convolution layer, pooling layer and activation function. It is the input feature layer of the feature point position decoder and is output after the softmax and pixel shuffle operations. H × W×1 feature layer, where each feature vector corresponds to each pixel on the image one by one, indicating the probability that the pixel is a feature point. The pixel positions that meet the threshold requirements are screened and finally output m ×2 size position matrix p A The feature point descriptor decoder is used to generate a descriptor with a vector length of 256 for each pixel on the image. The input feature layer D size of this module is W c × H c ×256, and then use bilinear interpolation to enlarge the width and height of the feature layer to the same as the original image, and then perform L2 normalization on each channel to ensure that the modulus length of each descriptor is 1. After the operation, the output H × W ×256 size feature layer, where each feature vector corresponds to each pixel on the image one by one, representing the descriptor of the pixel point, and the descriptor at the same position as the feature point position encoder is screened and finally output m The first descriptor matrix of size ×256 d A The second image is also processed through the above operation to obtain the second position matrix p B and the second descriptor matrix d B In addition, the training and verification dataset used by the original model is the COCO (Common Objects in Context) dataset. The images in this dataset are taken in land scenes. Due to the absorption and scattering of light by water, underwater optical images are prone to low contrast, fogging and color distortion, which greatly limits the success rate of feature point extraction of the original SuperPoint model and further affects the subsequent feature point matching effect. In order to solve the above problems, the present invention randomly extracts some images from four public underwater image datasets and uses Figure 4The two-stage training strategy shown in the figure fine-tunes the model parameters. First, a MagicPoint feature point extraction model pre-trained on the contextual common object COCO dataset is used, and it is re-trained using underwater images processed with image transformation to generate pseudo ground truth point labels for transfer learning of the SuperPoint model. These labels not only contain the original image information, but more importantly, provide reliable feature point location information. Then, the loss function is minimized based on the valid feature point labels to obtain the underwater MagicPoint model trained with underwater images. In the training process of the underwater MagicPoint model, in order to improve the diversity and representativeness of the training samples, the homography adaptation technology is used, which applies a series of random affine transformations to the input image, such as Figure 5 As shown in the figure, if the position error of the feature points detected in the original image and the feature points detected in the transformed image is within the threshold after the same transformation, they are regarded as valid feature points. This data enhancement strategy significantly expands the number of training samples while improving the adaptability of the model to different perspectives. The model is trained on a training set of 3,685 images and a validation set of 932 images constructed from four public underwater datasets, and uses the Adaptive Moment Estimation Adam optimizer. The initial learning rate is set to 0.0001, and the training is performed for 600,000 iterations, with validation performed on the validation dataset every 5,000 iterations.

[0028] In order to make the training results of the model converge more quickly, the present invention does not use the loss function of the descriptor in the SuperPoint method, but introduces a scale scaling factor d k (the number of channels of the input feature layer D), descriptor loss function L d The details are as follows: L d (D,D ' , S )=(∑ h,w ∑ h',w' ( l d ( d hw , d' h'w' ; s hwh'w' ))) / ( H c Wc ) 2 h , h' =1, 2, …, H c w , w' =1, 2, …, W c Among them, D and D ' Respectively represent the original image of the feature point descriptor decoder in the feature extraction layer of the input model and its homography relationship matrix S The output image at the same position after transformation, d hw and d' h'w' Denote the input original image D and the output image D respectively ' The eigenvectors in s hwh'w' Represents the homography relationship matrix S If the coordinates of the feature points extracted from the input original image D are consistent with the output image D after homography adaptation transformation, ' If the Euclidean distance of the corresponding feature point coordinates after the inverse transformation of the extracted feature points is less than 8, then s hwh'w' =1, otherwise s hwh'w' =0; l d ( ) represents descriptor loss; T represents transpose; H c and W c Denote the input original image D and the output image D respectively ' height and width.

[0029] The present invention uses the scale factor d k Introducing descriptor loss l d The details are as follows: l d ( d hw , d' h'w' ; s hwh'w' )= λ d × s hwh'w' × max (0, mp -( d hw T d' h'w' ) / ( d k ) 1 / 2 )+(1- s hwh'w' )× max (0,( d hw T d' h'w' ) / ( d k ) 1 / 2 - m n ) in, λ d , m p and m n Denote the descriptor loss respectively. l d The preset first, second, and third hyperparameters of .

[0030] Introducing a scaling factor d k Can prevent d hw T and d' h'w' The inner product of the two eigenvectors is too large or too small. When it is too large, the loss function value curve will oscillate during training. When it is too small, the model parameters cannot be effectively updated. Both of these situations will cause the model parameters to fail to converge quickly and even lead to training failure.

[0031] The precision and recall curves of the model training records are as follows: Figure 6 (a) and Figure 6 As shown in (b), the precision and recall of the model show an upward trend during the model training process, which indicates that the extraction results of feature points by the training network are satisfactory.

[0032] In order to further demonstrate the superiority of the optimized underwater feature extraction layer in feature point extraction in underwater extreme lighting environments, the present invention selected extreme scene images in two sequences in two public data sets for feature point extraction experiments in specific implementation. The extreme scene images are underwater weak texture scene images and underwater dynamic lighting scene images, respectively. The original SuperPoint model and the optimized model perform feature point extraction experiments on these images, and record the total number of feature points extracted in each series and the number of images in which the optimized model extracts more feature points than the original SuperPoint on a single image divided by the total number of images. The percentage is recorded as BR (Better Rate), as shown in Table 1. The results show that the model after transfer learning fine-tuning is better than the original model in terms of both the total number of feature points in each sequence and the performance improvement rate BR value.

[0033] Table 1 Feature extraction performance under different underwater conditions

[0034] Among them, AQ represents the AQUALOC dataset, FLS represents the FLSea_VI dataset, WT represents low texture, DL represents dynamic lighting, and FC represents the number of feature points.

[0035] Then the feature matching workflow is as follows: The first position matrix output by the feature extractor p A , the first descriptor matrix d A , the second position matrix p B and the second descriptor matrix d B As the input of the attention layer for feature matching, the first position matrix p A First perform relative position rotation position encoding and then compare it with the first descriptor matrix d A Together as the input of the self-attention block to calculate the self-attention score a ij , as follows: a ij = q i T R ( p j - p i ) k j in, q i and k j They are the first descriptor matrices d A The key vector and query vector obtained by decomposition after different linear transformations, R () is the relative position rotation position encoding between feature points in d-dimensional space, R ()∈ R d×d , p i , p j ∈ p A , in order to obtain the influence weight coefficient of each feature point in the first image on each other; then the output of the self-attention module is combined with the first descriptor matrix d A Together they serve as the input of the multilayer perceptron (MLP) to adjust the weight coefficients of the influence of each feature point on each other. The multilayer perceptron consists of a hidden layer, a layer normalization layer, an activation function GELU layer, and a convolution layer. The second position matrix p B and the second descriptor matrix d B The above operation is repeated in the same way; after the above operation, the output of the first multi-layer perceptron is used as the input of the cross-attention block to calculate the cross-attention score. b ij , as follows: b ij = K i AT K j B in, K i A It comes from the feature points in the first image i The key vector of K j B It comes from the feature points in the second image j The key vector is used to obtain the influence weight coefficients between different image feature points; the output of the cross attention module and the output of the first multi-layer perceptron are used as the input of the second multi-layer perceptron to adjust the descriptors of each image feature point. Assume that the output of the second multi-layer perceptron is xi , calculate the confidence of the adjusted descriptor c i , as follows: c i = Sigmoid ( x i )∈[0,1] Sigmoid ( x )=1 / (1+e -x ) When the confidence meets the threshold condition, the next attention layer inference is exited and the similarity matrix module is entered to perform similarity calculation. If the condition is not met, the pruning strategy is executed to remove the feature point information with low confidence to reduce the calculation amount of the next attention layer. The input of the similarity matrix module is the descriptor of the first image adjusted by the influence weight coefficient. x 1 A , x 2 A , …, x i A , …, x m A And the descriptor of the second image adjusted by the influence weight coefficient x 1 B , x 2 B , …, x j B , …, x n B , each descriptor x i A ( i ∈[1, m ]) are all related to the descriptor x j B ( j ∈[1, n ]) Calculate the similarity score, form a similarity matrix, and select the top similarity score with the highest similarity score greater than the threshold. S And find the feature points of the two images they correspond to, and then you can get the successfully matched feature points.

[0036] Table 2 Feature extraction and matching performance under different underwater conditions

[0037] Among them, the total number is the number of matching feature points after initial matching; the correct number is the number of internal points after RANSAC filtering; the accuracy is the percentage of correct matches (correct number / total number × 100%).

[0038] As shown in Table 2, the quantitative indicators of four feature point extraction and matching methods in two images with similar time in the presence of a common view area are shown. The four methods are the combination of the GFTT (Good Features To Track) feature point extraction method represented by the monocular visual-inertial system VINS-Mono (Visual-Inertial System Monocular) and the visual-inertial multi-sensor fusion system VINS-Fusion (Visual-Inertial System Fusion) and the sparse optical flow method KLT matching method, the ORB feature point extraction method represented by the ORB-SLAM series and the Fast Library for Approximate Nearest Neighbor Fast Search Library FLANN (Fast Library for Approximate Nearest Neighbor Fast Search Library FLANN) and the combination of the GFTT (Good Features To Track) feature point extraction method represented by the ORB-SLAM series and the sparse optical flow method KLT matching method, ...GFTT (Good Features To Track) feature point matching method, and the GFTT (Good Features To Track) feature point matching method, and the GFTT (Good Features To Track) feature point matching method, and the GFTT (Good Features To Track) feature point matching method, and the GFTT (Good Features To Track) feature point matching method, and the GFTT (Good Features To Track) feature point matching method, and the GFTT (Good Features To Track) feature point matching method, and the GFTT (Good Features To Track) feature point matching method, and the GFTT (Good Features To Track) feature point matching Neighbors) feature point matching method, and a combination of the SuperPoint feature point extraction method and the LightGlue matching method. The feature point extraction and matching method combination used in the present invention, the evaluation indicators are respectively the total number of matching feature points, the total number of correctly matched feature points obtained after matching feature points screening using the random sampling consistency method, and the matching accuracy obtained by dividing the total number of correctly matched feature points by the total number of matching feature points. The present invention is tested on two public underwater data sets under representative natural environment conditions. The deep-sea underwater data set is the AQUALOC data set. The first sequence of the underwater archaeological remains data set in the data set is selected. The feature of this sequence is that there are many weak-texture scenes, but the illumination is provided by the underwater submersible and is therefore relatively stable. The shallow-sea underwater data set is the FLSea_VI data set. The tiny_canyon sequence in the data set is selected. The feature of this sequence is that the texture is rich but there is an underwater optical environment with drastic changes in light intensity. It can be seen from the table that in the deep-sea environment with stable illumination and weak texture represented by the AQUALOC data set, such as Figure 3The feature extraction and matching results shown in the black and white image in the figure show that the visual front end composed of the high-quality trackable feature point GFTT feature point extraction method and the sparse optical flow method KLT feature point matching method has advantages in the deep-sea underwater optical environment with stable lighting and weak texture, but in the shallow sea underwater optical environment with drastic changes in light intensity, a large number of mismatched feature points and a low matching accuracy rate appear; the ORB feature point extraction method and the approximate nearest neighbor fast search library FLANN feature point matching method perform poorly whether in the deep-sea underwater optical environment with stable lighting but weak texture or in the shallow sea underwater optical environment with rich texture but drastic changes in light intensity, proving that the visual odometer under this combination is difficult to detect. To adapt to the underwater optical environment; the SuperPoint feature point extraction method and the LightGlue matching method also performed poorly in the above two environments, proving that simply splicing deep learning models and using the pre-trained weights of the models cannot show excellent performance in the specific underwater environment; the method in the present invention not only uses deep learning models for feature point extraction and matching, but also deeply optimizes the model for the underwater environment. The matching accuracy of the two extreme visual scenes in the table shows the strong robustness and accuracy of the front end of the underwater visual inertial odometer proposed in the present invention in the underwater optical environment, proving that it exhibits excellent performance in both weak texture and dynamic lighting environments.

[0039] After the training is completed, the underwater deep learning visual odometry model is obtained. Applying the underwater deep learning visual odometry model to the underwater lighting unstable and weak texture environment can significantly increase the number of reliable matching feature points, making the position and attitude estimation of the underwater camera more accurate and stable.

[0040] Then, an underwater video to be located is obtained by an underwater robot equipped with a monocular camera, and underwater video frames of continuous frames are extracted and input into the underwater deep learning visual odometer model. For each image pair, the two underwater video frames in the image pair are input into the feature extraction layer for processing, and the feature extraction layer extracts each feature point and its descriptor in the two underwater video frames respectively. The descriptors of each feature point in the two underwater video frames output by the feature extraction layer are input into the feature matching layer for processing, and the feature matching layer matches each descriptor one by one, so as to associate each feature point between the two underwater video frames in the image pair, and publish the information of the successfully associated feature points through the robot operating system ROS communication interface; then the nonlinear optimization backend subscribes to the node data of the robot operating system ROS communication interface, and the underwater robot is also equipped with an inertial measurement unit IMU. After the nonlinear optimization backend subscribes to the node data of the robot operating system ROS communication interface, it is processed through the sliding window optimization algorithm to minimize the weighted sum of the feature point reprojection error and the inertial measurement unit IMU pre-integration error, and finally outputs the monocular camera pose of the camera frequency and the inertial measurement unit IMU frequency. The workflow of the nonlinear optimization backend is as follows: The data of the inertial measurement unit (IMU) consists of three-axis angular velocity and three-axis acceleration. The data of the inertial measurement unit (IMU) is pre-integrated and initialized with the matching feature points on the image. The initialization only needs to be performed once, that is, the matching feature points of the two images are triangulated to obtain the depth information of each feature point from the two-dimensional information on the image, and the depth information is calibrated by integrating the inertial measurement unit (IMU) data. At the same time, the rotation matrix and translation vector of the camera of the two images can be obtained from the matching feature point position information through the epipolar geometry constraint algorithm, and the deviation in the angular velocity measured by the inertial measurement unit (IMU) is calibrated through the rotation matrix, and the initial world coordinate system is determined in the process. After initialization, the pre-integrated information of the inertial measurement unit (IMU) and the matched feature point position information are maintained through the sliding window algorithm. The camera posture and the position of the three-dimensional points in the space corresponding to the feature points are simultaneously optimized through bundle adjustment. The posture forward propagation based on the inertial measurement unit (IMU) is performed, and finally the camera posture estimation of two frequencies is output, namely the camera posture estimation with the same frequency as the inertial measurement unit (IMU) and the camera posture estimation with the same frequency as the camera frame rate.

[0041] like Figure 2 (a) Figure 2 (b) and Figure 2As shown in (c), in underwater scenes with unstable lighting, the feature extraction and matching method of underwater visual inertial odometer based on deep learning feature extraction and matching method can accurately match more feature points than the traditional sparse optical flow method KLT and ORB feature point extraction method; Figure 3 (a) Figure 3 (b) and Figure 3 As shown in (c), in an underwater scene with stable lighting but weak texture, the feature extraction and matching method of the underwater visual inertial odometer based on the deep learning feature extraction and matching method can also obtain sufficient matching feature point position information.

[0042] The present invention makes a quantitative comparison with the visual inertial odometer in visual inertial SLAM such as the monocular visual-inertial system VINS-Mono and the visual-inertial multi-sensor fusion system VINS-Fusion in sequences 3, 4, 5, 6, 8, and 9 of the underwater dataset AQUALOC. The root mean square error of the absolute trajectory error between the three-dimensional trajectory of camera motion generated by the visual inertial odometer and the accurate three-dimensional trajectory of camera motion officially provided by the dataset is used as a comparison indicator, as shown in Table 3. It can be seen that the method of the present invention performs better, and the smaller the number in the table, the better.

[0043] Table 3 Quantitative comparison of visual inertial odometry

[0044] Among them, F means system crash.

[0045] As shown in Table 3, the present invention showed consistent and reliable performance in all test sequences and was the only method that successfully maintained stable operation throughout the evaluation process. The system achieved very good results in Sequences 3, 6, and 8, with positioning accuracies of 0.5m, 0.36m, and 0.60m, respectively. Most notably, in challenging scenes such as Sequence 6, the monocular visual-inertial system VINS-Mono and the visual-inertial multi-sensor fusion system VINS-Fusion, as well as the unoptimized original SuperPoint and LightGlue combination of lightweight sparse mapping visual inertial odometry LSM-VIO (Lightweight Sparse-Mapping Visual Interial Odometry) were unable to maintain operation (indicated as system crash F), while the underwater monocular visual-inertial odometry system LSM-VIO of the present invention continued to operate robustly while providing an excellent accuracy of 0.36m. More importantly, the underwater monocular visual inertial odometry system LSM-VIO achieved the best overall performance, with an average positioning error of 0.63m compared to the monocular visual-inertial system VINS-Mono (1.28m) and the visual-inertial multi-sensor fusion system VINSFusion (1.53m), which is a significant improvement of 50.8% and 58.8% respectively. This comprehensive performance advantage, coupled with its consistent operational stability under various underwater conditions, further demonstrates the excellent robustness and practical applicability of the underwater monocular visual-inertial odometry system LSM-VIO in actual underwater applications.

[0046] In order to ensure that the underwater monocular visual inertial odometer system LSM-VIO can run stably on a resource-constrained underwater vehicle platform and achieve a posture output speed of more than 10 frames per second, key optimization strategies at the system implementation level are carried out. The optimization method solves the two problems of computing resource allocation and model acceleration to jointly ensure real-time performance. In terms of computing resource allocation, the present invention adopts a decoupling strategy between the deep learning front end and the optimization back end. In order to improve the operating efficiency of the system and better utilize hardware resources, the system is split into a visual front end and a nonlinear optimization back end and runs on two threads respectively. Considering that deep learning reasoning is suitable for parallel processing, while optimization calculations require sequential operations, and visual odometry involves image feature extraction and matching, which involves a large number of repeated operations, the system mainly performs visual front-end calculations on the GPU device while running the back-end optimization module, and publishes the matched feature point information through the robot operating system ROS topic communication. The nonlinear optimization back-end involves a large number of complex operations and data access operations, so it runs on the CPU device to perform the pose optimization process of the monocular camera and IMU, and also subscribes to feature point information through the robot operating system ROS topic communication. After decoupling, this method can maximize the use of computing resources and speed up operation efficiency. The two modules communicate through the robot operating system ROS topic mechanism: the front end continuously publishes feature point matching information, while the back end subscribes in real time and performs state estimation. This heterogeneous computing design avoids resource contention between modules while ensuring processing performance. In addition, thanks to the modular design, each functional module maintains an independent parameter adjustment interface, which is convenient for targeted optimization for different application scenarios. At the level of deep learning model acceleration, we have implemented the following multi-layer optimization strategy. First, a basic acceleration environment was established by converting the PyTorch model into the engine format and using TensorRT as the inference framework. Secondly, according to the characteristics of the underwater visual positioning task, the operator fusion optimization target attention mechanism was implemented. Specifically, in the LightGlue model, the attention mechanism module contains multiple computationally intensive operations such as matrix multiplication and soft maximum calculation, and there are a large number of intermediate results transferred between these operations. Traditional methods require repeated reading and writing of these intermediate data in the GPU device memory, which results in substantial memory access overhead. By analyzing the data flow dependencies calculated in the attention module, the key and value matrices are cached to accelerate model memory access. In addition, the grouped query attention GQA (Grouped-Query Attention) method is used to reason about the attention mechanism module, achieving a balance between the speed of multi-query attention MQA (Multi-Query Attention) and the accuracy of multi-head attention MHA (Multi-Head Attention).These optimization strategies reduce the storage and access frequency of intermediate results, which not only reduces memory bandwidth pressure but also increases computing density, thereby significantly improving model reasoning efficiency.

[0047] The method of the present invention introduces a deep learning method into the image feature extraction and matching part of the underwater visual odometer, decouples this part from the nonlinear optimization backend, and runs them on a graphics processing unit (GPU) device and a central processing unit (CPU) device respectively, making full use of hardware computing resources.

[0048] The real-time performance of the proposed underwater visual inertial odometry has been evaluated on the NVIDIA Jetson Orin NX embedded development platform, a low-power embedded platform equipped with a 6-core NVIDIA ARM Cortex A78AE v8.2 64-bit CPU and a 1024-core NVIDIA Ampere GPU with 32 Tensor cores. The input consists of color images with a resolution of 968×608 pixels. For a fair comparison, 256 feature points were extracted from each image before and after model optimization. The evaluation used 1012 consecutive images captured at intervals of 0.05 seconds to ensure that there are enough common view areas in the images for feature point extraction and matching.

[0049] Table 4 Comparison of feature extraction and matching time optimization

[0050] As shown in Table 4, the optimization significantly improves the computational efficiency of the proposed method, making it very suitable for embedded platforms. The original implementation takes 0.175 seconds for each feature processing cycle. After applying the optimization method proposed in the present invention, the processing time is greatly reduced to 0.048s, which is 3.6 times faster. Detailed analysis shows that all processing stages have been substantially improved, with feature extraction and matching time reduced by 74.6% and 69.4% respectively, resulting in an overall efficiency improvement of 72.6%. This optimization enables the underwater monocular visual inertial odometry system LSM-VIO to run at a speed of approximately 15 Hz, meeting the real-time requirements of underwater visual positioning tasks.

[0051] It should be understood by those skilled in the art that the embodiments of the present application may be provided as methods, systems or computer program products. Therefore, the present application may take the form of a complete hardware embodiment, a complete software embodiment or an embodiment combining software and hardware. Moreover, the present application may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, optical storage, etc.) containing computer-usable program codes. The schemes in the embodiments of the present application may be implemented in various computer languages. The present application is described according to the flowcharts of the methods, systems and computer program products of the embodiments of the present application.

[0052] Although the preferred embodiments of the present application have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the present invention is intended to be interpreted as including the preferred embodiments and all changes and modifications falling within the scope of the present application.

[0053] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application belong to the scope of equivalent technologies of the present invention, the present application is also intended to include these modifications and variations.

Claims

1. An underwater visual positioning method based on deep learning visual inertial odometer, characterized in that: include: S1: Build an underwater monocular visual inertial odometry system, which includes a deep learning visual odometry model based on a descriptor loss function with a scale scaling factor and a nonlinear optimization backend; S2: Acquire several underwater videos through an underwater robot equipped with a monocular camera and extract underwater video frames of continuous frames, and form an image pair with each two underwater video frames, and each group of image pairs is constructed as a training set; Use the training set to train the deep learning visual odometry model, so as to obtain a trained underwater deep learning visual odometry model; S3: The underwater video to be located is acquired through an underwater robot equipped with a monocular camera, and the underwater video frames of continuous frames are extracted, which are then input into the underwater deep learning visual odometer model. After the processing is completed, the feature point information of successful association is output and published through the robot operating system ROS communication interface; S4: The nonlinear optimization backend subscribes to the node data of the robot operating system ROS communication interface, and outputs the position and pose of the monocular camera after processing to achieve underwater visual positioning.

2. The underwater visual positioning method based on deep learning visual inertial odometer according to claim 1 is characterized in that: In step S1, the deep learning visual odometer model includes a feature extraction layer and a feature matching layer connected in sequence.

3. The underwater visual positioning method based on deep learning visual inertial odometer according to claim 2 is characterized in that: In step S1, the descriptor loss function of the scale scaling factor is introduced L d as follows: L d (D,D ' , S )=(∑ h,w ∑ h',w' ( l d ( d hw , d' h'w' ; s hwh'w' ))) / ( H c W c ) 2 l d ( d hw , d' h'w' ； s hwh'w' )= λ d × s hwh'w' × max (0， m p -( d hw T d' h'w' ) / ( d k ) 1 / 2 )+(1- s hwh'w' )× max (0，( d hw T d' h'w' ) / ( d k ) 1 / 2 - m n ) h ， h' =1、2、…、 H c w ， w' =1、2、…、 W c Among them, D and D ' They represent the original image and its homography relationship matrix input into the decoder in the feature extraction layer of the deep learning visual odometry model. S The transformed output image, d hw and d' h'w' They represent the original input image D and the transformed output image D respectively. ' The eigenvectors in s hwh'w' Represents the homography relationship matrix S Elements in l d ( ) represents descriptor loss; T represents transpose; H c and W c Denote the input original image D and the output image D respectively ' height and width; λ d , m p and m n Denote the descriptor loss respectively. l d The preset first, second and third hyperparameters of ; d k Represents the scale factor.

4. The underwater visual positioning method based on deep learning visual inertial odometer according to claim 2, characterized in that: In the step S2, for each image pair, two underwater video frames in the image pair are input into the feature extraction layer for processing, and the feature extraction layer extracts each feature point and its descriptor in the two underwater video frames respectively.

5. The underwater visual positioning method based on deep learning visual inertial odometer according to claim 4 is characterized in that: In the step S2, for each image pair, the descriptors of each feature point in the two underwater video frames output by the feature extraction layer are input into the feature matching layer for processing. The feature matching layer matches each descriptor one by one, thereby associating each feature point between the two underwater video frames in the image pair, and publishing the successfully associated feature point information through the robot operating system ROS communication interface.

6. The underwater visual positioning method based on deep learning visual inertial odometer according to claim 2, characterized in that: In the step S2, the deep learning visual odometry model is trained, that is, the feature extraction layer is trained. First, the underwater video frame is subjected to random affine transformation and then input into the feature extraction layer, so as to perform transfer learning on the feature extraction layer to generate pseudo-true value labels, obtain a trained underwater feature extraction layer, and finally obtain a trained underwater deep learning visual odometry model.

7. The underwater visual positioning method based on deep learning visual inertial odometer according to claim 1, characterized in that: In the step S4, an inertial measurement unit IMU is also installed on the underwater robot. After the nonlinear optimization backend subscribes to the node data of the robot operating system ROS communication interface, it is processed through a sliding window optimization algorithm to minimize the weighted sum of the feature point reprojection error and the inertial measurement unit IMU pre-integration error, and finally outputs the monocular camera pose of the camera frequency and the inertial measurement unit IMU frequency.

8. The underwater visual positioning method based on deep learning visual inertial odometer according to claim 1, characterized in that: In the step S4, the deep learning visual odometry model of the underwater monocular visual inertial odometry system runs on a graphics processing unit (GPU) device, and the nonlinear optimization backend runs on a central processing unit (CPU) device.

9. An electronic device, characterized in that: include: A memory and a processor coupled to each other, wherein the memory stores program data, and the processor calls the program data to execute the method according to any one of claims 1 to 8.

10. A computer-readable storage medium having program data stored thereon, characterized in that: When the program data is executed by a processor, the method according to any one of claims 1 to 8 is implemented.

Images