Autonomous positioning method and device for unmanned boat

Abstract

The present invention provides an autonomous positioning method and device for an unmanned vehicle. The method comprises: establishing a waterside segmentation model; extracting and matching feature points using an improved SuperPoint method to establish a deep learning feature point extraction model; optimizing weighted reprojection errors; constructing an autonomous positioning model for the unmanned vehicle; and inputting a sequence of real-time image frames acquired by an onboard camera into the constructed autonomous positioning model to estimate the unmanned vehicle's navigation trajectory. The present invention achieves precise positioning of the unmanned vehicle by integrating a traditional SLAM framework with an improved deep learning method.

Description

Technical Field

[0001] The present invention relates to the technical field of autonomous positioning of unmanned boats, and in particular to an autonomous positioning method and device for unmanned boats used in inland rivers. Background Art

[0002] In recent years, unmanned boats (UAVs) have become increasingly prominent due to their ability to perform complex tasks on the water. Strong demand from the commercial, scientific, and environmental communities has accelerated their application and development. Common applications for UAVs include hydrographic surveying and mapping, marine resource exploration, water quality testing, and floating waste removal. Compared to maritime and coastal UAVs, UAVs for use in narrow inland waterways are closely related to human life and hold greater potential research value.

[0003] Unlike sailing in open sea waters, the challenges posed by narrow inland waterways, such as the complex distribution of obstacles in the environment, interference with GPS signals, dynamic ripples on the water surface, fog, and strong light reflections, all hinder the application of unmanned boats in narrow inland waterways. Based on the needs and challenges of perception and positioning of unmanned boats in narrow inland waterways, the Simultaneous Localization and Mapping (SLAM) method can be used to achieve accurate positioning of unmanned boats in narrow inland waterways. However, since LiDAR SLAM is affected by fog and strong light reflections on the water surface, the pose estimation is inaccurate, and its expensive sensor equipment is not suitable for small civilian narrow inland waterway unmanned boats.

[0004] Due to the rapid development of deep learning, feature point detection and matching methods based on convolutional neural networks have been applied in SLAM systems. Deep learning methods are integrated into new SLAM systems by replacing the corresponding feature point extraction and matching methods in the original SLAM system. However, due to the dependence of feature point detection and matching methods on training scenarios, in order to improve the positioning accuracy of SLAM systems in complex scenarios of narrow inland rivers, and comprehensively considering factors such as the impact of dynamic water ripples on feature point extraction in narrow inland river environments, the non-robustness of traditional feature point extraction, and the impact of repeated textures of shore vegetation on pose estimation, a SLAM system suitable for autonomous positioning of unmanned boats in narrow inland rivers was constructed from the perspective of module replacement of the traditional SLAM framework with deep learning. Summary of the Invention

[0005] In view of the defects in the prior art, the purpose of the present invention is to provide a method and device for autonomous positioning of an unmanned boat, which realizes the precise positioning of the unmanned boat by integrating the traditional SLAM framework with the improved deep learning method.

[0006] In order to solve the above problems, the technical solution of the present invention is:

[0007] An autonomous positioning method for an unmanned boat comprises the following steps:

[0008] Establish a waterside segmentation model;

[0009] Use the improved SuperPoint method to extract and match feature points and establish a deep learning feature point extraction model;

[0010] Construct weighted reprojection error optimization;

[0011] Construct an autonomous positioning model for unmanned boats;

[0012] The real-time image frame sequence acquired by the onboard camera is input into the constructed unmanned vehicle autonomous positioning model to obtain the unmanned vehicle navigation trajectory estimation.

[0013] Preferably, the step of establishing a waterbank segmentation model specifically includes: establishing a waterbank segmentation model Inland-DeepLab v3+ based on the DeepLab v3+ algorithm, and the Inland-DeepLab v3+ algorithm is obtained by outputting an additional layer of low-level features in the input layer of the DeepLab v3+ algorithm encoder and repeating its middle layer 8 times.

[0014] Preferably, the step of establishing the water bank segmentation model specifically includes: setting the weight ratio of the water surface area and the background area Where w0 is the background pixel weight and w1 is the water surface area pixel weight.

[0015] Preferably, the step of using the improved SuperPoint method to extract and match feature points and establish a deep learning feature point extraction model specifically includes: adding the CBAM attention mechanism to the SuperPoint method to establish a deep learning feature point extraction model Attention-SuperPoint.

[0016] Preferably, the step of constructing weighted reprojection error optimization specifically includes: performing image saliency detection on the image with extracted feature points to obtain a grayscale image, normalizing the weights Among them, w i is the weight of the i-th feature point; p i is the pixel value in the saliency image of each feature point; b is the weight optimization coefficient. After determining the weight coefficient, the weighted reprojection error formula is obtained.

[0017] Among them, P L Represents the pose of the current key frame and the local key frame with which the current key frame has a co-viewing relationship, T L represents the coordinates of all observable three-dimensional points in the local keyframe, ρ represents the Huber function, and χj is the set of feature points that match the mapping points of the current frame and the jth local key frame, and E(j,k) is defined as

[0018] Preferably, the step of constructing an unmanned boat autonomous positioning model specifically includes: constructing an unmanned boat autonomous positioning model Inland-SLAM, wherein the Inland-SLAM removes the water surface noise containing dynamic water ripples and reflections of riverbank vegetation through a water bank segmentation method, and extracts and matches feature points of image frames through an Attention-SuperPoint method based on deep learning, and optimizes the weighted reprojection error.

[0019] Furthermore, the present invention also provides an autonomous positioning device for an unmanned boat, the device comprising:

[0020] Sensor information reading module: used to establish the waterfront segmentation model Inland-DeepLab v3+ based on the DeepLab v3+ algorithm;

[0021] Visual odometry module: used to add the CBAM attention mechanism to the SuperPoint method and establish the deep learning feature point extraction model Attention-SuperPoint;

[0022] Backend optimization module: used to optimize the weighted reprojection error during the unmanned boat pose and 3D point optimization process through image saliency SalGAN;

[0023] Loop closure detection module: used to establish a bag-of-words model based on Attention-SuperPoint feature points to achieve fast and accurate matching of identical images and obtain a globally consistent map;

[0024] Model fusion and application module: used to output the unmanned boat navigation trajectory estimation for the input image frame through the unmanned boat autonomous positioning model Inland-SLAM.

[0025] Preferably, the Inland-DeepLab v3+ algorithm used by the sensor information reading module is based on the DeepLab v3+ algorithm, which is obtained by outputting an extra layer of low-level features in its encoder input layer and repeating its intermediate layer 8 times.

[0026] Preferably, the improved Attention-SuperPoint algorithm in the visual odometry module is obtained by adding the CBAM attention mechanism to its backbone network.

[0027] Preferably, the weighted reprojection error optimization formula is:

[0028] Among them, P L represents the pose of the current keyframe and the local keyframes that have a co-viewing relationship with the current keyframe, T L represents the coordinates of all observable three-dimensional points in the local keyframe, ρ represents the Huber function, and χ j is the set of feature points that match the mapping points of the current frame and the jth local key frame. E(j,k) is defined as

[0029] Compared with the existing technology, the present invention shields water surface noise through water bank segmentation, extracts and matches feature points through an improved attention mechanism, enhances adaptability to the water surface environment, and reduces the weights of feature points in repeated texture areas through weighted reprojection error optimization, thereby obtaining accurate and robust unmanned boat trajectory estimation in narrow inland river environments, making the autonomous positioning of the unmanned boat closer to the real trajectory. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Other features, objects and advantages of the present invention will become more apparent upon reading the detailed description of non-limiting embodiments with reference to the following drawings:

[0031] Figure 1 A flowchart of the autonomous positioning method for an unmanned boat provided by an embodiment of the present invention;

[0032] Figure 2 Flowchart of the Inland-DeepLab v3+ algorithm provided by an embodiment of the present invention;

[0033] Figure 3 Flowchart of the Attention-SuperPoint algorithm provided by an embodiment of the present invention;

[0034] Figure 4 A flowchart for constructing Inland VOC provided by an embodiment of the present invention;

[0035] Figure 5 This is a structural block diagram of the autonomous positioning device for an unmanned boat provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0036] The present invention will be described in detail below with reference to specific embodiments. The following examples will help those skilled in the art to further understand the present invention, but are not intended to limit the present invention in any form. It should be noted that, for those skilled in the art, several changes and improvements can be made without departing from the scope of the present invention. These all fall within the scope of protection of the present invention.

[0037] Specifically, in response to the navigation difficulties of unmanned boats in narrow inland river environments, the present invention provides an autonomous positioning method for unmanned boats in inland rivers, such as Figure 1As shown, the method includes the following steps:

[0038] S1: Establish a waterside segmentation model;

[0039] Specifically, the application scenario in this embodiment is a narrow inland waterway, aiming to achieve water bank segmentation. In order to solve the problems of dynamic water surface ripples and strong light reflection on the water surface, the present invention designs a water bank segmentation network based on DeepLab v3+ and establishes a water bank segmentation model Inland-DeepLab v3+. For the sake of convenience, the network that has been improved on DeepLab v3+ for the self-collected data set is called Inland-DeepLab v3+. The specific process of the Inland-DeepLab v3+ algorithm is as follows: Figure 2 As shown, the Inland-DeepLab v3+ water bank segmentation algorithm is used to preprocess the image sequence. This method increases feature information through multi-layer low-level feature output on the basis of DeepLab v3+ semantic segmentation, and can directly and effectively remove water surface noise interference. In order to further improve the efficiency of the algorithm, Inland-DeepLab v3+ repeats the convolution operation of the middle layer of the Xception model 8 times without much loss of accuracy. At the same time, in order to more finely distinguish small edge areas, Inland-DeepLab v3+ outputs one more level of low-level features during the decoding process, and finally obtains the final output of the water bank segmentation result after a two-channel convolution. Due to the imbalance in the number of pixels in the water surface area and the background area in the training set, it may cause the loss function of the water bank segmentation algorithm to oscillate, so it is necessary to set the weight ratio of the water surface area and the background area. Where w0 is the background pixel weight and w1 is the water surface area pixel weight.

[0040] To evaluate the effectiveness of the improved Inland-DeepLab v3+ algorithm and the original DeepLab v3+ algorithm for narrow inland river bank segmentation, the results were evaluated using the mean intersection over union (IoU) and time, important metrics for measuring image segmentation accuracy. The results are shown in Table 1. The experimental results show that in narrow inland waterway environments, the improved Inland-DeepLab v3+ achieves slightly higher overall segmentation performance than the DeepLab v3+ network. To demonstrate that the system incorporating the Inland-DeepLab v3+ module can effectively remove surface feature points, concentrating them in a more robust region and thus achieving more accurate pose estimation, the root mean square error (RMSE) of the absolute trajectory errors of the unmanned vehicle (UV) obtained by the two algorithms is compared in Table 2, where Seq represents the sequence number in the USVInland dataset for narrow inland river scenarios, and Length represents the trajectory length. This experiment validates the feasibility of incorporating Inland-DeepLab v3+ for SLAM tasks in narrow inland waterway environments.

[0041] network Average Intersection-Union Ratio Time / ms DeepLab v3+ 94.15％ 68 Inland-DeepLab v3+ 94.55％ 63

[0042] Table 1

[0043]

[0044] Table 2

[0045] S2: Use the improved SuperPoint method to extract and match feature points and establish a deep learning feature point extraction model;

[0046] Specifically, a deep learning feature point extraction model is established. For ease of understanding, this method will refer to the SuperPoint method with the addition of the CBAM attention mechanism as Attention-SuperPoint. It first needs to train the basic network through a synthetic virtual dataset with a learning rate of 0.001. Secondly, the COCO dataset is annotated using the trained basic network. Finally, the Attention-SuperPoint is trained using the trained dataset to obtain the feature points in the image and their corresponding descriptors. It is worth noting that homography transformation is used in the process of training the basic network, which aims to improve the generalization ability of the algorithm in real images. Figure 3 As shown, the Attention-SuperPoint backbone network is composed of conv3-64, conv3-64, max-pooling, conv3-64, conv3-64, max-pooling, CBAM, conv3-128, conv3-128, max-pooling, conv3-128, conv3-128.

[0047] To better illustrate the network's improved performance after adding the attention mechanism, experiments were conducted on feature point repetition rate on the improved network before and after the improvement. The experimental parameters remained largely unchanged, with the slight difference being that, because the CBAM module includes convolutional and fully connected layers, L2 regularization was applied to the weight parameters of these layers to prevent overfitting, with a weight decay coefficient of 0.0005. The experimental results for evaluating the repetition rate of feature point extraction for the improved and improved algorithms in image pairs with varying illumination are shown in Table 3 below. This demonstrates that adding the CBAM module to the SuperPoint backbone network effectively improves its feature point detection capabilities.

[0048] algorithm Lighting changes Time / ms SuperPoint 58.9％ 48.4％ Attention-SuperPoint 60.7％ 50.9％

[0049] Table 3

[0050] S3: Construct weighted reprojection error optimization;

[0051] Aiming at the interference of repeated texture of shore vegetation on feature point matching, the method of the present invention annotates the image sequence with image saliency and reassigns the Attention-SuperPoint feature points in the image according to different regions, aiming to reduce the weight of feature points in the repeated texture area in the reprojection error and optimize the posture of the unmanned boat, thereby realizing weighted reprojection error (Weight-BA).

[0052] Specifically, first perform image saliency detection on the image with feature points extracted in step S2 to obtain a grayscale image, where the value of the white part is 255 and the value of the black part is 0. In order to use the saliency map as the weight, the weight is normalized. Among them, w i is the weight of the i-th feature point; p i is the pixel value in the saliency image of each feature point; b is the weight optimization coefficient, which aims to avoid tracking failure due to too small saliency value. After determining the weight coefficient, the weighted reprojection error formula is:

[0053] Among them, P L Represents the pose of the current key frame and the local key frame with which the current key frame has a co-viewing relationship, T L represents the coordinates of all observable 3D points in the local keyframe. ρ represents the Huber function, χ j is the set of feature points that match the mapping points of the current frame and the jth local key frame, and E(j,k) is defined as

[0054] S4: Build an autonomous positioning model for unmanned boats.

[0055] Specifically, in this embodiment, an autonomous positioning model Inland-SLAM for unmanned boats in narrow inland rivers is constructed. Inland-SLAM is divided into three threads, namely, image input and tracking thread, local mapping thread and closed-loop detection thread. In the image input and tracking thread, Inland-SLAM removes the surface noise containing dynamic water ripples and reflections of riverbank vegetation through the narrow inland river water bank segmentation method Inland-DeepLab v3+, and then extracts and matches feature points of the narrow inland river image frame through the Attention-SuperPoint method based on deep learning. After filtering the feature points through image saliency, the pose of the unmanned boat is estimated based on the matching of the feature points. At this time, the estimated pose is not accurate and needs to be optimized by weighted reprojection error (Weight Bundle Adjustment, Weight-BA) later. In the local mapping thread of Inland-SLAM, the pose estimated by the visual odometry is obtained in real time, and the estimated pose information is optimized through Weight-BA, marginalization, etc. The closed-loop detection thread of Inland-SLAM compares the real-time image information obtained by the onboard high-definition camera with the historical image information in the database based on the established narrow inland river visual word bag model Inland VOC, thereby generating a closed-loop response, eliminating the accumulated error of the visual SLAM system under long-term operation, and obtaining an accurate navigation trajectory of the unmanned boat in narrow inland rivers. The construction process of the narrow inland river visual word bag model Inland VOC is as follows: Figure 4 shown.

[0056] Table 4 below shows an evaluation of the bag-of-words model for narrow inland waterways. ORB2 VOC represents the original bag-of-words model used by ORB-SLAM2, while Inland VOC represents the bag-of-words model used by the Inland-SLAM software. Under the same experimental environment and scenario, the query time for the original bag-of-words model used by ORB-SLAM2 in narrow inland waterways was 9.315ms, while the bag-of-words model used by Inland-SLAM only took 5.198ms. This demonstrates that the bag-of-words model for narrow inland waterways not only improves the stability and robustness of the overall system in surface water environments, but also further meets the real-time requirements for unmanned vehicle positioning in narrow inland waterways.

[0057] Bag of Words Model Query time / ms ORB2 VOC 9.315 Inland VOC 5.198

[0058] Table 4

[0059] S5: The real-time image frame sequence acquired by the onboard camera is input into the constructed autonomous positioning model of the unmanned boat to obtain the estimated navigation trajectory of the unmanned boat.

[0060] Specifically, by inputting the real-time image frame sequence acquired by the high-definition camera on board into the Inland-SLAM narrow inland river unmanned boat autonomous positioning model in step S4, an accurate and robust estimation of the unmanned boat navigation trajectory can be obtained. By comparing the root mean square error (RMSE / APE) of the absolute trajectory error of the navigation trajectory of ORB-SLAM2, SuperPoint-SLAM and the method Inland-SLAM of the present invention in a narrow inland river environment, the specific values are shown in Table 5 below, where Seq represents the sequence of the used scene in the USVInland dataset, and Length represents the length of the narrow inland river. The specific application of the model demonstrates that the method of the present invention can obtain a positioning trajectory that is closer to the actual navigation trajectory of the unmanned boat, and demonstrates that the improved Inland-SLAM algorithm has strong practicality and robustness in a narrow inland river environment.

[0061]

[0062] Table 5

[0063] Further, if Figure 5 As shown, the present invention also provides an autonomous positioning device for an unmanned boat, the device comprising:

[0064] Sensor Information Reading Module 1: Used to build the Inland-DeepLab v3+ waterfront segmentation model based on the DeepLab v3+ algorithm. The Inland-DeepLab v3+ algorithm is achieved by outputting an additional layer of low-level features in the input layer of the DeepLab v3+ algorithm encoder and repeating the middle layer eight times.

[0065] In narrow inland river environments, effective preprocessing of the image frame sequences captured by the camera is required. First, due to the narrow channel, images captured on the surface of narrow inland rivers are relatively rippled. Dynamic water ripples can affect visual SLAM positioning. Second, strong light reflections from the water surface and the reflections of vegetation and buildings on both sides of the river can affect positioning accuracy. Therefore, the Inland-DeepLab v3+ water and bank segmentation network was trained for water surface environments to achieve effective preprocessing of narrow inland river images.

[0066] Visual odometry module 2: used to add the CBAM attention mechanism to the SuperPoint method and establish the deep learning feature point extraction model Attention-SuperPoint;

[0067] This module uses multiple frames of images captured by the unmanned vehicle's high-definition camera to roughly estimate the vehicle's position and trajectory within narrow inland waterways. By replacing traditional feature point extraction with an improved deep learning feature extraction and matching method called Attention-SuperPoint, the module aims to achieve more robust and accurate feature point extraction and matching in challenging scenarios with significant changes in lighting and perspective.

[0068] Backend optimization module 3: used to optimize the weighted reprojection error in the process of optimizing the posture of the unmanned boat and the three-dimensional points through image saliency SalGAN:

[0069] Among them, P L represents the pose of the current keyframe and the local keyframes that have a co-viewing relationship with the current keyframe, T L represents the coordinates of all observable 3D points in the local keyframe. ρ represents the Huber function, χ j is the set of feature points that match the mapping points of the current frame and the jth local key frame. E(j,k) is defined as

[0070] Loop closure detection module 4: used to establish an Attention-SuperPoint feature point bag-of-words model to quickly and accurately match identical images and obtain a globally consistent map;

[0071] Model fusion application module 5: For the input image frame, the tracking thread, local mapping thread, and loop detection thread in the unmanned boat autonomous positioning model Inland-SLAM output accurate and robust unmanned boat navigation trajectory estimation in narrow inland environments.

[0072] In summary, the present invention uses Inland-DeepLab v3+ to shield water surface noise, uses the improved attention mechanism Attention-SuperPoint to extract and match feature points, enhances adaptability to the water surface environment, and reduces the weight of feature points in repeated texture areas through weighted reprojection error optimization, thereby obtaining accurate and robust UAV trajectory estimation in narrow inland river environments, making the autonomous positioning of the UAV closer to the real trajectory.

[0073] The above describes specific embodiments of the present invention. It should be understood that the present invention is not limited to the specific embodiments described above, and those skilled in the art may make various changes or modifications within the scope of the claims, which do not affect the essence of the present invention. The embodiments of this application and the features in the embodiments may be combined with each other in any manner unless there is a conflict.

Claims

1. An autonomous positioning method for an unmanned boat, characterized in that: The method comprises the following steps: Establish a waterfront segmentation model, specifically including: establishing the Inland-DeepLab v3+ waterfront segmentation model based on the DeepLab v3+ algorithm. The Inland-DeepLab v3+ algorithm is obtained by outputting an additional layer of low-level features in the input layer of the DeepLab v3+ algorithm encoder and repeating its middle layer 8 times; The improved SuperPoint method is used to extract and match feature points, and a deep learning feature point extraction model is established. Specifically, the CBAM attention mechanism is added to the SuperPoint method to establish a deep learning feature point extraction model Attention-SuperPoint. Construct weighted reprojection error optimization, specifically including: performing image saliency detection on the image with extracted feature points to obtain a grayscale image, normalizing the weights Among them, w i is the weight of the i-th feature point; p i is the pixel value in the saliency image of each feature point; b is the weight optimization coefficient. After determining the weight coefficient, the weighted reprojection error formula is obtained. Among them, P L Represents the pose of the current key frame and the local key frame with which the current key frame has a co-viewing relationship, T L represents the coordinates of all observable three-dimensional points in the local keyframe, ρ represents the Huber function, and χ j is the set of feature points that match the mapping points of the current frame and the jth local key frame, and E(j,k) is defined as Construct an autonomous positioning model for unmanned boats; The real-time image frame sequence acquired by the onboard camera is input into the constructed unmanned vehicle autonomous positioning model to obtain the unmanned vehicle navigation trajectory estimation.

2. The autonomous positioning method of an unmanned boat according to claim 1, characterized in that: The steps of establishing the waterfront segmentation model specifically include: setting the weight ratio of the water surface area and the background area Where w0 is the background pixel weight and w1 is the water surface area pixel weight.

3. The autonomous positioning method for an unmanned boat according to claim 1, characterized in that: The steps of constructing the unmanned boat autonomous positioning model specifically include: constructing the unmanned boat autonomous positioning model Inland-SLAM, the Inland-SLAM removes the water surface noise containing dynamic water ripples and reflections of riverbank vegetation through the water bank segmentation method, and extracts and matches feature points of image frames through the Attention-SuperPoint method based on deep learning, and optimizes the weighted reprojection error.

4. An autonomous positioning device for an unmanned boat, characterized in that: The device comprises: Sensor Information Reading Module: This module is used to build the Inland-DeepLabv3+ waterfront segmentation model based on the DeepLab v3+ algorithm. The Inland-DeepLab v3+ algorithm is based on the DeepLab v3+ algorithm by outputting an additional layer of low-level features in its encoder input layer and repeating this process eight times on its intermediate layer. Visual odometry module: This module is used to add the CBAM attention mechanism to the SuperPoint method to establish the Attention-SuperPoint deep learning feature point extraction model. The Attention-SuperPoint algorithm is obtained by adding the CBAM attention mechanism to its backbone network. Backend optimization module: used to optimize the weighted reprojection error in the process of optimizing the posture of the unmanned boat and the three-dimensional points through image saliency SalGAN. The weighted reprojection error optimization formula is: Among them, P L represents the pose of the current keyframe and the local keyframes that have a co-viewing relationship with the current keyframe, T L represents the coordinates of all observable three-dimensional points in the local keyframe, ρ represents the Huber function, and χ j is the set of feature points that match the mapping points of the current frame and the jth local key frame. E(j,k) is defined as Loop detection module: used to establish a bag-of-words model based on Attention-SuperPoint feature points to achieve fast and accurate matching of identical images and obtain a globally consistent map; Model fusion and application module: used to output the unmanned boat navigation trajectory estimation for the input image frame through the unmanned boat autonomous positioning model Inland-SLAM.

Images