Underwater slam method based on combination of yolov8 and xfeat deep features

Abstract

The application discloses a method for underwater SLAM based on combination of YOLOv8 and XFeat deep features, which comprises preliminary underwater image processing, YOLOv8 image detection, extraction of XFeat features and matching, and then, in the framework of ORB-SLAM3, the whole underwater SLAM system is completed. First, the image extracted by the camera is enhanced. Then, the enhanced image is detected based on YOLOv8, the detection frame of static and dynamic targets is obtained and marked. The XFeat model is a kind of model based on a convolutional neural network architecture, which extracts and matches features by a special strategy, and then the matching result is transmitted to the SLAM system to complete functions such as map construction and camera pose estimation. The application can enable the underwater autonomous robot to overcome the poor optical environment underwater, weaken the interference of dynamic targets, realize accurate positioning of itself, and ensure real-time performance and robustness.

Description

Technical Field

[0001] The present invention belongs to the technical field of underwater robot navigation, and in particular relates to an underwater SLAM method based on the combination of YOLOv8 and XFeat depth features. Background Art

[0002] In recent years, rapid advances in science and technology have driven the rapid development of fundamental technologies related to underwater robotics, such as microprocessors, sensors, and intelligent control. Underwater robots can be equipped with a wide range of sensors, such as sonar, cameras, and depth gauges. These sensors provide information about their surroundings, and their onboard processors process this data autonomously and in real time. Their ability to operate in highly hazardous, polluted, and low-visibility waters has attracted the attention of scholars both domestically and internationally. Consequently, autonomous navigation technology for underwater robots has also continued to advance, attracting extensive research due to its broad application prospects.

[0003] In recent years, underwater robot swarms have garnered widespread attention in the civilian sector, finding applications in deep-sea exploration, environmental monitoring, resource exploration, underwater archaeology, and military applications. In marine resource exploration, underwater robots can map seafloor topography, survey resources, and monitor the surrounding ecosystem. In underwater disaster relief and oil pipeline inspection, they can gather disaster information, conduct underwater search and repair, and more. Furthermore, underwater robots have extensive military applications, performing missions such as intelligence reconnaissance, underwater surveillance, target tracking, and military strikes. In marine battlefield environments, underwater robots autonomously gather intelligence and can conceal themselves and conduct precise strikes, such as stealthily approaching and destroying enemy submarines, surface warships, and aircraft carriers. Major research centers around the world are conducting research on key technologies for unmanned and autonomous underwater robot systems to enable intelligent, autonomous, and efficient underwater combat.

[0004] Underwater robots (AUVs) are capable of operating in extreme marine environments and reaching inaccessible areas, playing an increasingly important role in exploring the unknown. Through continuous breakthroughs in advanced technologies such as autonomous navigation and control, navigation, positioning, and communication, energy and propulsion, target detection and recognition, and robotic manipulation, AUVs are poised to significantly advance their applications in marine environmental observation, deep-sea resource exploration and development, and scientific exploration of the abyss and polar regions. However, the complex underwater environment presents a significant challenge for AUVs to achieve navigation and positioning. Even well-established visual positioning systems on the ground often struggle to provide adequate results. Given space, volume, and energy constraints, a solution requiring ultra-high-performance processors is impractical. AUVs urgently need a cost-effective, yet robust, and functional underwater navigation and positioning solution. Summary of the Invention

[0005] In response to the above technical problems, the present invention provides an underwater SLAM method based on the combination of YOLOv8 and XFeat depth features.

[0006] The technical solution adopted by the present invention to solve the technical problem is:

[0007] An underwater SLAM method based on combining YOLOv8 with XFeat depth features comprises the following steps:

[0008] S100: Acquire an image captured by an underwater camera and perform image enhancement processing on the image to obtain an enhanced image;

[0009] S200: Use YOLOv8 to process the enhanced image, distinguish the objects in the image into dynamic targets and static targets, and mark the detection frame coordinates of these targets;

[0010] S300: Uses lightweight deep features Xfeat to extract features from the image passed by YOLOv8, removes feature points within the target detection box, performs feature matching on the retained feature points, and passes the matching results to the SLAM system, using the feature matching results to achieve precise positioning and mapping.

[0011] Preferably, S100 includes:

[0012] S110: Use the CLAHE algorithm to divide the input image into multiple small blocks and calculate the grayscale histogram of each small block;

[0013] S120: Equalizing the histogram of each small block. If the number of pixels at a certain gray level exceeds a preset threshold, the excess pixels will be evenly distributed to other gray levels.

[0014] S130: Using interpolation, for each pixel, calculate its weight at the center of four adjacent small blocks, and mix the equalization results of the four small blocks according to these weights, and finally output the enhanced image.

[0015] Preferably, S200 includes:

[0016] S210: Use the pre-trained YOLOv8 model to load weights, pass the enhanced image to the YOLOv8 model for object detection, and obtain a detection result; wherein the detection result includes the coordinates, category, and confidence of the detection box;

[0017] S220: According to the detection results, objects in the image are divided into dynamic targets and static targets in advance, and the detection frames of the detected targets are divided into dynamic detection frames and static detection frames.

[0018] Preferably, S300 includes:

[0019] S310: Preprocess and resize the image passed by YOLOv8, load the pre-trained Xfeat model, and pass the pre-processed image to the Xfeat model for feature extraction; the Xfeat model includes a backbone network and independent convolution branches and fusion blocks;

[0020] S320: Divide the preprocessed image into multiple 8×8 blocks, perform a 1×1 convolution operation on each block, and extract a key point heat map K to identify potential key point locations in the image. Each element of the key point heat map represents the probability of the corresponding location being a key point.

[0021] S330: When feature points are within 2 pixels around the acquired dynamic target area coordinates, they are determined to be dynamic feature points; these dynamic feature points are determined to be within the static detection frame. If they are not within any static detection frame, they are marked as feature points to be removed, and their coordinates are changed to (-1, -1) and removed.

[0022] S340: Processing the original input image through the backbone network to extract local feature descriptors from the image. The backbone network generates a compact 64-dimensional dense descriptor graph through a series of convolution operations. The network includes six convolution blocks, each of which downsamples the input image.

[0023] S350: The fusion block integrates features at different levels or scales and outputs the results to obtain a compact 64-D dense descriptor map F and a reliability heat map R. The reliability heat map is used to subsequently evaluate the reliability of each key point, where each element represents the reliability score of the key point at the corresponding position.

[0024] S360: Combine the key point heat map K, the reliability heat map R and perform matching operations on the dense local feature map F, then process the output of the matching operation through the refinement strategy to obtain the final matching result and pass it to the SLAM system.

[0025] Preferably, S360 specifically includes:

[0026] The matching operation is performed on the dense local feature map F. The fusion block extracts the corresponding feature descriptors from the descriptor map F based on the key point positions extracted from the key point heat map K, performs feature point matching, and combines the reliability scores in the reliability heat map R to exclude unreliable key points and pay more attention to reliable key points in the matching process.

[0027] Through the refinement strategy, the output image after the matching operation is first upsampled to the resolution of the original image, and then RANSAC local optimization is performed. Finally, the output after local optimization is weightedly fused with the reliability heat map to obtain the final matching result and pass it to the SLAM system.

[0028] Preferably, before S100, the method further includes:

[0029] Use the startup px4 file under ROS to establish a connection between the onboard computer and the flight controller's imu, and start the camera file through realsense. At the same time, start the SLAM algorithm and subscribe to the data of the imu and camera sensor in the SLAM system.

[0030] The present invention enables the underwater autonomous robot to overcome the harsh optical environment underwater, reduce interference from dynamic targets, and achieve relatively accurate positioning of itself while ensuring real-time and robustness. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 Flowchart of an underwater SLAM method based on the combination of YOLOv8 and XFeat depth features in one embodiment of the present invention;

[0032] Figure 2 This is a flowchart of an underwater SLAM method based on the combination of YOLOv8 and XFeat depth features in another embodiment of the present invention;

[0033] Figure 3 This is a diagram of the XFeat backbone network structure in one embodiment of the present invention;

[0034] Figure 4 It is a flowchart for removing feature points based on YOLOv8. DETAILED DESCRIPTION

[0035] In order to enable those skilled in the art to better understand the technical solution of the present invention, the present invention is further described in detail below with reference to the accompanying drawings.

[0036] In one embodiment, Figure 1 As shown, an underwater SLAM method based on the combination of YOLOv8 and XFeat depth features includes the following steps:

[0037] S100: Acquire an image captured by an underwater camera and perform image enhancement processing on the image to obtain an enhanced image;

[0038] S200: Use YOLOv8 to process the enhanced image, distinguish objects in the image into dynamic targets and static targets, mark the detection frame coordinates of these targets, and remove unnecessary dynamic feature points;

[0039] S300: Uses lightweight deep features Xfeat to extract features from the image passed by YOLOv8, removes feature points within the target detection box, performs feature matching on the retained feature points, and passes the matching results to the SLAM system, using the feature matching results to achieve precise positioning and mapping.

[0040] Specifically, the matching results are passed to the SLAM system, and the current frame is matched with the feature points in the previous frame or map to estimate the current position of the camera by minimizing the reprojection error. Based on the matching results, new map points are created and added to the map. At the same time, the system will also check and eliminate those map points that have not been observed for a long time or conflict with other map points, and use the results of feature matching to achieve accurate positioning and mapping.

[0041] This invention discloses a SLAM method for underwater autonomous robots based on the combination of YOLOv8 and XFeat deep features. This method, which belongs to the field of underwater SLAM, includes preliminary underwater image processing, YOLOv8 image detection, XFeat feature extraction and matching, and then completes the entire underwater SLAM system within the ORB-SLAM3 framework. This invention enables underwater autonomous robots to overcome harsh underwater optical environments, reduce interference from dynamic targets, and achieve relatively accurate positioning while ensuring real-time and robustness.

[0042] In one embodiment, S100 includes:

[0043] S110: Use the CLAHE algorithm to divide the input image into multiple small blocks and calculate the grayscale histogram of each small block;

[0044] S120: Equalizing the histogram of each small block. If the number of pixels at a certain gray level exceeds a preset threshold, the excess pixels will be evenly distributed to other gray levels.

[0045] S130: Using interpolation, for each pixel, calculate its weight at the center of four adjacent small blocks, and mix the equalization results of the four small blocks according to these weights, and finally output the enhanced image.

[0046] Specifically, the images captured by the underwater camera are simply processed and the contrast of the images is adjusted using CLAHE to make the lighting of the images as uniform as possible.

[0047] In one embodiment, Figure 2 As shown, S200 includes:

[0048] S210: Use the pre-trained YOLOv8 model to load weights, pass the enhanced image to the YOLOv8 model for object detection, and obtain a detection result; wherein the detection result includes the coordinates, category, and confidence of the detection box;

[0049] S220: According to the detection results, objects in the image are divided into dynamic targets and static targets in advance, and the detection frames of the detected targets are divided into dynamic detection frames and static detection frames.

[0050] Specifically, YOLOv8 is used for processing to distinguish objects in the image and divide them into dynamic targets and static targets. For example, schools of fish and floating objects are classified as dynamic targets, while stones and shipwreck debris are classified as static targets, and the detection frame coordinates of these targets are marked.

[0051] In one embodiment, Figure 3 and Figure 4 As shown, S300 includes:

[0052] S310: Preprocess and resize the image passed by YOLOv8, load the pre-trained Xfeat model, and pass the pre-processed image to the Xfeat model for feature extraction; the Xfeat model includes a backbone network and independent convolution branches and fusion blocks;

[0053] S320: Divide the preprocessed image into multiple 8×8 blocks, perform a 1×1 convolution operation on each block, and extract a key point heat map K to identify potential key point locations in the image. Each element of the key point heat map represents the probability of the corresponding location being a key point.

[0054] S330: When feature points are within 2 pixels around the acquired dynamic target area coordinates, they are determined to be dynamic feature points; these dynamic feature points are determined to be within the static detection frame. If they are not within any static detection frame, they are marked as feature points to be removed, and their coordinates are changed to (-1, -1) and removed.

[0055] S340: Processing the original input image through the backbone network to extract local feature descriptors from the image. The backbone network generates a compact 64-dimensional dense descriptor graph through a series of convolution operations. The network includes six convolution blocks, each of which downsamples the input image.

[0056] S350: The fusion block integrates features at different levels or scales and outputs the results to obtain a compact 64-D dense descriptor map F and a reliability heat map R. The reliability heat map is used to subsequently evaluate the reliability of each key point, where each element represents the reliability score of the key point at the corresponding position.

[0057] S360: Combine the key point heat map K, the reliability heat map R and perform matching operations on the dense local feature map F, then process the output of the matching operation through the refinement strategy to obtain the final matching result and pass it to the SLAM system.

[0058] Specifically, in S320, a convolution operation is performed on each small block, which can effectively extract the feature information within the small block while maintaining computational efficiency. Such a judgment condition is set in S330 because, when encountering feature points that are simultaneously in multiple detection frames, in order not to eliminate too much image information, as many valid points as possible are retained. In S340, each convolution block downsamples the input image, which can reduce the height and width of the image, reduce the amount of calculation while retaining the main feature information, and increase the number of channels through the convolution layer. The features of different levels or scales in S350 are specifically shallow features and deep features. The shallow features contain detail information, and the deep features contain semantic information.

[0059] In one embodiment, S360 specifically includes:

[0060] The matching operation is performed on the dense local feature map F. The fusion block extracts the corresponding feature descriptors from the descriptor map F based on the key point positions extracted from the key point heat map K, performs feature point matching, and combines the reliability scores in the reliability heat map R to exclude unreliable key points and pay more attention to reliable key points in the matching process.

[0061] Through the refinement strategy, the output image after the matching operation is first upsampled to the resolution of the original image, and then RANSAC local optimization is performed. Finally, the output after local optimization is weightedly fused with the reliability heat map to obtain the final matching result and pass it to the SLAM system.

[0062] In one embodiment, before S100, the method further includes:

[0063] Use the startup px4 file under ROS to establish a connection between the onboard computer and the flight controller's imu, and start the camera file through realsense. At the same time, start the SLAM algorithm and subscribe to the data of the imu and camera sensor in the SLAM system.

[0064] The above-mentioned underwater SLAM method based on the combination of YOLOv8 and XFeat deep features has the following initial underwater image processing step: contrast-limited adaptive histogram equalization is performed on the image captured by the camera; the YOLOv8 image detection step is to perform target detection on the image after the initial histogram equalization, obtain the detection box of fish (dynamic targets) and sea urchins (static targets) and remove dynamic feature points; XFeat features are based on a convolutional neural network (CNN) architecture, using a special strategy for feature extraction and matching, and then passing the results to the SLAM system to complete functions such as map construction and camera pose estimation;

[0065] The present invention has the following advantages:

[0066] 1. Most learning-based object detection methods often consume a significant amount of time and computing power, preventing the processor from achieving real-time processing. To address this issue, a YOLOv8-based filtering strategy was adopted on the underwater robot's onboard processor. This strategy minimizes the interference of dynamic objects while ensuring real-time performance and minimizing computing power consumption.

[0067] 2. Traditional feature points are sensitive to changes in lighting and other factors in complex environments, leading to unstable feature points and even loss of tracking. XFeat is used for feature extraction and matching, ensuring high efficiency and accuracy while being deployed on microcomputers. It is five times faster than traditional SuperPoint and far more accurate than feature points such as ORB under varying lighting conditions and large rotations.

[0068] 3. It is realized that in complex underwater environments, it can run in real time on the processor carried by the underwater robot, perceive the environment, locate and build maps, use active light sources to detect the environment, and obtain image information with a binocular camera. Through the above method, the feasibility of underwater SLAM is greatly enhanced, meeting the requirements of autonomous navigation of underwater robots underwater.

[0069] The above is a detailed introduction to the underwater SLAM method based on the combination of YOLOv8 and XFeat depth features provided by the present invention. This article uses specific examples to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the core idea of ​​the present invention. It should be pointed out that for ordinary technicians in this technical field, without departing from the principles of the present invention, the present invention can also be improved and modified, and these improvements and modifications also fall within the scope of protection of the claims of the present invention.

Claims

1. An underwater SLAM method based on the combination of YOLOv8 and XFeat depth features, characterized by: The method comprises the following steps: S100: Acquire an image captured by an underwater camera and perform image enhancement processing on the image to obtain an enhanced image; S200: Use YOLOv8 to process the enhanced image, distinguish the objects in the image into dynamic targets and static targets, and mark the detection frame coordinates of these targets; S300: Uses lightweight deep features Xfeat to extract features from the image passed by YOLOv8, removes feature points within the target detection frame, performs feature matching on the retained feature points, and passes the matching results to the SLAM system. The feature matching results are used to achieve precise positioning and mapping. S300 includes: S310: Preprocess and resize the image passed by YOLOv8, load the pre-trained Xfeat model, and pass the pre-processed image to the Xfeat model for feature extraction; the Xfeat model includes a backbone network and independent convolution branches and fusion blocks; S320: Divide the preprocessed image into multiple 8×8 blocks, perform a 1×1 convolution operation on each block, and extract a key point heat map K to identify potential key point locations in the image. Each element of the key point heat map represents the probability of the corresponding location being a key point. S330: When feature points are within 2 pixels around the acquired dynamic target area coordinates, they are determined to be dynamic feature points; these dynamic feature points are determined to be within the static detection frame. If they are not within any static detection frame, they are marked as feature points to be removed, and their coordinates are changed to (-1, -1) and removed. S340: Processing the original input image through the backbone network to extract local feature descriptors from the image. The backbone network generates a compact 64-dimensional dense descriptor graph through a series of convolution operations. The network includes six convolution blocks, each of which downsamples the input image. S350: The fusion block integrates features at different levels or scales and outputs the results to obtain a compact 64-D dense descriptor map F and a reliability heat map R. The reliability heat map is used to subsequently evaluate the reliability of each key point, where each element represents the reliability score of the key point at the corresponding position. S360: Combine the key point heat map K, the reliability heat map R and perform matching operations on the dense local feature map F, then process the output of the matching operation through the refinement strategy to obtain the final matching result and pass it to the SLAM system.

2. The method according to claim 1, characterized in that S100 includes: S110: Use the CLAHE algorithm to divide the input image into multiple small blocks and calculate the grayscale histogram of each small block; S120: Equalizing the histogram of each small block. If the number of pixels at a certain gray level exceeds a preset threshold, the excess pixels will be evenly distributed to other gray levels. S130: Using interpolation, for each pixel, calculate its weight at the center of four adjacent small blocks, and mix the equalization results of the four small blocks according to these weights, and finally output the enhanced image.

3. The method according to claim 2, characterized in that S200 includes: S210: Use the pre-trained YOLOv8 model to load weights, pass the enhanced image to the YOLOv8 model for object detection, and obtain a detection result; wherein the detection result includes the coordinates, category, and confidence of the detection box; S220: According to the detection results, objects in the image are divided into dynamic targets and static targets in advance, and the detection frames of the detected targets are divided into dynamic detection frames and static detection frames.

4. The method according to claim 3, characterized in that S360 specifically includes: The matching operation is performed on the dense local feature map F. The fusion block extracts the corresponding feature descriptors from the descriptor map F based on the key point positions extracted from the key point heat map K, performs feature point matching, and combines the reliability scores in the reliability heat map R to exclude unreliable key points and pay more attention to reliable key points in the matching process. Through the refinement strategy, the output image after the matching operation is first upsampled to the resolution of the original image, and then RANSAC local optimization is performed. Finally, the output after local optimization is weightedly fused with the reliability heat map to obtain the final matching result and pass it to the SLAM system.

5. The method according to claim 4, characterized in that Before S100, it also included: Use the startup px4 file under ROS to establish a connection between the onboard computer and the flight controller's imu, and start the camera file through realsense. At the same time, start the SLAM algorithm and subscribe to the data of the imu and camera sensor in the SLAM system.

Images