Underwater SLAM (Simultaneous Localization and Mapping) method based on combination of YOLOv8 and XFat depth features

Abstract

The invention discloses an underwater SLAM (Simultaneous Localization and Mapping) method based on combination of YOLOv8 and XFat depth features, and the underwater SLAM method comprises the steps: preliminary underwater image processing, YOLOv8 image detection, XFat feature extraction and matching, and then completion of a whole underwater SLAM system under an ORB-SLAM 3 framework. Firstly, an image extracted by a camera is enhanced; carrying out target detection on the enhanced image based on YOLOv8, and obtaining and marking a detection frame of a static target and a detection frame of a dynamic target; the XFeat model is based on a convolutional neural network architecture, performs feature extraction and matching by using a special strategy, and then transmits a matching result to an SLAM system to complete functions of map construction, camera pose estimation and the like; according to the method, the underwater autonomous robot can overcome the underwater severe optical environment, dynamic target interference is weakened, accurate positioning is achieved, and meanwhile real-time performance and robustness are guaranteed.

Description

Technical Field

[0001] The invention belongs to the technical field of underwater robot navigation, and in particular relates to an underwater SLAM method based on the combination of YOLOv8 and XFeat depth features. Background Art

[0002] In recent years, the rapid progress of science and technology has promoted the rapid development of basic technologies in the field of underwater robots, such as microprocessor technology, sensor technology and intelligent control technology. Underwater robots can be equipped with a variety of sensors, such as sonar, cameras and depth meters. Underwater robots obtain information about the surrounding environment through these sensors, and can process relevant data in real time and autonomously through their own processors. Underwater robots have attracted the attention of scholars at home and abroad for their ability to work in high-risk, polluted and zero-visibility water environments. Under this premise, the autonomous navigation technology of underwater robots has also been continuously developed, and researchers have been exploring it more comprehensively due to its broad application prospects.

[0003] In recent years, underwater robot clusters have received extensive attention in the civilian field, and have been widely used in deep-sea exploration, environmental monitoring, resource exploration, submarine archaeology, and military applications. In marine resource exploration, underwater robots can complete the mapping of seabed topography, resource exploration, and monitoring of surrounding ecology. In underwater disaster rescue, oil pipeline inspection and other tasks, underwater robots can complete disaster information collection, underwater search and repair tasks. At the same time, underwater robots are widely used in the military field and can perform combat tasks such as intelligence reconnaissance, underwater surveillance, target tracking, and military strikes. In the marine battlefield environment, underwater robots can autonomously collect intelligence and can conceal the environment and accurately strike, such as stealthily approaching and destroying enemy underwater submarines, surface warships, and aircraft carriers. Major research centers around the world are deploying key technology research on underwater robot systems in the unmanned autonomous aspect to achieve intelligent, autonomous, and efficient underwater combat.

[0004] Underwater robots can work in extreme marine environments and reach sea areas that are difficult for humans to reach, playing an increasingly important role in exploring the unknown world of mankind. Through continuous breakthroughs in high-tech technologies such as autonomous navigation control, navigation positioning and communication, energy power propulsion, target detection and identification, and manipulator operation, underwater robots will effectively promote their applications in the fields of marine environment observation, deep-sea resource exploration and development, and scientific exploration of the abyss and polar regions. However, the underwater environment is complex, and how to enable underwater robots to complete navigation and positioning is a huge problem. The very mature visual positioning system on the ground often does not work well. In the case of limited space, volume, and energy, it is not realistic to use ultra-high-performance processors. Underwater robots are in urgent need of a low-cost underwater navigation and positioning solution with perfect effects and functions. Summary of the invention

[0005] In view of the above technical problems, the present invention provides an underwater SLAM method based on the combination of YOLOv8 and XFeat depth features.

[0006] The technical solution adopted by the present invention to solve the technical problem is:

[0007] The underwater SLAM method based on the combination of YOLOv8 and XFeat depth features comprises the following steps:

[0008] S100: Acquire an image captured by an underwater camera, and perform enhancement processing on the image to obtain an enhanced image;

[0009] S200: YOLOv8 is used to process the enhanced image, distinguish the objects in the image into dynamic targets and static targets, and mark the detection frame coordinates of these targets;

[0010] S300: Use lightweight deep feature Xfeat to extract features from the image passed by YOLOv8, remove feature points within the target detection frame, perform feature matching on the retained feature points, pass the matching results to the SLAM system, and use the feature matching results to achieve precise positioning and mapping.

[0011] Preferably, S100 includes:

[0012] S110: using the CLAHE algorithm, the input image is divided into multiple small blocks, and the grayscale histogram of each small block is calculated;

[0013] S120: Equalizing the histogram of each small block. If the number of pixels at a certain gray level exceeds a preset threshold, the excess pixels will be evenly distributed to other gray levels.

[0014] S130: Using interpolation, for each pixel, calculate its weight at the center of four adjacent small blocks, and mix the equalization results of the four small blocks according to these weights, and finally output the enhanced image.

[0015] Preferably, S200 includes:

[0016] S210: Use the pre-trained YOLOv8 model to load weights, pass the enhanced image to the YOLOv8 model for target detection, and obtain a detection result; wherein the detection result includes the coordinates, category, and confidence of the detection box;

[0017] S220: According to the detection result, the objects in the image are divided into dynamic targets and static targets in advance, and the detection frame of the detection target is divided into a dynamic detection frame and a static detection frame.

[0018] Preferably, S300 includes:

[0019] S310: Preprocess the image passed by YOLOv8, resize it, load the pre-trained Xfeat model, and pass the pre-processed image to the Xfeat model for feature extraction; the Xfeat model includes a backbone network and independent convolution branches and fusion blocks;

[0020] S320: Divide the preprocessed image into multiple 8×8 small blocks, and perform a 1×1 convolution operation on each small block to extract a key point heat map K for identifying potential key point positions in the image. Each element of the key point heat map represents the probability of the corresponding position becoming a key point.

[0021] S330: When the feature points are within 2 pixels around the acquired dynamic target area coordinates, they are determined to be dynamic feature points; determine whether these dynamic feature points are in the static detection frame, if not in any static detection frame, mark them as feature points to be removed, change the coordinates of these feature points to (-1, -1), and remove them;

[0022] S340: Processing the original input image through the backbone network to extract local feature descriptors in the image; wherein the backbone network is a network that generates a compact 64-dimensional dense descriptor graph through a series of convolution operations, including six convolution blocks, each of which downsamples the input image;

[0023] S350: Integrate features at different levels or scales through the fusion block and output the results to obtain a compact 64-D dense descriptor map F and a reliability heat map R. The reliability heat map is used to subsequently evaluate the reliability of each key point, where each element represents the reliability score of the key point at the corresponding position;

[0024] S360: Combine the key point heat map K, the reliability heat map R and perform matching operations on the dense local feature map F, and then process the output of the matching operation through the refinement strategy to obtain the final matching result and pass it into the SLAM system.

[0025] Preferably, S360 is specifically:

[0026] The matching operation is performed on the dense local feature map F. The fusion block extracts the corresponding feature descriptors from the descriptor map F according to the key point positions extracted from the key point heat map K, performs feature point matching, and combines the reliability scores in the reliability heat map R to exclude unreliable key points and pay more attention to reliable key points during the matching process.

[0027] Through the refinement strategy, the output image after the matching operation is first upsampled to the resolution of the original image, and then RANSAC local optimization is performed. Finally, the local optimized output is weightedly fused with the reliability heat map to obtain the final matching result and pass it to the SLAM system.

[0028] Preferably, before S100, the method further includes:

[0029] Use the startup px4 file under ROS to establish a connection between the onboard computer and the imu of the flight controller, start the camera file through realsense, start the SLAM algorithm at the same time, and subscribe to the data of the imu and camera sensor in the SLAM system.

[0030] The present invention enables the underwater autonomous robot to overcome the harsh underwater optical environment, reduce the interference of dynamic targets, and achieve relatively accurate positioning of itself while ensuring real-time and robustness. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 This is a flow chart of an underwater SLAM method based on the combination of YOLOv8 and XFeat depth features in one embodiment of the present invention;

[0032] Figure 2 This is a flow chart of an underwater SLAM method based on the combination of YOLOv8 and XFeat depth features in another embodiment of the present invention;

[0033] Figure 3 is a diagram of the XFeat backbone network structure in one embodiment of the present invention;

[0034] Figure 4 It is a flowchart for removing feature points based on YOLOv8. DETAILED DESCRIPTION

[0035] In order to enable those skilled in the art to better understand the technical solution of the present invention, the present invention is further described in detail below in conjunction with the accompanying drawings.

[0036] In one embodiment, Figure 1 As shown, an underwater SLAM method based on combining YOLOv8 with XFeat depth features includes the following steps:

[0037] S100: Acquire an image captured by an underwater camera, and perform enhancement processing on the image to obtain an enhanced image;

[0038] S200: YOLOv8 is used to process the enhanced image, distinguish the objects in the image into dynamic targets and static targets, mark the detection frame coordinates of these targets, and remove unnecessary dynamic feature points;

[0039] S300: Use lightweight deep feature Xfeat to extract features from the image passed by YOLOv8, remove feature points within the target detection frame, perform feature matching on the retained feature points, pass the matching results to the SLAM system, and use the feature matching results to achieve precise positioning and mapping.

[0040] Specifically, the matching results are passed into the SLAM system, and the current position of the camera is estimated by minimizing the reprojection error by matching the current frame with the feature points in the previous frame or map. Based on the matching results, new map points are created and added to the map. At the same time, the system will also check and remove map points that have not been observed for a long time or that conflict with other map points, and use the results of feature matching to achieve precise positioning and mapping.

[0041] The present invention discloses an underwater autonomous robot SLAM method based on the combination of YOLOv8 and XFeat deep features, belonging to the field of underwater SLAM. The underwater SLAM method includes preliminary underwater image processing, YOLOv8 image detection, extraction and matching of XFeat features, and then the entire underwater SLAM system is completed under the ORB-SLAM3 framework. The invention enables the underwater autonomous robot to overcome the harsh optical environment underwater, reduce the interference of dynamic targets, and achieve more accurate positioning of itself, while ensuring real-time and robustness.

[0042] In one embodiment, S100 includes:

[0043] S110: using the CLAHE algorithm, the input image is divided into multiple small blocks, and the grayscale histogram of each small block is calculated;

[0044] S120: Equalizing the histogram of each small block. If the number of pixels at a certain gray level exceeds a preset threshold, the excess pixels will be evenly distributed to other gray levels.

[0045] S130: Using interpolation, for each pixel, calculate its weight at the center of four adjacent small blocks, and mix the equalization results of the four small blocks according to these weights, and finally output the enhanced image.

[0046] Specifically, the images captured by the underwater camera are simply processed, and the contrast of the images is adjusted using CLAHE to make the illumination of the images as uniform as possible.

[0047] In one embodiment, Figure 2 As shown, S200 includes:

[0048] S210: Use the pre-trained YOLOv8 model to load weights, pass the enhanced image to the YOLOv8 model for target detection, and obtain a detection result; wherein the detection result includes the coordinates, category, and confidence of the detection box;

[0049] S220: According to the detection result, the objects in the image are divided into dynamic targets and static targets in advance, and the detection frame of the detection target is divided into a dynamic detection frame and a static detection frame.

[0050] Specifically, YOLOv8 is used for processing to distinguish objects in the image and divide them into dynamic targets and static targets. For example, schools of fish and floating objects are classified as dynamic targets, while stones and shipwreck fragments are classified as static targets, and the detection frame coordinates of these targets are marked.

[0051] In one embodiment, Figure 3 and Figure 4 As shown, S300 includes:

[0052] S310: Preprocess the image passed by YOLOv8, resize it, load the pre-trained Xfeat model, and pass the pre-processed image to the Xfeat model for feature extraction; the Xfeat model includes a backbone network and independent convolution branches and fusion blocks;

[0053] S320: Divide the preprocessed image into multiple 8×8 small blocks, and perform a 1×1 convolution operation on each small block to extract a key point heat map K for identifying potential key point positions in the image. Each element of the key point heat map represents the probability of the corresponding position becoming a key point.

[0054] S330: When the feature points are within 2 pixels around the acquired dynamic target area coordinates, they are determined to be dynamic feature points; determine whether these dynamic feature points are in the static detection frame, if not in any static detection frame, mark them as feature points to be removed, change the coordinates of these feature points to (-1, -1), and remove them;

[0055] S340: Processing the original input image through the backbone network to extract local feature descriptors in the image; wherein the backbone network is a network that generates a compact 64-dimensional dense descriptor graph through a series of convolution operations, including six convolution blocks, each of which downsamples the input image;

[0056] S350: Integrate features at different levels or scales through the fusion block and output the results to obtain a compact 64-D dense descriptor map F and a reliability heat map R. The reliability heat map is used to subsequently evaluate the reliability of each key point, where each element represents the reliability score of the key point at the corresponding position;

[0057] S360: Combine the key point heat map K, the reliability heat map R and perform matching operations on the dense local feature map F, and then process the output of the matching operation through the refinement strategy to obtain the final matching result and pass it into the SLAM system.

[0058] Specifically, in S320, a convolution operation is performed on each small block, which can effectively extract the feature information within the small block while maintaining computational efficiency. Such a judgment condition is set in S330 because, when encountering feature points that are simultaneously in multiple detection frames, in order not to eliminate too much image information, as many valid points as possible are retained. In S340, each convolution block downsamples the input image to reduce the height and width of the image, thereby reducing the amount of calculation while retaining the main feature information, and increasing the number of channels through the convolution layer. In S350, features of different levels or scales are specifically shallow features and deep features, where shallow features contain detail information and deep features contain semantic information.

[0059] In one embodiment, S360 specifically includes:

[0060] The matching operation is performed on the dense local feature map F. The fusion block extracts the corresponding feature descriptors from the descriptor map F according to the key point positions extracted from the key point heat map K, performs feature point matching, and combines the reliability scores in the reliability heat map R to exclude unreliable key points and pay more attention to reliable key points during the matching process.

[0061] Through the refinement strategy, the output image after the matching operation is first upsampled to the resolution of the original image, and then RANSAC local optimization is performed. Finally, the local optimized output is weightedly fused with the reliability heat map to obtain the final matching result and pass it to the SLAM system.

[0062] In one embodiment, before S100, the method further includes:

[0063] Use the startup px4 file under ROS to establish a connection between the onboard computer and the imu of the flight controller, start the camera file through realsense, start the SLAM algorithm at the same time, and subscribe to the data of the imu and camera sensor in the SLAM system.

[0064] In the above-mentioned underwater SLAM method based on the combination of YOLOv8 and XFeat deep features, the preliminary underwater image processing step is to perform contrast-limited adaptive histogram equalization on the image extracted by the camera; the YOLOv8 image detection step is to perform target detection on the image after the preliminary histogram equalization, obtain the detection frames of fish (dynamic targets) and sea urchins (static targets) and remove dynamic feature points; XFeat features are based on a convolutional neural network (CNN) architecture, which uses a special strategy for feature extraction and matching, and then passes the results to the SLAM system to complete map construction, camera pose estimation and other functions;

[0065] The present invention has the following advantages:

[0066] 1. Most learning-based object detection often takes a lot of time and computing power, which makes the processor unable to achieve real-time processing. To address this problem, a screening strategy based on YOLOv8 is adopted on the onboard processor of the underwater robot. Under the premise of ensuring real-time performance, the interference of dynamic objects is eliminated as much as possible without consuming too much computing power.

[0067] 2. Traditional feature points are sensitive to changes in lighting and other factors in complex environments, which leads to unstable feature points and even loss of tracking. XFeat is used for feature extraction and matching to ensure high efficiency and high precision. It can be deployed on a microcomputer and is 5 times faster than traditional SuperPoint. In the case of lighting changes and large rotations, the accuracy is much higher than that of feature points such as ORB.

[0068] 3. It is realized that in complex underwater environments, the processor carried by the underwater robot can run in real time, perceive the environment, locate and build maps, use active light sources to detect the environment, and obtain image information with a binocular camera. Through the above methods, the feasibility of underwater SLAM is greatly enhanced, meeting the requirements of autonomous navigation of underwater robots.

[0069] The above is a detailed introduction to the underwater SLAM method based on the combination of YOLOv8 and XFeat depth features provided by the present invention. This article uses specific examples to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the core idea of ​​the present invention. It should be pointed out that for ordinary technicians in this technical field, without departing from the principles of the present invention, the present invention can also be improved and modified, and these improvements and modifications also fall within the scope of protection of the claims of the present invention.

Claims

1. An underwater SLAM method based on YOLOv8 and XFeat deep features, characterized in that: The method comprises the following steps: S100: Acquire an image captured by an underwater camera, and perform enhancement processing on the image to obtain an enhanced image; S200: YOLOv8 is used to process the enhanced image, distinguish the objects in the image into dynamic targets and static targets, and mark the detection frame coordinates of these targets; S300: Use lightweight deep feature Xfeat to extract features from the image passed by YOLOv8, remove feature points within the target detection frame, perform feature matching on the retained feature points, pass the matching results to the SLAM system, and use the feature matching results to achieve precise positioning and mapping.

2. The method according to claim 1, characterized in that S100 includes: S110: using the CLAHE algorithm, the input image is divided into multiple small blocks, and the grayscale histogram of each small block is calculated; S120: Equalizing the histogram of each small block. If the number of pixels at a certain gray level exceeds a preset threshold, the excess pixels will be evenly distributed to other gray levels. S130: Using interpolation, for each pixel, calculate its weight at the center of four adjacent small blocks, and mix the equalization results of the four small blocks according to these weights, and finally output the enhanced image.

3. The method according to claim 2, characterized in that S200 includes: S210: Use the pre-trained YOLOv8 model to load weights, pass the enhanced image to the YOLOv8 model for target detection, and obtain a detection result; wherein the detection result includes the coordinates, category, and confidence of the detection box; S220: According to the detection result, the objects in the image are divided into dynamic targets and static targets in advance, and the detection frame of the detection target is divided into a dynamic detection frame and a static detection frame.

4. The method according to claim 3, characterized in that S300 includes: S310: Preprocess the image passed by YOLOv8, resize it, load the pre-trained Xfeat model, and pass the pre-processed image to the Xfeat model for feature extraction; the Xfeat model includes a backbone network and independent convolution branches and fusion blocks; S320: Divide the preprocessed image into multiple 8×8 small blocks, and perform a 1×1 convolution operation on each small block to extract a key point heat map K for identifying potential key point positions in the image. Each element of the key point heat map represents the probability of the corresponding position becoming a key point. S330: When the feature points are within 2 pixels around the acquired dynamic target area coordinates, they are determined to be dynamic feature points; determine whether these dynamic feature points are in the static detection frame, if not in any static detection frame, mark them as feature points to be removed, change the coordinates of these feature points to (-1, -1), and remove them; S340: Processing the original input image through the backbone network to extract local feature descriptors in the image; wherein the backbone network is a network that generates a compact 64-dimensional dense descriptor graph through a series of convolution operations, including six convolution blocks, each of which downsamples the input image; S350: Integrate features at different levels or scales through the fusion block and output the results to obtain a compact 64-D dense descriptor map F and a reliability heat map R. The reliability heat map is used to subsequently evaluate the reliability of each key point, where each element represents the reliability score of the key point at the corresponding position; S360: Combine the key point heat map K, the reliability heat map R and perform matching operations on the dense local feature map F, and then process the output of the matching operation through the refinement strategy to obtain the final matching result and pass it into the SLAM system.

5. The method according to claim 4, characterized in that S360 is specifically: The matching operation is performed on the dense local feature map F. The fusion block extracts the corresponding feature descriptors from the descriptor map F according to the key point positions extracted from the key point heat map K, performs feature point matching, and combines the reliability scores in the reliability heat map R to exclude unreliable key points and pay more attention to reliable key points during the matching process. Through the refinement strategy, the output image after the matching operation is first upsampled to the resolution of the original image, and then RANSAC local optimization is performed. Finally, the local optimized output is weightedly fused with the reliability heat map to obtain the final matching result and pass it to the SLAM system.

6. The method according to claim 5, characterized in that Before S100, it also included: Use the startup px4 file under ROS to establish a connection between the onboard computer and the imu of the flight controller, start the camera file through realsense, start the SLAM algorithm at the same time, and subscribe to the data of the imu and camera sensor in the SLAM system.

Images