Underwater real-time target detection method based on visual SLAM

By applying the YOLOV5 object detection framework in the underwater vision SLAM system, combined with specific improved technology, the problem of low positioning and mapping accuracy of underwater vision SLAM in dynamic environments is solved, and more efficient object detection and autonomous recognition are achieved, improving The robustness and semantic information acquisition of underwater operations.

CN118918307AActive Publication Date: 2024-11-08JIANGSU OCEAN UNIV

Patent Information

Application Number
CN202410881739.1
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-03
Publication Date
2024-11-08
Estimated Expiration
2044-07-03

AI Technical Summary

Technical Problem

The underwater visual SLAM system has low positioning and mapping accuracy in dynamic environments, and cannot effectively identify and track targets, resulting in poor robustness of robots' autonomous operations and traditional methods cannot provide semantic information.

Method used

The YOLOV5 object detection framework is adopted, combined with the CBAM attention mechanism, variable convolution network and improved EIOU loss function, filtering, enhancing and feature point removal of underwater images is carried out to improve the accuracy and efficiency of object detection.

Benefits of technology

It achieves higher measurement accuracy and autonomous identification efficiency in dynamic underwater environments, improves the accuracy and system stability of underwater target detection, and provides support for underwater operations and resource exploration.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118918307A_ABST
    Figure CN118918307A_ABST
Patent Text Reader

Abstract

The invention discloses an underwater real-time target detection method based on visual SLAM, and the method comprises the steps: carrying out the filtering, initial dust removal and image enhancement of underwater collected image data, carrying out the detection of a target image frame of a dynamic scene through an improved deep learning target frame, transmitting the target image frame to a visual SLAM front end, and carrying out the elimination of dynamic feature points, analysis and verification are carried out through a precision evaluation tool, the precision of the visual SLAM underwater autonomous recognition system is relatively improved, an underwater target object can be detected in real time, and powerful support is provided for underwater operation exploration and underwater resource exploration.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The invention belongs to the technical field of target detection, and in particular relates to a real-time underwater target detection method based on visual SLAM. Background Art

[0002] Underwater visual SLAM is to complete the trajectory positioning of the camera and the construction of the surrounding environment by matching and tracking the features of the collected images in underwater scenes. It has great application potential in unmanned surveys, free cruising and other fields. There are huge differences between the underwater environment and the terrestrial environment. First of all, underwater, especially in deep water, mainly relies on artificial light sources for lighting, and the absorption and scattering of light by water leads to a limited lighting range and insufficient clarity. It is difficult to extract underwater image features and the feature matching is not high. Among them, target recognition and tracking are the key technologies for the autonomous operation of underwater mobile robot systems. In terms of underwater target recognition, the quality of underwater images has a great impact on advanced visual recognition tasks. At the same time, it plays an important role in the decision-making of underwater robots for motion planning such as obstacle avoidance and grasping of targets.

[0003] At present, when underwater mobile robots perform simultaneous localization and mapping (SLAM) in dynamic scenes, the positioning algorithm and mapping accuracy will be affected, thereby reducing the robustness of the system. The stability of the current visual SLAM system in a dynamic environment is greatly affected by the environment and has high requirements for system hardware and software, and it is impossible to take all of these into account at the same time. Given that the more mature open source SLAM systems are all established based on static environments, a large number of erroneous feature point matches will be generated in the presence of dynamic objects, which makes the SLAM system in the application conditions where there are dynamic objects. The accuracy of motion subject pose estimation and mapping accuracy cannot meet the actual application requirements. However, the environment with dynamic objects is more in line with people's actual application needs, so it is of practical significance to study SLAM based on the presence of dynamic objects.

[0004] Traditional simultaneous localization and mapping (SLAM) mostly uses low-level geometric features such as points, lines, and surfaces, which cannot provide semantic information. Semantic SLAM can provide semantic information for the environment, which helps the robot understand the environment at the geometric and content levels. Object detection is an important part of obtaining image semantic information, which can locate the target instance in the image. This method applies YOLO to the visual SLAM framework ORB_SLAM2, and uses the YOLOV5 network to separate dynamic and static objects in key frames, so that higher measurement accuracy and autonomous recognition efficiency can be achieved with higher efficiency. Summary of the invention

[0005] The purpose of the present invention is to design a real-time underwater target detection method based on visual SLAM, which achieves higher measurement accuracy and autonomous recognition efficiency.

[0006] In order to achieve the above object, the technical solution adopted by the present invention is as follows:

[0007] A real-time underwater target detection method based on visual SLAM.

[0008] The first step is underwater image acquisition

[0009] A high-resolution visual camera is used to collect continuous underwater raw image sequences;

[0010] Step 2: Underwater image filtering and dust removal

[0011] Image filtering and dust removal uses non-local mean filtering, where A(i,j) is the pixel value of point (i,j) in the neighborhood block of point A, and B(i,j) is the pixel value of the same position point in the neighborhood block of point B. The rows and columns of the neighborhood blocks are the same, assuming that they are all m rows and n columns, then the MSE (root mean square error) is calculated as follows:

[0012]

[0013] Where: A(i,j) is the pixel value of point (i,j) in the neighborhood block of point A, B(i,j) is the pixel value of the same position point in the neighborhood block of point B, the rows and columns of the neighborhood blocks are the same, m represents the number of image rows, and n represents the number of image columns; i and j are the indexes of the rows and columns of A and B; i represents the row index, and j represents the column index;

[0014] The filter value of point A is obtained by weighted average of the pixel values ​​of all points in the search window, w(A,B) is the Gaussian weight of point A and any point B in the search window; I(B) represents the image pixel value or intensity at position B; h represents the filter parameter, which controls the degree of smoothing; e represents the base of the natural logarithm of the mathematical constant; NLmeans(A) = 2w(A,B)*I(B) is calculated from the MSE similarity of the neighborhood blocks of the two points.

[0015]

[0016] The third step is underwater image feature enhancement

[0017] The image enhancement algorithm uses histogram equalization. For a grayscale image G of size , the grayscale value of the pixel corresponding to any position is (referring to the total number of grayscale levels in the image), and the probability of occurrence of different grayscale levels can be expressed as:

[0018]

[0019] Where: Pi represents the probability of grayscale i, that is, the proportion of pixels with grayscale value i in the image to the total pixels; x and y represent the pixel coordinates in the image, x represents the horizontal coordinate of the pixel, and y represents the vertical coordinate of the pixel; g(x,y) represents the pixel value at position (x,y); i represents the grayscale level of the image;

[0020] Therefore, for the grayscale image G, the method of image enhancement using histogram equalization can be described as:

[0021]

[0022] f(i) represents the gray value of gray level i after equalization; L represents the number of gray levels in the image; P k Represents the cumulative distribution function value of gray level K, that is, the sum of the probabilities of gray level k and all the following gray levels;

[0023] Here, the function represents the mapping relationship between the grayscale of the image before enhancement and the grayscale after enhancement. Generally speaking, the grayscale mapping expression commonly used in histogram equalization is:

[0024]

[0025] Step 4: Improvement of YOLOv5 target detection framework

[0026] The F-CBAM attention mechanism improved by CBAM is added to the YOLOv5 baseline model, and an adaptive learning mechanism is introduced to more accurately adjust the feature weights, thereby optimizing the efficiency of feature fusion and enabling the model to focus more on the key features in the image. The ordinary convolution is replaced by the variable convolutional network AKConv. This convolution method enhances the network's adaptability to different target shapes and sizes, increases the model's adaptability to different scenarios, and thus improves generalization capabilities. Finally, the improved EIOU loss function H-EIOU is introduced to more efficiently measure the angular difference between the predicted box and the real box, thereby helping the model to more accurately locate targets with obvious directions.

[0027] Step 5: Autonomous recognition of underwater image target detection

[0028] Using the improved deep learning target detection framework YOLOv5, we first calculate the prior frame of the underwater RGB-D image frame, then make predictions based on the prior frame during network training, output the predicted frame, compare it with the label frame, and finally perform gradient back propagation. Finally, we can get the RGB-D image frame with the target detection frame and the target position coordinate file in the image frame corresponding to the output detection target frame.

[0029] Step 6: Eliminate underwater image target feature points

[0030] The tracking thread in the visual SLAM framework passes the target position coordinate file in the RGB-D image frame, removes the feature points in the dynamic target frame in the RGB-D image frame, and generates a camera pose trajectory file.

[0031] Step 7: Verify the pose accuracy of underwater real-time and efficient target detection method

[0032] The pose estimation experiment used an accuracy evaluation tool and a visual SLAM system to estimate the camera pose. At the same time, a comparative test was performed between the real pose and the camera pose estimated by the system.

[0033] The following beneficial effects can be obtained through the above technical solution:

[0034] This method applies YOLO to the visual SLAM framework ORB_SLAM2, and uses the YOLOV5 network to separate dynamic and static objects in key frames, which can achieve higher measurement accuracy and autonomous recognition efficiency with higher efficiency.

[0035] Optimization and improvement are made on the dynamic visual SLAM system, and the deep learning target detection system YOLOv5 is used to eliminate and optimize dynamic feature points in dynamic environments, thereby relatively reducing the error and improving the positioning accuracy. Under certain conditions, the system operation time and efficiency are also relatively improved. It can detect underwater target objects in real time, providing strong support for underwater operation exploration and underwater resource survey. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 Comparison chart of trajectory, coordinates and angles of the method pose before improvement, the method pose after improvement and the true pose;

[0037] Figure 2 Improve the absolute trajectory comparison between the previous method and the true pose, and the absolute trajectory error numerical distribution diagram;

[0038] Figure 3 Comparison of absolute trajectories between the improved method and the true pose, and numerical distribution of absolute trajectory errors. DETAILED DESCRIPTION

[0039] The present invention will be further described below in conjunction with the accompanying drawings:

[0040] The present invention discloses a real-time underwater target detection method based on visual SLAM. Firstly, the image data collected underwater is filtered to remove dust and perform image enhancement. Then, the target image frame of the dynamic scene is detected through a deep learning target framework. Then, the frame is transmitted to the visual SLAM front end to remove dynamic feature points. The method is analyzed and verified through an accuracy evaluation tool. The accuracy of the visual SLAM underwater autonomous recognition system is relatively improved, and underwater target objects can be detected in real time, which provides strong support for underwater operation exploration and underwater resource survey.

[0041] The above-mentioned underwater autonomous identification method based on visual SLAM includes the following steps:

[0042] The first step is underwater image acquisition

[0043] A continuous sequence of underwater raw images is collected by a high-resolution visual camera.

[0044] Step 2: Underwater image filtering and dust removal

[0045] Image filtering and dust removal uses non-local mean filtering, where A(i,j) is the pixel value of point (i,j) in the neighborhood block of point A, and B(i,j) is the pixel value of the same position point in the neighborhood block of point B. The rows and columns of the neighborhood blocks are the same, assuming that they are all m rows and n columns, then the MSE (root mean square error) is calculated as follows:

[0046]

[0047] Where: A(i,j) is the pixel value of point (i,j) in the neighborhood block of point A, B(i,j) is the pixel value of the same position point in the neighborhood block of point B, the rows and columns of the neighborhood blocks are the same, m represents the number of image rows, n represents the number of image columns; i and j are the indexes of the rows and columns of A and B; i represents the row index, j represents the column index

[0048] The filter value of point A is obtained by weighted average of the pixel values ​​of all points in the search window, w(A, B) is the Gaussian weight of point A and any point B in the search window; I(B) represents the image pixel value or intensity at position B; h represents the filter parameter, which controls the degree of smoothing; e represents the base of the natural logarithm of the mathematical constant; and it is calculated by the MSE similarity of the neighborhood blocks of the two points:

[0049] NLmeans(A)=∑w(A,B))*I(B)

[0050]

[0051] The third step is underwater image feature enhancement

[0052] The image enhancement algorithm uses histogram equalization. For a grayscale image G of size , the grayscale value of the pixel corresponding to any position is (referring to the total number of grayscale levels in the image), and the probability of occurrence of different grayscale levels can be expressed as:

[0053]

[0054] Where: Pi represents the probability of grayscale i, that is, the proportion of pixels with grayscale value i in the image to the total pixels; x and y represent the pixel coordinates in the image, x represents the horizontal coordinate of the pixel, and y represents the vertical coordinate of the pixel; g(x,y) represents the pixel value at position (x,y); i represents the grayscale level of the image;

[0055] Therefore, for the grayscale image G, the method of image enhancement using histogram equalization can be described as:

[0056]

[0057] f(i) represents the gray value of gray level i after equalization; L represents the number of gray levels in the image; P k represents the cumulative distribution function value of gray level K, that is, the sum of the probabilities of gray level k and all gray levels below it; here, the function represents the mapping relationship between the gray level of the image before enhancement and the gray level after enhancement. Generally speaking, the gray level mapping expression commonly used in histogram equalization is:

[0058]

[0059] Step 4: Improvement of YOLOv5 target detection framework

[0060] The F-CBAM attention mechanism improved by CBAM is added to the YOLOv5 baseline model, and an adaptive learning mechanism is introduced to more accurately adjust the feature weights, thereby optimizing the efficiency of feature fusion and enabling the model to focus more on the key features in the image. The ordinary convolution is replaced by the variable convolutional network AKConv. This convolution method enhances the network's adaptability to different target shapes and sizes, increases the model's adaptability to different scenarios, and thus improves generalization capabilities. Finally, the improved EIOU loss function H-EIOU is introduced to more efficiently measure the angular difference between the predicted box and the real box, thereby helping the model to more accurately locate targets with obvious directions.

[0061] Step 5: Autonomous recognition of underwater image target detection

[0062] Using the improved deep learning target detection framework YOLOv5, we first calculate the prior frame of the underwater RGB-D image frame, then make predictions based on the prior frame during network training, output the predicted frame, compare it with the label frame, and finally perform gradient back propagation. Finally, we can get the RGB-D image frame with the target detection frame and the target position coordinate file in the image frame corresponding to the output detection target frame.

[0063] Step 6: Eliminate underwater image target feature points

[0064] The tracking thread in the visual SLAM framework passes the target position coordinate file in the RGB-D image frame, removes the feature points in the dynamic target frame in the RGB-D image frame, and generates a camera pose trajectory file.

[0065] Step 7: Verify the pose accuracy of underwater real-time and efficient target detection method

[0066] The pose estimation experiment used an accuracy evaluation tool and a visual SLAM system to estimate the camera pose. At the same time, the real pose and the camera pose estimated by the system were compared and measured.

[0067] Embodiment: A continuous underwater original image sequence is collected by a high-resolution visual camera, and a non-mean filtering underwater image filtering dust removal and a histogram equalization image enhancement algorithm are used. The target position coordinate file in the corresponding image frame is output through improved underwater image target detection and autonomous recognition. The file is passed to the tracking thread of the visual SLAM framework, and the feature points in the dynamic target frame in the image frame are removed. Finally, the pose accuracy is verified.

[0068] Figure 1 The figure shows the comparison of the trajectory, coordinates and angle of the method pose before improvement, the method pose after improvement and the real pose. Figure 1 It can be seen that the pose of the method before the improvement is poorly fitted to the real pose in terms of trajectory, coordinates and angles due to the comprehensive influence of the dynamic feature points generated by the dynamic object. The pose of the method after the improvement is better fitted to the real pose in terms of trajectory, coordinates and angles.

[0069] Use the trajectory accuracy evaluation tool to compare the pose trajectory of the method before improvement and the pose trajectory of the method after improvement with the actual pose trajectory, and align the two through the timestamps of the two camera poses. Then calculate the difference between the two camera poses to generate the absolute trajectory comparison graph and absolute trajectory error value distribution graph of the pose of the method before improvement and the actual pose, as well as the absolute trajectory comparison graph and absolute trajectory error value distribution graph of the pose of the method after improvement and the actual pose, such as Figure 2 , Figure 3 As shown by Figure 2 In (a), we can see that the absolute trajectory of the pose of the improved method is poorly fitted to the true pose trajectory; Figure 2 (b) shows that due to the error caused by the feature points generated by the dynamic object, the absolute trajectory error value distribution diagram cannot show a normal distribution. Figure 3 In (a), we can see that the absolute trajectory of the improved method fits the real trajectory better. Figure 3 In (b), it can be seen that the feature points generated by the dynamic objects are eliminated, and the errors caused by the dynamic feature points are removed, so that the numerical distribution diagram of the absolute trajectory error can present a normal distribution display.

[0070] This method optimizes and improves the dynamic visual SLAM system, and uses the deep learning target detection system YOLOv5 to eliminate and optimize dynamic feature points in dynamic environments, thereby relatively reducing the error and improving the positioning accuracy, and also relatively improving the system operation time and efficiency under certain conditions.

[0071] The above are all preferred embodiments of the present invention. For ordinary technicians in this technical field, without departing from the principle of the present invention, various equivalent modifications to the present invention belong to the protection scope of the claims attached to this application.

Claims

1. A real-time underwater target detection method based on visual SLAM, characterized by: The method is as follows: S1: Underwater image acquisition, collecting continuous underwater raw image sequences through a high-resolution visual camera; S2: filtering and dust removal of the image obtained in S1, the image filtering and dust removal adopts non-local mean filtering; S3: Enhance the image features in S2. The image enhancement algorithm uses histogram equalization. S4: Improve the YOLOv5 target detection framework, add the F-CBAM attention mechanism improved by CBAM to the YOLOv5 baseline model, introduce an adaptive learning mechanism, adjust feature weights, optimize the efficiency of feature fusion, and make the model focus on the key features in the image; enhance the adaptability of the network to different target shapes and sizes through the variable convolution network AKConv, increase the adaptability of the model to different scenes, and finally introduce the improved EIOU loss function H-EIOU to measure the angle difference between the predicted box and the real box, and locate targets with obvious directionality; S5: Autonomous recognition of underwater image target detection. The improved deep learning target detection framework YOLOv5 is used to first calculate the prior frame of the underwater RGB-D image frame. Then, during network training, prediction is made based on the prior frame. The predicted frame is then output and compared with the label frame. Finally, gradient back propagation is performed. Finally, an RGB-D image frame with a target detection frame and a target position coordinate file in the image frame corresponding to the output detection target frame can be obtained. S6: Underwater image target feature point removal: the tracking thread in the visual SLAM framework passes the target position coordinate file in the RGB-D image frame, removes the feature points in the dynamic target frame in the RGB-D image frame, and generates a camera pose trajectory file; S7: Pose accuracy verification, using the visual SLAM system to estimate the camera pose, and using the real pose and the camera pose estimated by the system for comparative testing.

2. The method for underwater real-time target detection based on visual SLAM according to claim 1, characterized in that: The non-local mean filtering in S2 is specifically processed by the following method: The root mean square error MSE calculation formula is as follows: Where: A(i,j) is the pixel value of point (i,j) in the neighborhood block of point A, B(i,j) is the pixel value of the same position point in the neighborhood block of point B, the rows and columns of the neighborhood blocks are the same, m represents the number of image rows, and n represents the number of image columns; i and j are the indexes of the rows and columns of A and B; i represents the row index, and j represents the column index; The filter value of point A is obtained by weighted average of the pixel values ​​of all points in the search window, w(A,B) is the Gaussian weight of point A and any point B in the search window; I(B) represents the image pixel value or intensity at position B; h represents the filter parameter, which controls the degree of smoothing; e represents the base of the natural logarithm of the mathematical constant; and it is calculated by the MSE similarity of the neighborhood blocks of the two points. NLmeans(A)=∑w(A,B))*I(B) 。 3. The method for underwater real-time target detection based on visual SLAM according to claim 1, characterized in that: In S3, for the grayscale image G of size, the total number of grayscale levels of the image corresponding to the grayscale value of the pixel at any position, the probability of occurrence of different grayscale levels can be expressed as: Where: Pi represents the probability of grayscale i, that is, the proportion of pixels with grayscale value i in the image to the total pixels; x and y represent the pixel coordinates in the image, x represents the horizontal coordinate of the pixel, and y represents the vertical coordinate of the pixel; g(x,y) represents the pixel value at position (x,y); i represents the grayscale level of the image; f(i) represents the gray value of gray level i after equalization; L represents the number of gray levels in the image; Pk represents the cumulative distribution function value of gray level K, that is, the sum of the probabilities of gray level k and all gray levels below it; for gray image G, the method of image enhancement using histogram equalization can be described as: The function represents the mapping relationship between the grayscale of the image before enhancement and the grayscale after enhancement. The commonly used grayscale mapping expression for histogram equalization is: 。 4. The method for underwater real-time target detection based on visual SLAM according to claim 1, characterized in that: The pose includes trajectory, coordinates, and angle.

5. The method for underwater real-time target detection based on visual SLAM according to claim 1, characterized in that: In S7, the trajectory accuracy evaluation tool is used to compare the pose trajectory of the method before improvement and the pose trajectory of the method after improvement with the actual pose trajectory respectively, align the two camera poses by the timestamps of the two camera poses, and then calculate the difference between the two camera poses to generate the absolute trajectory comparison diagram and the absolute trajectory error value distribution diagram of the pose of the method before improvement and the actual pose, as well as the absolute trajectory comparison diagram and the absolute trajectory error value distribution diagram of the pose of the method after improvement and the actual pose.

Citation Information

Patent Citations

  • Semantic vision SLAM positioning method based on target detection in indoor dynamic scene

    CN114677323A

  • Visual SLAM optimization method in dynamic scene based on deep learning and GPU acceleration

    CN118097265A

  • Method and system for real-time target tracking based on deep learning

    US20200065976A1

  • Visual slam method applicable to indoor dynamic environment

    WO2022041596A1

Cited By

  • Underwater SLAM (Simultaneous Localization and Mapping) method based on combination of YOLOv8 and XFat depth features

    CN119533475A

  • Dynamic feature point elimination method based on normal distribution

    CN120451261A