An underwater real-time target detection method based on visual slam

Abstract

The application discloses a kind of underwater real-time target detection methods based on visual SLAM, first filtering image data collected underwater is filtered and dusted initially and carries out image enhancement, then the target image frame of dynamic scene is detected by improved deep learning target framework, then it is transmitted to visual SLAM front end and is removed to dynamic feature point, analysis and verification are carried out by precision evaluation tool, the precision of visual SLAM underwater autonomous identification system has relatively promotion, and real-time detection can be carried out to underwater target object in real time, and provides strong support for underwater operation exploration and underwater resource exploration.

Description

Technical Field

[0001] This invention belongs to the field of target detection technology, specifically relating to a real-time underwater target detection method based on visual SLAM. Background Technology

[0002] Underwater visual SLAM, in underwater scenarios, involves feature matching and tracking of acquired images to locate camera trajectories and construct the surrounding environment. It holds immense application potential in unmanned surveying and free-roaming applications. The underwater environment differs significantly from the terrestrial environment. Firstly, underwater, especially in deep water, illumination relies primarily on artificial light sources. Furthermore, water's absorption and scattering of light results in limited illumination range and insufficient clarity, making feature extraction from underwater images difficult and leading to low feature matching accuracy. Target recognition and tracking are crucial technologies for autonomous operation of underwater mobile robot systems. In underwater target recognition, the quality of underwater images significantly impacts advanced visual recognition tasks and plays a vital role in the underwater robot's motion planning, including obstacle avoidance and grasping.

[0003] Currently, when underwater mobile robots perform Simultaneous Localization and Mapping (SLAM) in dynamic scenes, the localization algorithm and mapping accuracy are affected, thus reducing the robustness of the system. Current visual SLAM systems are highly susceptible to environmental influences in dynamic environments and have high hardware and software requirements, making it impossible to simultaneously address these factors. Since mature open-source SLAM systems are built on static environments, the presence of dynamic objects leads to numerous erroneous feature point matching errors. This results in SLAM systems failing to meet practical application requirements for pose estimation and mapping accuracy in applications with dynamic objects. However, environments with dynamic objects better meet real-world application needs; therefore, research on SLAM based on dynamic objects is of practical significance.

[0004] Traditional Simultaneous Localization and Mapping (SLAM) methods primarily utilize low-level geometric features such as points, lines, and polygons, failing to provide semantic information. Semantic SLAM, on the other hand, provides semantic information about the environment, enabling robots to understand their surroundings at both the geometric and content levels. Object detection is a crucial component in acquiring semantic information from images, allowing the localization of target instances. This method applies YOLO to the visual SLAM framework ORB_SLAM2, leveraging the YOLOv5 network to separate dynamic and static objects in keyframes, thereby achieving higher measurement accuracy and more efficient autonomous object recognition. Summary of the Invention

[0005] The purpose of this invention is to design a real-time underwater target detection method based on visual SLAM, which achieves higher measurement accuracy and autonomous recognition efficiency.

[0006] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0007] A real-time underwater target detection method based on visual SLAM.

[0008] Step 1: Underwater Image Acquisition

[0009] Acquire a continuous sequence of raw underwater images using a high-resolution visual camera;

[0010] The second step is underwater image filtering and dust removal.

[0011] Image filtering and dust removal employs nonlocal mean filtering, where A(i,j) is the pixel value of point (i,j) in the neighborhood block of point A, and B(i,j) is the pixel value of points at the same location in the neighborhood block of point B. The neighborhood blocks have the same number of rows and columns, assuming each is m rows and n columns. The MSE (Root Mean Square Error) is calculated as follows:

[0012]

[0013] In the formula: A(i,j) is the pixel value of point (i,j) in the neighborhood block of point A, B(i,j) is the pixel value of the point at the same position in the neighborhood block of point B, the neighborhood blocks have the same number of rows and columns, m represents the number of rows of the image, n represents the number of columns and rows of the image; i and j are the row and column indices of A and B, respectively; i represents the row index and j represents the column index.

[0014] The filtered value for point A is obtained by weighted averaging of the pixel values ​​of all points within the search window, where w(A,B) is the Gaussian weight between point A and any point B in the search window; I(B) represents the image pixel value or intensity at position B; h represents the filtering parameter, controlling the degree of smoothing; e represents the base of the natural logarithm; and NLmeans(A) is calculated from the MSE similarity of the neighborhood blocks of the two points, where NLmeans(A) = 2w(A,B)*I(B).

[0015]

[0016] Step 3: Underwater Image Feature Enhancement

[0017] Image enhancement algorithms employ histogram equalization. For a grayscale image G of size , where the grayscale value of a pixel at any location is (referring to the total number of grayscale levels in the image), the probability of different grayscale levels occurring can be expressed as:

[0018]

[0019] In the formula: Pi g(x,y) represents the probability of gray level i, which is the proportion of pixels in the image that can only have gray level i; x and y represent the pixel coordinates in the image, where x is the horizontal coordinate of the pixel and y is the vertical coordinate of the pixel; g(x,y) represents the pixel value at position (x,y); and i represents the gray level of the image.

[0020] Therefore, the method of image enhancement using histogram equalization for a grayscale image G can be described as follows:

[0021]

[0022] f(i) represents the gray value of gray level i after equalization; L represents the number of gray levels in the image; P k This represents the cumulative distribution function value of gray level K, which is the sum of the probabilities of gray level k and all gray levels below it.

[0023] Here, the function represents the mapping relationship between the gray levels of the image before enhancement and the gray levels after enhancement. Generally, the commonly used gray level mapping expression for histogram equalization is:

[0024]

[0025] Step 4: Improvements to the YOLOv5 object detection framework

[0026] The YOLOv5 baseline model is improved by adding an F-CBAM attention mechanism, which introduces an adaptive learning mechanism. This allows for more precise adjustment of feature weights, thereby optimizing the efficiency of feature fusion and enabling the model to focus more on key features in the image. A variable convolutional network, AKConv, replaces ordinary convolutions. This convolutional method enhances the network's adaptability to different target shapes and sizes, increasing the model's adaptability to different scenes and thus improving generalization ability. Finally, an improved EIOU loss function, H-EIOU, is introduced, which can more efficiently measure the angular difference between the predicted bounding box and the ground truth bounding box, helping the model to more accurately locate directional targets.

[0027] Step 5: Autonomous Target Detection and Recognition in Underwater Images

[0028] The improved deep learning object detection framework YOLOv5 is used to first calculate the prior bounding boxes of underwater RGB-D image frames. Then, during network training, predictions are made based on the prior bounding boxes, and the predicted bounding boxes are output. These are then compared with the label bounding boxes, and finally, gradient backpropagation is performed. Ultimately, RGB-D image frames with object detection bounding boxes are obtained, along with the target location coordinates file in the corresponding image frame.

[0029] Step 6: Underwater Image Target Feature Point Removal

[0030] In the visual SLAM framework, the tracking thread takes in the target position coordinate file from the RGB-D image frame, removes feature points from the dynamic target bounding box in the RGB-D image frame, and generates a camera pose trajectory file.

[0031] Step 7: Verification of pose accuracy of the underwater real-time and efficient target detection method

[0032] The pose estimation experiment employed an accuracy evaluation tool and used a visual SLAM system to estimate the camera pose. The actual pose and the camera pose estimated by the system were then compared and tested.

[0033] The above technical solution can achieve the following beneficial effects:

[0034] This method applies YOLO to the visual SLAM framework ORB_SLAM2, and uses the YOLOv5 network to separate dynamic and static objects in keyframes, thereby achieving higher measurement accuracy and autonomous recognition efficiency with greater efficiency.

[0035] The dynamic visual SLAM system was optimized and improved by using the YOLOv5 deep learning target detection system to eliminate dynamic feature points in dynamic environments. This reduced the error and improved the positioning accuracy. Under certain conditions, it also improved the system's running time and efficiency, enabling real-time detection of underwater targets and providing strong support for underwater exploration and resource surveys. Attached Figure Description

[0036] Figure 1 Comparison of the trajectory, coordinates, and angles of the method pose before improvement, the method pose after improvement, and the true pose;

[0037] Figure 2 A comparison of the absolute trajectories of the previous method pose and the true pose, and a numerical distribution diagram of the absolute trajectory error;

[0038] Figure 3 The improved method pose and the true pose are compared in terms of absolute trajectory, and the absolute trajectory error distribution is shown in the figure. Detailed Implementation

[0039] The invention will be further described below with reference to the accompanying drawings:

[0040] This invention discloses a real-time underwater target detection method based on visual SLAM. First, underwater image data is filtered for initial dust removal and then enhanced. Next, target image frames of dynamic scenes are detected using a deep learning target framework. Then, the data is fed into the visual SLAM front-end for dynamic feature point removal. The results are analyzed and verified using an accuracy evaluation tool. This method improves the accuracy of the visual SLAM underwater autonomous recognition system and enables real-time detection of underwater targets, providing strong support for underwater exploration and underwater resource surveys.

[0041] The above-mentioned underwater autonomous identification method based on visual SLAM includes the following steps:

[0042] Step 1: Underwater Image Acquisition

[0043] A continuous sequence of raw underwater images was acquired using a high-resolution visual camera.

[0044] The second step is underwater image filtering and dust removal.

[0045] Image filtering and dust removal employs nonlocal mean filtering, where A(i,j) is the pixel value of point (i,j) in the neighborhood block of point A, and B(i,j) is the pixel value of points at the same location in the neighborhood block of point B. The neighborhood blocks have the same number of rows and columns, assuming each is m rows and n columns. The MSE (Root Mean Square Error) is calculated as follows:

[0046]

[0047] In the formula: A(i,j) is the pixel value of point (i,j) in the neighborhood block of point A, B(i,j) is the pixel value of the point at the same position in the neighborhood block of point B, the neighborhood blocks have the same number of rows and columns, m represents the number of rows in the image, and n represents the number of columns and rows in the image; i and j are the row and column indices of A and B, respectively; i represents the row index and j represents the column index.

[0048] The filtered value for point A is obtained by weighted averaging of the pixel values ​​of all points within the search window; w(A,B) is the Gaussian weight between point A and any point B within the search window; I(B) represents the image pixel value or intensity at position B; h represents the filtering parameter, controlling the degree of smoothing; e represents the base of the natural logarithm; and the value is calculated from the MSE similarity of the neighborhood blocks of the two points.

[0049] NLmeans(A)=∑w(A,B))*I(B)

[0050]

[0051] Step 3: Underwater Image Feature Enhancement

[0052] Image enhancement algorithms employ histogram equalization. For a grayscale image G of size , where the grayscale value of a pixel at any location is (referring to the total number of grayscale levels in the image), the probability of different grayscale levels occurring can be expressed as:

[0053]

[0054] In the formula: Pi represents the probability of gray level i, that is, the proportion of pixels in the image that can only be gray level i; x and y represent the pixel coordinates in the image, x represents the horizontal coordinate of the pixel, and y represents the vertical coordinate of the pixel; g(x,y) represents the pixel value at position (x,y); i represents the gray level of the image.

[0055] Therefore, the method of image enhancement using histogram equalization for a grayscale image G can be described as follows:

[0056]

[0057] f(i) represents the gray value of gray level i after equalization; L represents the number of gray levels in the image; P k Let represent the cumulative distribution function value of gray level K, which is the sum of the probabilities of gray level k and all gray levels below it; here, the function represents the mapping relationship between the gray levels of the image before enhancement and the gray levels after enhancement. Generally speaking, the commonly used gray level mapping expression for histogram equalization is:

[0058]

[0059] Step 4: Improvements to the YOLOv5 object detection framework

[0060] The YOLOv5 baseline model is improved by adding an F-CBAM attention mechanism, which introduces an adaptive learning mechanism. This allows for more precise adjustment of feature weights, thereby optimizing the efficiency of feature fusion and enabling the model to focus more on key features in the image. A variable convolutional network, AKConv, replaces ordinary convolutions. This convolutional method enhances the network's adaptability to different target shapes and sizes, increasing the model's adaptability to different scenes and thus improving generalization ability. Finally, an improved EIOU loss function, H-EIOU, is introduced, which can more efficiently measure the angular difference between the predicted bounding box and the ground truth bounding box, helping the model to more accurately locate directional targets.

[0061] Step 5: Autonomous Target Detection and Recognition in Underwater Images

[0062] The improved deep learning object detection framework YOLOv5 is used to first calculate the prior bounding boxes of underwater RGB-D image frames. Then, during network training, predictions are made based on the prior bounding boxes, and the predicted bounding boxes are output. These are then compared with the label bounding boxes, and finally, gradient backpropagation is performed. Ultimately, RGB-D image frames with object detection bounding boxes are obtained, along with the target location coordinates file in the corresponding image frame.

[0063] Step 6: Underwater Image Target Feature Point Removal

[0064] In the visual SLAM framework, the tracking thread takes in the target position coordinate file from the RGB-D image frame, removes feature points from the dynamic target bounding box in the RGB-D image frame, and generates a camera pose trajectory file.

[0065] Step 7: Verification of pose accuracy of the underwater real-time and efficient target detection method

[0066] The pose estimation experiment employed an accuracy evaluation tool and used a visual SLAM system to estimate the camera pose. The actual pose and the camera pose estimated by the system were then compared.

[0067] Example: A continuous sequence of raw underwater images is acquired by a high-resolution visual camera. An underwater image filtering and dust removal algorithm with non-mean filtering and histogram equalization is used for image enhancement. An improved underwater image target detection and autonomous recognition algorithm is used to output the target position coordinate file in the corresponding image frame. The file is then fed into the tracking thread of the visual SLAM framework to remove feature points in the dynamic target bounding boxes in the image frames. Finally, the pose accuracy is verified.

[0068] Figure 1 The image shows a comparison of the pose of the method before improvement, the pose of the method after improvement, and the true pose with respect to trajectory, coordinates, and angle. Figure 1 As can be seen, the pose of the previous method, due to the combined effects of dynamic feature points generated by dynamic objects, resulted in a poor fit to the true pose in terms of trajectory, coordinates, and angle. The improved method, however, provides a better fit to the true pose in terms of trajectory, coordinates, and angle.

[0069] Using a trajectory accuracy evaluation tool, the pose trajectories of both the improved and unimproved methods were compared with the true pose trajectories, aligning them using the timestamps of the two camera poses. The differences between the two camera poses were then calculated, generating absolute trajectory comparison maps and absolute trajectory error distribution maps for both the unimproved and improved methods. Figure 2 , Figure 3 As shown, by Figure 2 As can be seen in (a) above, the previous method had a poor fit between the absolute pose trajectory and the true pose trajectory. Figure 2 As can be seen in (b), the error caused by the feature points generated by the dynamic object prevents the absolute trajectory error numerical distribution map from displaying a normal distribution; Figure 3 As can be seen from (a) in the improved method, the absolute pose trajectory fits the real pose trajectory better. Figure 3 As can be seen from (b) in the figure, the error caused by the feature points generated by the dynamic object is removed by eliminating the feature points, so that the absolute trajectory error numerical distribution map can be displayed in a normal distribution.

[0070] This method is optimized and improved on a dynamic visual SLAM system. The deep learning object detection system YOLOv5 is used to eliminate and optimize dynamic feature points in dynamic environments, thereby reducing the error and improving the localization accuracy. Under certain conditions, it also improves the system's running time and efficiency.

[0071] The above descriptions are all preferred embodiments of the present invention. For those skilled in the art, any modifications to the present invention in various equivalent forms without departing from the principle of the present invention shall fall within the protection scope of the appended claims.

Claims

1.A method for underwater real-time target detection based on visual SLAM, characterized in that: The method is as follows: S1: underwater image acquisition, a continuous sequence of underwater original images is acquired by a high-resolution visual camera; S2: filtering and dust removal of the images obtained in S1, the image filtering and dust removal adopts non-local mean filtering; S3: image feature enhancement in S2, the image enhancement algorithm adopts histogram equalization; S4: improving the YOLOv5 target detection framework, adding the F-CBAM attention mechanism improved by CBAM to the benchmark model of YOLOv5, introducing an adaptive learning mechanism, adjusting the feature weight, optimizing the efficiency of feature fusion, making the model focus on the key features in the image; through the variable convolution network AKConv, the adaptability of the network to different target shapes and sizes is enhanced, the adaptability of the model to different scenes is increased, and finally the improved EIOU loss function H-EIOU is introduced to measure the angle difference between the predicted box and the real box, and to locate the target with obvious directionality; S5: underwater image target detection autonomous identification, using the improved deep learning target detection framework YOLOv5, first calculating the prior box of the underwater collected RGB-D image frame, then predicting on the basis of the prior box during network training, then outputting the predicted box, and then comparing with the label box, finally performing gradient back propagation, and finally obtaining the RGB-D image frame with the target detection box and the target position coordinate file in the output detection target box corresponding image frame; S6: underwater image target feature point removal, the target position coordinate file in the RGB-D image frame is transmitted to the tracking thread in the visual SLAM framework, and the feature points in the dynamic target box in the RGB-D image frame are removed, thereby generating a camera pose trajectory file; S7: pose accuracy verification, the camera pose is estimated by using the visual SLAM system, and the real pose and the estimated camera pose of the system are compared and tested; The non-local mean filtering in S2 is processed by the following method: the mean square error MSE calculation formula is as follows: ； In the formula: A(i,j) is the pixel value of point A in the neighborhood block, B(i,j) is the pixel value of the same position point in the neighborhood block, the row and column of the neighborhood block are the same, m represents the number of image rows, n represents the number of image columns; i and j are the row and column indexes of A and B; i represents the row index, and j represents the column index; The filtered value of point A is obtained by weighted average of the pixel values of all points in the search window, w(A,B) is the Gaussian weight of point A and any point B in the search window; I(B) represents the image pixel value or intensity at position B; h represents the filtering parameter, which controls the degree of smoothing; e represents the base of the natural logarithm; it is calculated by the MSE similarity of the neighborhood blocks of the two points; ； In S3, for a gray image G with a size of, the probability of occurrence of different gray levels is represented as: ； where: P i represents the probability of gray level i, i.e. the proportion of pixels with gray value i in the total number of pixels in the image; x and y represent the pixel coordinates in the image, x represents the horizontal coordinate of the pixel and y represents the vertical coordinate of the pixel; g(x, y) represents the pixel value at position (x, y); i represents the gray level of the image; m represents the number of rows of the image and n represents the number of columns of the image; f(i) represents the gray value of gray level i after equalization; L represents the number of gray levels of the image; P k f(i) represents the gray value of gray level i after equalization; L represents the number of gray levels of the image; P The cumulative distribution function value of gray level K, that is, the sum of the probabilities of gray level k and all the following gray levels; for the gray image G, the method of image enhancement by histogram equalization can be described as: ； The function represents the mapping relationship between the gray levels before enhancement and the gray levels after enhancement, and the commonly used gray level mapping expression of histogram equalization is: ； The pose includes trajectory, coordinate and angle. In S7, the trajectory accuracy evaluation tool is used to compare the pose trajectory before improvement and the true pose trajectory, and the pose trajectory after improvement and the true pose trajectory, respectively. The two camera poses are aligned through the time stamps, and the difference between the two camera poses is calculated to generate the absolute trajectory comparison graph and the absolute trajectory error numerical distribution graph of the pose before improvement and the true pose, and the absolute trajectory comparison graph and the absolute trajectory error numerical distribution graph of the pose after improvement and the true pose.

Images