An unmanned surface vehicle front underwater net cage detection method and system based on forward-looking sonar

Abstract

The application discloses a method and system for detecting a net cage in front of an unmanned ship based on a forward-looking sonar, and the method comprises the following steps: firstly, performing two color mappings on a sonar image, namely, Jet and Copper color mappings, and performing noise reduction processing on the sonar image; then extracting a net cage target foreground through the Jet color sonar image, considering global gray information and edge feature information in the Copper color image, obtaining two binary images with the net cage target, and fusing the two binary images; through statistical pixel points and morphological processing, the target region is more complete; finally, the position of the net cage target is determined through joint detection of continuous frames, the actual position of the net cage is determined in combination with IMU information, and path planning collision avoidance is realized. The application provides a net cage detection method based on a forward-looking sonar, and the position and size of the net cage target can be accurately acquired, and the real-time detection requirement is met.

Description

Technical Field

[0001] This invention relates to the field of underwater environmental perception technology for unmanned surface vessels (USVs), specifically to a method for detecting underwater net cages in front of USVs based on forward-looking sonar. Background Technology

[0002] When unmanned surface vessels (USVs) navigate in unfamiliar waters, visible light cameras and radar, among other environmental sensing devices, cannot detect underwater net cage obstacles. Collisions with these cages can damage the USV, making underwater net cage detection essential. The underwater environment is complex; waves, seabed sediment, water depth, and currents can cause refraction, diffusion, and absorption of sonar waves, leading to noise interference and aliasing distortion in sonar images. Forward-looking sonar (FLASH) emits pulse signals forward in a fan-shaped pattern, and receives the echo signals from the target via a receiving array. Based on the arrival time and intensity of the echoes, a sonar image of the underwater environment is created, clearly indicating the underwater area ahead of the vessel. Furthermore, FLASH sonar equipment is characterized by its real-time performance, long range, and strong anti-interference capabilities, making it suitable for detecting and locating underwater net cage targets.

[0003] When an unmanned surface vessel (USV) arrives in unfamiliar waters, it activates forward-viewing sonar to detect the surrounding underwater environment. The forward-viewing sonar converts the acoustic reflection characteristics of the underwater environment in front of the USV into a color image in real time. In the sonar color image, net cage targets typically exhibit relatively regular image variations, with clear edge transitions and stronger pixel intensity compared to the surrounding environment, making automatic detection of net cages feasible. After identifying the net cages, their size and location information can be obtained based on coordinate transformation relationships and the USV's attitude, thereby reducing the safety risks caused by USV collisions or entanglement with net cages. Moreover, the location distribution of underwater net cages is rarely recorded, whether in inland lakes, rivers, or at sea, significantly increasing the risks for USVs navigating in these areas. Therefore, a method for detecting underwater net cage targets is urgently needed to improve the safe navigation of USVs in unfamiliar water environments. Summary of the Invention

[0004] The technical problem solved by this invention is to provide a method and system for detecting cages based on forward-looking sonar, which can accurately obtain the position and size of the cage target and meet the requirements of real-time detection.

[0005] The solution of this invention is: an underwater target extraction method based on cascaded features, comprising:

[0006] By utilizing different color images of sonar images, color and edge information are extracted, combined with global information, the extracted color and edge information are fused, and the target region is smoothed through morphological processing to obtain the processed sonar image frame.

[0007] The underwater target is filtered using the detection results of consecutive frames to obtain a stable target. Then, the underwater target is fitted to obtain the underwater target information.

[0008] Preferably, the method further includes using the fitted underwater target information to perform path planning, thereby planning the optimal path for the unmanned surface vessel's navigation.

[0009] Preferably, the extraction of color and edge information includes:

[0010] Two color mapping methods, Jet and Copper, are used to obtain two sonar images, which are then denoised.

[0011] The cascaded features of underwater targets in the two sonar images after noise reduction are extracted. After extracting features from the two sonar images respectively, the suspicious target regions are extracted to obtain a preliminary binary image of the target.

[0012] The cascaded features include pixel features, global grayscale features, and edge features.

[0013] Preferably, the noise reduction process includes using spatial and frequency domain methods to denoise the obtained sonar image, removing underwater reverberation noise and signal transmission noise from the sonar image.

[0014] Preferably, cascaded feature extraction includes:

[0015] Convert the sonar image in Jet color to HSV color space, and extract the underwater target foreground in the HSV color space;

[0016] Global grayscale information and edge feature information are extracted from the sonar image with Copper color.

[0017] Preferably, the processed sonar image frame is obtained in the following manner:

[0018] The two preliminary binary target images are added together, and the white target pixels in the binary images are statistically analyzed to remove interference regions. The connected components are processed using morphological methods to trim the burrs of the target in the sonar image and fill the holes.

[0019] Preferably, fitting the underwater target includes:

[0020] Based on the size and position of the underwater target in the image, the underwater target is fitted into a regular rectangle, and the length, width, and angle information are used to describe the regular rectangle.

[0021] A forward-looking sonar-based underwater cage detection system for unmanned surface vessels includes a forward-looking sonar device, an underwater obstacle detection module, and a data fusion module.

[0022] Before detection, the underwater obstacle detection module calibrates the forward-looking sonar equipment and establishes the correspondence between a single pixel in the image and the direction and angle in the real world, that is, it establishes the transformation relationship between the sonar image coordinate system and the world coordinate system.

[0023] The forward-looking sonar device generates a sonar image based on the received command and transmits the image to the underwater obstacle detection module;

[0024] The underwater obstacle detection module detects sonar images, and when a stable target is detected, it transmits the target's size and location information to the data fusion module.

[0025] The data fusion module fuses the target's size and location information with target information detected by other environmental perception sensors to obtain underwater target information around the unmanned surface vessel, and sends the fused underwater target information to the shore for remote monitoring.

[0026] Preferably, the system further includes a path planning module, which receives the fused underwater target information sent by the data fusion module, automatically plans the optimal path according to the navigation characteristics of the unmanned surface vessel, and sends it to the ship control system to realize the autonomous obstacle avoidance and navigation of the unmanned surface vessel.

[0027] Preferably, the underwater obstacle detection module is implemented as follows:

[0028] The system receives sonar images transmitted from forward-looking sonar equipment, obtains two sonar images using Jet color mapping and Copper color mapping respectively, and performs noise reduction processing on the two images.

[0029] After extracting cascaded features from the two sonar images, the suspicious target region is extracted to obtain a preliminary binary image of the target;

[0030] The two preliminary binary target images are added together, and the white target pixels in the binary images are counted to remove interference regions. The connected components are processed by morphological methods to trim the burrs of the target in the sonar image and fill the holes to obtain the sonar image frame.

[0031] By relying on consecutive sonar image frames and filtering underwater targets through the correlation of adjacent frames, a stable target is obtained. Then, the underwater target is fitted to obtain the size and position information of the underwater target.

[0032] Preferably, the initial binary image of the sonar image for Jet color is obtained in the following manner:

[0033] Apply Gaussian blur to the sonar image of Jet color;

[0034] Convert the color image after Gaussian blurring to the HSV color space;

[0035] Determine the thresholds for hue, lightness, and tint within the HSV color space. In an image within the HSV color space, use high and low thresholds to extract the selected target color regions. Construct a binary image from the extracted target regions, where the white areas are the initially extracted target regions.

[0036] Preferably, the initial binary image for the Copper color sonar image is obtained in the following manner:

[0037] The S401 converts the Copper-colored sonar image into a grayscale image, applies Gaussian blur to the grayscale image, and then uses Haar wavelet transform to filter out low-frequency noise.

[0038] S402 binarizes the grayscale image obtained in step 401 and calculates the initial threshold T;

[0039] S403 performs Canny detection on the grayscale image obtained in step 401. First, Gaussian blur is applied, then the horizontal and vertical gradient magnitudes are calculated, and non-maximum suppression is applied to the magnitudes according to the gradient direction. Finally, edges are preserved and connected by high and low thresholds. After performing Canny edge detection on the image, the number of all connected pixels in the edge detection image is calculated, and all connected components N with fewer than 20 pixels are calculated.

[0040] If N is less than 200, the threshold T is directly used to binarize the grayscale image obtained in step 401; if N is greater than 200, the number of connected pixels N after step 403 is calculated sequentially when the threshold is T+1, ..., 255, until the number of connected components N is less than 200, and a new threshold is obtained and denoted as T0. The grayscale image obtained in step 401 is then binarized according to the new threshold T0.

[0041] After S404 thresholding, a binary image is obtained, where the white area is the target area.

[0042] The advantages of this invention compared to the prior art are:

[0043] This method extracts the features of fish cages from sonar images using image processing techniques that extract cascaded features. Then, it locates the fish cage obstacles through coordinate transformation, enabling the unmanned surface vessel (USV) to avoid underwater obstacles in unfamiliar waters. Two color mapping methods are used: Jet color images, which have high contrast and effectively highlight targets in the image, but ignore smooth transition areas; and Cooper color images, which offer smoother color transitions without sacrificing sharpness, are used to detect obvious fish cage boundaries (JET images) and fish cage edges (Cooper images). The former extracts pixel features from the sonar image, while the latter extracts edge features and global grayscale features, forming a fused cascaded feature set from the sonar image.

[0044] Based on the extracted cascaded features of sonar images, the target region is obtained. Morphological processing is then used to smooth obstacle edges and fill the holes in the center of the obstacles, thus acquiring the preliminary position and shape of the net cage. To increase the stability of net cage detection, the target is filtered in consecutive frames of the sonar images to eliminate unstable interference results. Finally, a stable net cage target is fitted, and the fitted length, width, and angle information, combined with the attitude information and IMU information of the unmanned surface vessel (USV), is used to determine the actual position of the obstacle. These advantages enable the detection of underwater net cages and provide a certain reference value for targets such as shoals, rail fortifications, and reefs. Furthermore, it can effectively improve the underwater environmental perception and autonomous obstacle avoidance capabilities of USVs in unfamiliar waters. Attached Figure Description

[0045] Figure 1 This is a diagram illustrating the composition of the unmanned surface vessel obstacle avoidance system provided by the present invention;

[0046] Figure 2 This is a flowchart of the forward-looking sonar image cage target detection algorithm provided by the present invention;

[0047] Figure 3 This is a flowchart of the Jet color sonar image target extraction process provided by the present invention;

[0048] Figure 4 This is a flowchart of the target extraction process for Copper color sonar images provided by the present invention. Detailed Implementation

[0049] To make the objectives, technical solutions, and advantages of the present invention clearer, specific embodiments of the present invention will be described in further detail below with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are for illustrative purposes only and are not intended to limit the scope of the invention.

[0050] It should also be noted that, for ease of description, the accompanying drawings only show the parts relevant to the invention and not the entirety. Before discussing exemplary embodiments in more detail, it should be mentioned that some exemplary embodiments are described as processes or methods depicted as flowcharts. Although the flowcharts depict operations (or steps) as sequential processes, many of these operations can be performed in parallel, concurrently, or simultaneously. Furthermore, the order of the operations can be rearranged. The process can be terminated when its operation is completed, but may also have additional steps not included in the drawings. The process can correspond to a method, function, procedure, subroutine, subprogram, etc.

[0051] The purpose of this invention is to automatically detect unknown underwater obstacles such as net cages using forward-looking sonar. This implementation is applicable to, but not limited to, the automatic detection and location of underwater obstacles distributed around net cages. This method can be executed by an unmanned surface vessel (USV) underwater obstacle avoidance system based on forward-looking sonar. It can be combined with environmental perception devices such as visible light cameras, lidar, and marine radar, mounted on the USV, to detect both surface and underwater obstacles. Therefore, this invention first designs an underwater target extraction method based on cascaded features. This method utilizes different color images from sonar images to extract color and edge information, combines it with global information, and then fuses the extracted results. Morphological processing is used to smooth the target area. To increase the robustness of target detection, the detection results of consecutive frames are used to filter the target. After obtaining a stable target, the obstacle (underwater target) is fitted, and the fitted obstacle information is sent to the path planning module to provide obstacle information for autonomous obstacle avoidance, ensuring the smooth navigation of the USV in unfamiliar waters.

[0052] Figure 1 This invention provides a method for underwater obstacle avoidance by unmanned surface vessels in unfamiliar waters based on forward-looking sonar.

[0053] This method is divided into a device layer and a data processing layer. The underwater cage detection based on forward-looking sonar provided by this invention is located in the underwater obstacle detection module of the data processing layer. Before detection, the forward-looking sonar device is calibrated to establish the transformation relationship between the world coordinate system and the image coordinate system, and then to establish the correspondence between a single pixel in the image and the direction and angle in the real world, thereby obtaining the accurate obstacle location for use by the unmanned surface vessel's autonomous obstacle avoidance module. The process of this method is specifically divided into the following steps:

[0054] When the S101 unmanned surface vessel is preparing to navigate in unfamiliar waters, the forward-looking sonar equipment receives instructions, generates sonar images, and transmits the images to the underwater cage obstacle detection module.

[0055] S102 underwater cage obstacle detection module application Figure 2 The algorithm is used to detect sonar images. When a stable target is detected, the target's size and location information are transmitted to the data fusion module.

[0056] The S103 data fusion module fuses target information of underwater cage obstacles with target information detected by other environmental perception sensors to obtain obstacle information around the unmanned surface vessel, and then sends it to the path planning module.

[0057] The S104 data fusion module sends the fused obstacle information to the shore base for remote monitoring.

[0058] After receiving obstacle information, the S105 path planning module automatically plans the optimal path based on the navigation characteristics of the unmanned surface vessel (USV) and sends it to the ship control system, enabling the USV to autonomously avoid obstacles and navigate.

[0059] Figure 2 This invention provides an operational method for detecting underwater cages ahead of an unmanned surface vessel (USV) based on forward-looking imaging sonar. This method can be executed by a shipborne system based on forward-looking sonar and other environmental sensing devices, and specifically includes the following steps:

[0060] 201 When the unmanned surface vessel (USV) navigates in unfamiliar waters, the sonar equipment receives instructions from the upper-level module, adjusts the image intensity threshold and Gamma parameter, obtains forward-looking sonar data, and transmits it to the underwater cage obstacle detection module. 202-1 When converting the forward-looking sonar data into a two-dimensional image, Jet color mapping is used to obtain the first color sonar image for subsequent feature extraction.

[0061] When converting forward-looking sonar into a two-dimensional image, Copper color mapping is used to obtain a second color sonar image for subsequent feature extraction.

[0062] 203-1 Denoises the image in the spatial domain by applying Gaussian blur to the Jet color image to remove underwater reverberation and signal transmission noise.

[0063] 203-2 converts the Copper color image to a grayscale image, applies Gaussian blur to the grayscale image in the spatial domain, and then uses Haar wavelet transform in the frequency domain to filter out low-frequency noise.

[0064] Cascaded features of the cage target in the sonar image are extracted, including pixel features, global grayscale features, and edge features. After extracting features from the sonar image, the suspicious target region is extracted to obtain a preliminary binary image of the target, as follows:

[0065] 204-1 According to Appendix Figure 3 The steps involve converting the Jet color image to the HSV color space and extracting the color features of the mesh pixels.

[0066] 204-2 According to Appendix Figure 4 The steps involve combining the global grayscale features and edge features of the Copper image to extract the target of the cage.

[0067] 205. The target image after feature extraction in step 204 is converted into two binary images respectively.

[0068] 206. The two binary images are fused at a ratio of 1:1 to obtain a preliminary binarized target image.

[0069] 207. To make the target object blend more seamlessly, a morphological closing operation is used to fill the gaps using a 5×5 structuring element. Then, an expansion operation is performed on the obstacle using a 10×10 structuring element.

[0070] 208 The above steps are performed on each frame of the forward-looking sonar image to obtain the target binary image of each frame. Then, the results of the previous and next frames are combined to filter out interfering targets and retain stable targets.

[0071] After obtaining the target in the sonar image, the 209 calculates the actual position of the target through the transformation relationship between the sonar image coordinate system and the world coordinate system, fits the shape of the target, and fits the irregular net cage target into a regular rectangle. This reduces redundant information of the target outline and sends information such as the length, width, angle, and position of the net cage target to the path planning module to achieve autonomous obstacle avoidance of the unmanned surface vessel.

[0072] Figure 3 The flowchart for obstacle target extraction based on Jet color sonar images provided by this invention can extract obstacle targets from forward-looking sonar images. The specific steps are as follows:

[0073] S301 performs color mapping on underwater images obtained by forward-looking sonar equipment and uses the Jet color mapping algorithm to obtain color images.

[0074] S302 applies a Gaussian blur to the color image, where the Gaussian kernel size is 3×3.

[0075] The S303 converts a Gaussian-filtered color image to the HSV color space.

[0076] S304 determines the thresholds for hue, lightness, and hue within the HSV color space. In an image within the HSV color space, the high and low thresholds are used to extract the selected target color regions.

[0077] S305 constructs a binary image from the extracted target region, where the white area is the initially extracted target region.

[0078] S306 removes noise regions with fewer than 20 pixels from the binary image and then fills the gaps through morphological dilation to obtain the first binary image.

[0079] Figure 4 The flowchart for obstacle target extraction based on Copper color sonar images provided by this invention can extract obstacle targets from forward-looking sonar images. The specific steps are as follows:

[0080] S401 performs color mapping on underwater images acquired by forward-looking sonar equipment, and uses the Copper color mapping algorithm to obtain color images.

[0081] S402 converts the color image to a grayscale image, applies a Gaussian blur to the grayscale image with a Gaussian kernel size of 3×3, and then uses a Haar wavelet transform to filter out low-frequency noise.

[0082] S403 uses the Otsu algorithm to binarize the grayscale image obtained in step 402 and calculates the initial threshold T.

[0083] S404 performs Canny detection on the grayscale image obtained in step 402. First, it applies Gaussian blur with a 5×5 kernel. Then, it calculates the horizontal and vertical gradient magnitudes and performs non-maximum suppression based on the gradient direction. Finally, it preserves and connects edges using high and low thresholds. After Canny edge detection, it calculates the number of all connected pixels in the edge-detected image and calculates all connected components N with fewer than 20 pixels.

[0084] S405 If N is less than 200, directly use the threshold T to perform binarization on the grayscale image obtained in step 402.

[0085] S406 If N is greater than 200, calculate the number of connected pixels N after step 404 when the threshold is T+1, ..., 255, until the number of connected components N is less than 200, and obtain a new threshold T0. Perform binarization operation on the grayscale image according to the new threshold T0.

[0086] After S407 threshold segmentation, a binary image is obtained, where the white area represents the target region. Noise regions with fewer than 20 pixels in the binary image are removed, and then morphological dilation is used to fill the gaps, resulting in the second binary image.

[0087] Although the present invention has been disclosed above with reference to preferred embodiments, it is not intended to limit the present invention. Any person skilled in the art can make possible changes and modifications to the technical solutions of the present invention by utilizing the methods and techniques disclosed above without departing from the spirit and scope of the present invention. Therefore, any simple modifications, equivalent changes and alterations made to the above embodiments based on the technical essence of the present invention without departing from the content of the technical solutions of the present invention shall fall within the protection scope of the technical solutions of the present invention.

Claims

1. A method for underwater target extraction based on cascaded features, characterized in that... include: Two color mapping methods, Jet and Copper, are used to obtain two sonar images, which are then denoised. The cascaded features of underwater targets in the two sonar images after noise reduction are extracted. After extracting features from the two sonar images respectively, the suspicious target regions are extracted to obtain a preliminary binary image of the target. Cascaded feature extraction includes: converting the Jet-colored sonar image to the HSV color space and extracting color features in the HSV color space; and extracting global grayscale information and edge feature information from the Copper-colored sonar image. Based on the preliminary binary image of the target and global information, the extracted color and edge information are fused, and the target region is smoothed through morphological processing to obtain the processed sonar image frame. The above steps are performed on each frame of the forward-looking sonar image to obtain the target binary image of each frame. After combining the results of the previous and next frames, interfering targets are filtered out to obtain a stable underwater target. Then, the underwater target is fitted to obtain the underwater target information.

2. The method according to claim 1, characterized in that: It also includes using fitted underwater target information for path planning to plan the optimal path for unmanned surface vessels.

3. The method according to claim 1, characterized in that: The noise reduction process includes using spatial and frequency domain methods to denoise the obtained sonar images, removing underwater reverberation noise and signal transmission noise from the sonar images.

4. The method according to claim 1, characterized in that: The processed sonar image frame is obtained in the following manner: The two preliminary binary target images are added together, and the white target pixels in the binary images are statistically analyzed to remove interference regions. The connected components are processed using morphological methods to trim the burrs of the target in the sonar image and fill the holes.

5. The method according to claim 1, characterized in that: Fitting underwater targets includes: Based on the size and position of the underwater target in the image, the underwater target is fitted into a regular rectangle, and the length, width, and angle information are used to describe the regular rectangle.

6. A forward-looking sonar-based underwater cage detection system for unmanned surface vessels, the system being used to implement the underwater target extraction method based on cascaded features as described in any one of claims 1-5, characterized in that: Includes forward-looking sonar equipment, underwater obstacle detection module, and data fusion module; Before detection, the underwater obstacle detection module calibrates the forward-looking sonar equipment and establishes the correspondence between a single pixel in the image and the direction and angle in the real world, that is, it establishes the transformation relationship between the sonar image coordinate system and the world coordinate system. The forward-looking sonar device generates a sonar image based on the received command and transmits the image to the underwater obstacle detection module; The underwater obstacle detection module detects sonar images, and when a stable target is detected, it transmits the target's size and location information to the data fusion module. The data fusion module fuses the target's size and location information with target information detected by other environmental perception sensors to obtain underwater target information around the unmanned surface vessel, and sends the fused underwater target information to the shore for remote monitoring.

7. The system according to claim 6, characterized in that: It also includes a path planning module, which receives the fused underwater target information sent by the data fusion module, automatically plans the optimal path based on the navigation characteristics of the unmanned surface vessel, and sends it to the ship control system to realize the unmanned surface vessel's autonomous obstacle avoidance and navigation.

8. The system according to claim 6, characterized in that: The underwater obstacle detection module is implemented as follows: The system receives sonar images transmitted from forward-looking sonar equipment, obtains two sonar images using Jet color mapping and Copper color mapping respectively, and performs noise reduction processing on the two images. After extracting cascaded features from the two sonar images, the suspicious target region is extracted to obtain a preliminary binary image of the target; The two preliminary binary target images are added together, and the white target pixels in the binary images are counted to remove interference regions. The connected components are processed by morphological methods to trim the burrs of the target in the sonar image and fill the holes to obtain the sonar image frame. By relying on consecutive sonar image frames and filtering underwater targets through the correlation of adjacent frames, a stable target is obtained. Then, the underwater target is fitted to obtain the size and position information of the underwater target.

9. The system according to claim 8, characterized in that: The initial binary image of the Jet color sonar image is obtained in the following manner: Apply Gaussian blur to the sonar image with Jet color; Convert the color image after Gaussian blurring to the HSV color space; Determine the thresholds for hue, lightness, and tint within the HSV color space. In an image within the HSV color space, use high and low thresholds to extract the selected target color regions. Construct a binary image from the extracted target regions, where the white areas are the initially extracted target regions.

10. The system according to claim 8, characterized in that: The initial binary image of the sonar image for the Copper color is obtained in the following manner: The S401 converts the Copper-colored sonar image into a grayscale image, applies Gaussian blur to the grayscale image, and then uses Haar wavelet transform to filter out low-frequency noise. S402 binarizes the grayscale image obtained in step 401 and calculates the initial threshold T; S403 performs Canny detection on the grayscale image obtained in step 401. First, Gaussian blur is applied, then the horizontal and vertical gradient magnitudes are calculated, and non-maximum suppression is applied to the magnitudes according to the gradient direction. Finally, edges are preserved and connected by high and low thresholds. After performing Canny edge detection on the image, the number of all connected pixels in the edge detection image is calculated, and all connected components N with fewer than 20 pixels are calculated. If N is less than 200, the threshold T is directly used to binarize the grayscale image obtained in step 401; if N is greater than 200, the number of connected pixels N after step 403 is calculated sequentially when the threshold is T+1, ..., 255, until the number of connected components N is less than 200, and a new threshold is obtained and denoted as T0. The grayscale image obtained in step 401 is then binarized according to the new threshold T0. After S404 thresholding, a binary image is obtained, where the white area is the target area.

Images