Underwater fishing net sensing method, system and equipment based on monocular vision differential distance measurement

Abstract

The invention discloses an underwater fishing net sensing method, system and equipment based on monocular vision differential distance measurement, and belongs to the technical field of machine vision image processing. The method comprises the following steps: acquiring an underwater fishing net image through an underwater camera; enhancing the acquired underwater fishing net image; performing fishing net line edge extraction on the enhanced fishing net image; extracting fishing net structure features in the fishing net image through HOUGH transformation; constructing a time sequence difference distance measurement mathematical model, and calculating a fishing net distance according to a pixel distance between adjacent net lines of the fishing net; and filtering a sequence distance measurement result by adopting Kalman filtering, removing abnormal values, and obtaining a final fishing net distance. According to the underwater fishing net sensing method disclosed by the invention, the underwater fishing net can be detected in the movement process, and the distance of the underwater fishing net can be measured in a non-contact manner without priori knowledge.

Description

Technical Field

[0001] This invention belongs to the field of machine vision image processing technology, and relates to target detection technology based on visible light, and particularly to an underwater fishing net sensing method, system and device based on monocular visual differential ranging. Background Technology

[0002] With the continuous increase in the frequency of marine fishing activities, abandoned and lost fishing nets are widely present in the world's oceans. These abandoned fishing nets not only pose a threat of entanglement and accidental catch to marine life such as fish, sea turtles, and whales, but also easily attach to underwater facilities or ship propellers, interfering with their normal operation and even causing serious consequences. Currently, underwater fishing net cleaning operations mostly rely on manual diving, which has problems such as low personnel safety and low operational efficiency. With the upgrading of industrial automation and intelligence, underwater robots carrying cutting tools for underwater fishing net cleaning operations is the future development trend. However, the underwater environment is complex and the light intensity is weak. During the autonomous fishing net sensing process of underwater robots, inaccurate target identification or unstable positioning of fishing nets is very likely to occur, resulting in problems such as low operational efficiency, accidental injury to marine life, and cutting failure, which seriously restricts the efficiency and safety of underwater cleaning operations.

[0003] Optical vision sensors combined with active lighting can effectively achieve near-range underwater environmental perception. Compared to acoustic and electromagnetic detection methods, visual detection is relatively mature. Image-based target detection and recognition algorithms have been widely applied in fields such as autonomous driving and surveillance. However, in underwater environments, interference such as currents and swells increases the difficulty of detection and recognition, and the scarcity of underwater fishing net data limits the use of data-driven neural network target detection algorithms. Therefore, proposing a robust visual perception-based fishing net detection and ranging algorithm tailored to the characteristics of underwater fishing nets is an urgent problem to be solved. Summary of the Invention

[0004] In view of the above-mentioned defects or improvement needs of the existing technology, the present invention provides an underwater fishing net sensing method, system and equipment based on monocular visual differential ranging, thereby solving the technical problems of underwater fishing net detection and ranging.

[0005] To achieve the above objectives, according to a first aspect of the present invention, an underwater fishing net sensing method based on monocular visual differential ranging is provided, comprising the following steps: S1: Underwater fishing net images are captured using an underwater camera; S2: Enhancement processing of the acquired underwater fishing net images; S3: Extract the edges of the fishing net lines from the enhanced fishing net image; S4: Extract the structural features of fishing nets from fishing net images using HOUGH transform; S5: Construct a temporal differential ranging mathematical model to calculate the distance between fishing nets based on the pixel distance between adjacent net lines. S6: Use Kalman filtering to filter the sequence ranging results, remove outliers, and obtain the final fishing net distance.

[0006] In summary, compared with the prior art, the above-described technical solutions conceived by this invention can achieve the following beneficial effects: 1. Underwater fishing nets can be detected during movement. This invention can dynamically measure fishing nets and monitor their position underwater in real time; 2. Non-contact distance measurement of underwater fishing nets without prior knowledge. This invention combines the dynamic measurement process of fishing nets with the principle of monocular perception. Fishing net perception can be achieved by comparing the distance changes of the net lines at different locations, without requiring prior knowledge of the net's size. Attached Figure Description

[0007] Figure 1 This is a flowchart illustrating an underwater fishing net sensing method based on monocular visual differential ranging, provided in an embodiment of the present invention. Detailed Implementation

[0008] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention. Furthermore, the technical features involved in the various embodiments of this invention described below can be combined with each other as long as they do not conflict with each other.

[0009] Example 1: This invention provides an underwater fishing net sensing method based on monocular visual differential ranging. The core idea is to use the differences in the three-dimensional geometric structure of the target and the background in the event point cloud to separate the target from the background, thereby realizing the extraction of moving targets.

[0010] like Figure 1 As shown, the process begins by acquiring underwater fishing net images from a distance using an underwater telephoto camera. Then, algorithms such as blur enhancement are employed to improve the grayscale difference between the foreground and background. Further, edge detection and morphological filtering are used to extract the fishing net's edge structure. Next, the Hough line detection method is used to initially detect fishing net lines in the image, including using adaptive thresholding to improve the robustness of Hough line detection and performing post-processing operations through filtering algorithms to remove interfering lines and reduce the false detection rate. Further, a temporal difference ranging mathematical model is constructed, combined with the monocular ranging principle, to calculate the fishing net distance. Finally, Kalman filtering is used to filter out abnormal fishing net distances. The specific steps include the following: S1: Acquire underwater fishing net images. Specifically, in one embodiment of the present invention, underwater fishing net images are acquired by an underwater vision sensor, such as an underwater camera, and transmitted to an image processing board through the net port. The method of acquiring underwater fishing net images is not limited to this. The above is only one implementation method and is not a limitation of the present invention.

[0011] S2: Enhancement processing is applied to the acquired underwater fishing net images. In one embodiment, due to the uneven distribution of the underwater medium, light propagating underwater encounters suspended particles and is scattered, causing a fogging effect and resulting in a severe decrease in image contrast. Therefore, this invention uses a Gaussian filter to smooth the acquired images. Specifically, express The Gaussian function weight value corresponding to the pixel. Let the standard deviation of the Gaussian distribution be represented by the following mathematical expression for the Gaussian function:

[0012] Assuming the input image is The output image after Gaussian filtering It can be represented as:

[0013] Read image data and obtain its minimum grayscale value. With the maximum value And calculate the grayscale median. :

[0014] The median This serves as the balance point for the blur mapping, defining the center of symmetry for enhancing intensity. Furthermore, for each pixel's grayscale value... Perform normalization processing and calculate its fuzzy input.

[0015]

[0016] When the denominator is zero, it is replaced with the system's smallest floating-point number to avoid numerical overflow.

[0017] In one implementation, the linear generalized fuzzy mapping operator LGFO is defined as follows:

[0018] in This indicates the result of fuzz enhancement. Indicates fuzzy input. This represents the fuzzy enhancement adjustment factor.

[0019] This operator performs compression or expansion mapping on the input within different grayscale regions, thereby enhancing the grayscale difference between edges and low-contrast areas. The result is then blurred and output. Map back to the original grayscale space to obtain the image mapping result. :

[0020] in, This indicates a blurred output of the image. This represents the median gray level of the image. This represents the minimum grayscale value of the image.

[0021] S3: Extract the edges of the fishing net lines in the enhanced fishing net image. Specifically, perform iterative threshold segmentation on the input fishing net image to distinguish between foreground and background regions. If the input is a color image, convert it to grayscale; if the pixel value is not between 0 and 255, perform normalization. Set the initial threshold value to the image's grayscale mean or an externally set value. Then, based on the current threshold, divide the fishing net image into a foreground set and a background set, and calculate the average grayscale value of each. and Update the threshold according to the following formula:

[0022] in, This represents the average gray value of the foreground set in the fishing net image. This represents the average gray value of the background set in the fishing net image. This represents the updated value of the current threshold.

[0023] Threshold convergence is considered achieved when two consecutive threshold changes are less than a preset tolerance, or when the number of iterations exceeds a maximum limit. The convergent threshold is then used to binarize the image, resulting in a segmented image with foreground and background separation. After obtaining the binary segmentation result, an edge detection operation is performed using the gradient-based Canny operator to extract the target contour structure. This operator calculates the horizontal and vertical gradient components of the image to obtain gradient magnitude and direction information, and combines non-maximum suppression with a double-threshold connection algorithm to extract edge pixels, highlighting gray-level abrupt change regions at the target boundary.

[0024] Furthermore, morphological optimization processing is performed on the edge-detected image to smooth the edge structure and eliminate isolated noise points. First, median filtering is applied to the edge map with a 5×5 filter window to reduce high-frequency noise and maintain local continuity. Then, morphological operations are performed on the image using erosion and dilation structuring elements of size 3×3. Erosion is used to refine the edges and remove isolated pixels, while dilation is used to restore the main boundary morphology, making the edge structure more coherent.

[0025] In one implementation, to further remove false edges and isolated small regions, connected component analysis is performed on the resulting image, all connected regions are labeled, and their pixel areas are calculated. If a region with an area smaller than a preset threshold is detected, the corresponding region's pixels are set to zero, thereby automatically eliminating small noise blocks.

[0026] S4: Extract the structural features of the fishing net in the image. Specifically, after edge extraction, the principle of line detection is used to perform line detection on the binarized edge image to extract the structural features of the fishing net in the image. This includes two sub-steps: adaptive line detection and interference filtering. S41: Adaptive line detection is performed on the binarized edge image. To adaptively determine the detection sensitivity under different image density conditions, this invention proposes an adaptive HOUGH threshold adjustment mechanism. Let the total number of edge pixels be... The total number of pixels in the image is Define edge density:

[0027] According to edge density The size of the threshold adjustment factor is used to calculate the threshold adjustment factor. :

[0028] in, This is the steepness coefficient. As the density center, For dynamic range, the final detection threshold Calculated using the following formula:

[0029] in, and These represent the lower and upper thresholds, respectively. Through this adaptive threshold mechanism, the sensitivity of the Hough transform is automatically adjusted based on the edge sparsity, avoiding over-detection or under-detection issues under weak edge or high-noise conditions.

[0030] After the threshold is determined, the Hough transform line detection algorithm is used to detect edges in the binarized image. First, a mathematical representation model of the line is established in the polar coordinate parameter space. Let any line in the image be represented by the parametric equation:

[0031] in, This represents the distance from the line to the origin. This represents the angle between the line normal and the horizontal axis. Using this model, all possible lines on the image plane can be transformed to... Represented in parameter space.

[0032] Then iterate through all pixels in the image, and for each edge point with a gray value of "1"... According to the parameter space relation Calculate at different angles The corresponding The value is calculated iteratively within a preset angular step size range, and the corresponding value is calculated in the two-dimensional accumulator space. The index is used for cumulative voting. The value of each cell in the accumulator represents how many edge points support the line with that parameter combination.

[0033] When the accumulated value of a cell exceeds the aforementioned adaptive threshold When a corresponding significant straight line is found, the system is considered to have existed. The system then extracts the parameters of all lines that satisfy the condition based on the local extrema in the accumulator. The parameters are then sorted according to detection confidence or cumulative strength. To improve subsequent calculation efficiency, all detected line parameters are stored in an array structure for use by the angle statistics, cluster analysis, and spatial filtering modules.

[0034] S42: The image after adaptive Hough threshold adjustment is subjected to interference filtering. Since the Hough transform may generate a large number of noisy lines or isolated boundary lines in complex images, this invention uses a multi-level filtering mechanism to gradually remove redundant or abnormal lines in a coarse-to-fine order, retaining the main directional feature lines. Specifically, this includes: S421: Perform quantity filtering on the image after adaptive HOUGH threshold adjustment. When the number of detected lines is less than the preset lower limit, the detection result is directly determined to be invalid, thereby avoiding incorrect structure judgment when the noise is too strong or the target is missing.

[0035] S422: Perform density filtering on the set of lines that have passed quantity filtering. First, check the angles of all lines. The data is converted to angular units and restricted to the range [0, 180°). Based on the angular intervals, a two-stage clustering mechanism is employed: First-stage loose clustering: using a step size larger than the first angular interval, such as 15-20°, lines in similar directions are divided into coarse clusters; Second-stage strict clustering: within the main clusters, further subdivided using a second angular interval, such as 0-5°, to extract the main direction line groups. This clustering process counts the number of lines in each angular cluster, retaining the line sets within the largest cluster, suppressing non-main direction line segments, and highlighting the structurally consistent main boundary directions.

[0036] S423: Perform spatial aggregation and distance filtering on line segments that have undergone angle density filtering. First, the polar coordinate parameters of each line are... Convert to Cartesian coordinates:

[0037] in, Represents the polar coordinate parameters of the i-th line. The coordinate points in the Cartesian coordinate system are used as feature vectors and input into the DBSCAN clustering algorithm for spatial aggregation. The line parameters within each cluster are merged using an averaging method with a preset step size to obtain the merged average line parameter. polar coordinate parameters To further improve the consistency of the results, nearest neighbor distance statistics and box filter were performed on line segments within the same angle cluster. The interquartile range (IQR) of the distance distribution was obtained by calculating the minimum lateral distance between the midpoints of each line segment, and a distance threshold was defined. :

[0038] Where λ is an adjustable coefficient, Q3 represents the upper quartile, which is the threshold of the upper 25% of the distance distribution, and Q1 represents the lower quartile, which is the threshold of the lower 25% of the distance distribution. The interquartile range represents the distance distribution, where isolated lines with a distance greater than a threshold are discarded, and the group of main directional lines that are densely distributed in space are retained.

[0039] S5: Construct a temporal differential ranging mathematical model to calculate the distance between adjacent lines of the fishing net. Specifically, after outputting the line segment information filtered by the main direction line group, this invention estimates the distance between the fishing net and the camera using a temporal differential ranging method. The specific construction steps are as follows: S51: Assume motion for the camera, for example, the observation platform carrying the camera moves at a constant speed v along a straight line during the first time interval. This first time interval is usually a short interval, such as 0-1 second. Two frames are captured consecutively, with an inter-frame time interval of [missing information]. . S52: Matching line groups and calculating pixel displacements for consecutive frames t and For identical fishing net line pairs, calculate the pixel spacing p and pixel displacement between two fishing net lines. ,in The pixel distance between the fishing net lines in the previous frame. This represents the pixel distance between the fishing net lines in the next frame.

[0040] S53: Pixel-to-actual-distance mapping. Let the focal length of the camera be f, the vertical spacing of the fishing net segments in actual space be L, and the distance between the camera and the fishing net be D. Then the pinhole imaging relationship exists:

[0041] Therefore, pixel displacement between two frames Corresponding actual spatial difference for:

[0042] Where D represents the distance between the camera and the fishing net, and f represents the focal length of the camera. Indicates the pixel displacement between two image frames. It represents the spatial difference corresponding to the pixel displacement between two image frames.

[0043] S54: Calculate the fishing net distance based on the camera's movement speed, since the camera moves along the direction of movement. = Pixel displacement The distance D between the net and the actual fishing net satisfies a geometric relationship:

[0044] Will Substituting into the above formula, we can obtain the distance of the fishing net:

[0045] S55: Aggregate multiple line segments, and calculate separately for each matched main direction line segment. Ultimately, the fishing net was far from... Using the mean:

[0046] Where N is the number of matched fishing net line pairs, The corresponding calculated fishing net distance for the matched fishing net line pairs.

[0047] Using the above formula, the actual spatial distance between the fishing net and the camera can be accurately calculated based on the pixel displacement of the fishing net line between consecutive frames, the camera's movement speed, and the known focal length.

[0048] S6: Kalman filtering is used to filter the time-series ranging results and remove outliers. To further improve the stability and accuracy of the ranging results, the time-series ranging result sequence obtained in S5 is further filtered. Filtering is performed. The Kalman filter algorithm is used for dynamic estimation and anomaly suppression of the distance sequence. First, the ranging state equation and the observation equation are established:

[0049] in, This represents the true distance state at the current time t. This represents the true distance state at the previous time t-1. The observation distance is calculated using time-series difference. and These are process noise and observation noise, respectively. The filtering process includes two steps: prediction and update. The prediction phase estimates and predicts the current distance based on the state from the previous step.

[0050]

[0051] in, This represents the predicted distance at the current time t. This represents the optimal distance estimate at the previous time t-1. This represents the prediction covariance at the current time t. This represents the optimal covariance at the previous time t-1. This represents the process noise covariance.

[0052] The update phase incorporates current observations to revise the predicted values;

[0053]

[0054] in, Indicates Kalman gain, Represents the observation noise covariance. This represents the observation value at the current time t. This represents the optimal distance estimate at the current time t. This represents the optimal covariance at the current time t.

[0055] By iteratively executing the above process, a smooth estimation of the distance sequence can be achieved, effectively suppressing abnormal jumps caused by changes in illumination, edge detection errors, or motion jitter, resulting in more stable and continuous ranging results.

[0056] Example 2: This invention also discloses an underwater fishing net sensing system based on monocular visual differential ranging, comprising: an underwater image acquisition module for acquiring underwater fishing net images from a distance; an underwater image enhancement module for acquiring high-quality and high-contrast underwater fishing net images through histogram equalization; a fishing net line detection module for acquiring the edges of the fishing net lines through adaptive HOUGH line detection and filtering modules; a fishing net distance calculation module for calculating the fishing net distance through monocular perception and temporal differential ranging principles; and an abnormal distance removal module for filtering out abnormal fishing net distances through Kalman filtering to obtain the final fishing net ranging result.

[0057] The process of implementing the specific functions of the above modules is described in Example 1, and will not be repeated here to avoid redundancy.

[0058] Example 3: This invention provides an electronic device, comprising: Computer-readable storage media and processors; Computer-readable storage media are used to store executable instructions; The processor is used to read executable instructions stored in a computer-readable storage medium and execute the method as described in any of the embodiments 1 above.

[0059] In summary, this invention provides an underwater fishing net sensing method, system, and device based on monocular visual differential ranging. By enhancing image contrast, using edge detection algorithms to extract the main structure of the fishing net, introducing an adaptive Hough threshold detection method, and designing a series of filtering algorithms to filter out interfering straight lines, a group of main directional straight lines conforming to the fishing net structure is obtained, reducing the false detection rate of straight lines. A temporal differential ranging mathematical model is constructed, the fishing net distance is calculated by combining the monocular ranging principle, and Kalman filtering is used to filter out abnormal fishing net distances.

[0060] It should be noted that the purpose of disclosing the embodiments is to help further understand the present invention. However, those skilled in the art will understand that various substitutions and modifications are possible without departing from the spirit and scope of the present invention and the appended claims. Therefore, the present invention should not be limited to the content disclosed in the embodiments, and the scope of protection of the present invention is defined by the scope of the claims.

[0061] Those skilled in the art will readily understand that the above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for sensing underwater fishing nets based on monocular visual differential ranging, characterized in that, Includes the following steps: S1: Underwater fishing net images are captured using an underwater camera; S2: Enhancement processing of the acquired underwater fishing net images; S3: Extract the edges of the fishing net lines from the enhanced fishing net image; S4: Extract the structural features of fishing nets from fishing net images using HOUGH transform; S5: Construct a temporal differential ranging mathematical model to calculate the distance between fishing nets based on the pixel distance between adjacent net lines. S6: Use Kalman filtering to filter the sequence ranging results, remove outliers, and obtain the final fishing net distance.

2. The underwater fishing net sensing method based on monocular vision differential ranging according to claim 1, characterized in that, Step S4 includes: S41: Perform adaptive line detection on the binarized edge image; S42: Perform interference filtering on the image after adaptive HOUGH threshold adjustment, removing redundant or abnormal straight lines in order from coarse to fine, and retaining the main directional feature lines.

3. The underwater fishing net sensing method based on monocular visual differential ranging according to claim 2, characterized in that, Step S42, which involves interference filtering of the image, includes: S421: Perform quantity filtering on the image after adaptive HOUGH threshold adjustment. If the number of detected lines is less than the preset lower limit, the detection result is deemed invalid. S422: Perform density filtering on the set of lines that have passed quantity filtering, and adjust the angles of all lines. Convert to angular scale and limit to the range [0, 180). Based on the angular interval, a two-level clustering mechanism is adopted: First-level loose clustering: with the first angular interval as the step size, lines in similar directions are divided into coarse clusters; Second-level strict clustering: within the main cluster, the lines are further subdivided with the second angular interval to extract the main direction line group. The first angle ranges from 15 to 20 degrees, and the second angle is smaller than the first angle, ranging from 0 to 5 degrees.

4. The underwater fishing net sensing method based on monocular visual differential ranging according to claim 3, characterized in that, Step S42 further includes: S423: Perform spatial aggregation and distance filtering on the line segments after angle density filtering, and convert the polar coordinate parameters of each line. Convert to Cartesian coordinates: in, Represents the polar coordinate parameters of the i-th line. The corresponding coordinate points in the Cartesian coordinate system are used as feature vectors and input into the DBSCAN clustering algorithm for spatial aggregation. The line parameters within each cluster are merged by averaging according to a preset step size, and the merged average line parameter is calculated. polar coordinate parameters For line segments within the same angular cluster, perform nearest neighbor distance statistics and box filter to calculate the minimum lateral distance between the midpoints of each line segment, obtain the interquartile range (IQR) of the distance distribution, and define a distance threshold: Where λ is an adjustable coefficient, Q3 represents the upper quartile, which is the threshold of the upper 25% of the distance distribution, and Q1 represents the lower quartile, which is the threshold of the lower 25% of the distance distribution. It represents the interquartile range of the distance distribution, where isolated lines with a distance greater than a threshold are discarded, and the group of main directional lines that are densely distributed in space are retained.

5. The underwater fishing net sensing method based on monocular visual differential ranging according to claim 1, characterized in that, Step S5 includes: S51: Assume motion for the camera, assuming the observation platform carrying the camera moves at a constant speed v along a straight line during the first time interval, and continuously captures two frames of images with an inter-frame time interval of v. ; S52: Matching line groups and calculating pixel displacements for consecutive frames t and For identical fishing net line pairs, calculate the pixel spacing p and pixel displacement between two fishing net lines. ,in The pixel distance between the fishing net lines in the previous frame. The pixel distance between the fishing net lines in the next frame; S53: Pixel-to-actual-distance mapping. Let the camera's focal length be f, the vertical spacing of the fishing net segments in actual space be L, and the distance between the camera and the fishing net be D. Pinhole imaging relationship: Pixel shift between two frames Corresponding actual spatial difference for: Where D represents the distance between the camera and the fishing net, and f represents the focal length of the camera. Indicates the pixel displacement between two image frames. This represents the spatial difference corresponding to the pixel displacement between two image frames; S54: Calculate the distance to the fishing net based on the camera's movement speed; the camera moves along the direction of movement. = Pixel displacement The geometric relationship between the distance D from the actual fishing net and the net is as follows: Will = Substituting into the above formula, we can obtain the distance of the fishing net: S55: Aggregate multiple line segments, and calculate separately for each matched main direction line segment. Fishing net distance Using the mean: Where N is the number of matched fishing net line pairs, The corresponding calculated fishing net distance for the matched fishing net line pairs.

6. The underwater fishing net sensing method based on monocular visual differential ranging according to claim 5, characterized in that, This includes the time-series distance measurement results of the calculated fishing net distance. Perform filtering processing, where This indicates the real-time distance measurement result of the fishing net.

7. A system for implementing the underwater fishing net sensing method based on monocular vision differential ranging as described in any one of claims 1-6, characterized in that, include: Underwater image acquisition module: used to acquire underwater fishing net images over long distances; Underwater image enhancement module: used to obtain high-quality and high-contrast underwater fishing net images through histogram equalization; Fishing Net Straight Line Detection Module: Used to obtain the edge of the fishing net lines through adaptive HOUGH straight line detection and filtering modules; Fishing Net Distance Calculation Module: Used to calculate the distance to the fishing net using monocular vision and time-series differential ranging principles; Anomaly Distance Removal Module: Used to filter out abnormal fishing net distances using Kalman filtering to obtain the final fishing net distance measurement result.

8. An electronic device, characterized in that, include: Computer-readable storage media and processors; The computer-readable storage medium is used to store executable instructions; The processor is configured to read executable instructions stored in the computer-readable storage medium and execute the method as described in any one of claims 1-6.

Images