Method and device for detecting and grabbing underwater cultured objects

Abstract

The present application belongs to the technical field of underwater detection, and particularly relates to a method and device for detecting and grabbing underwater cultured objects. The method comprises: acquiring a left view and a right view photographed by two cameras installed in a parallel binocular form; restoring the left view and the right view; detecting the cultured objects in the restored left view and right view to obtain a target frame of the cultured objects; performing stereo matching on the target frame region of the cultured objects to obtain a disparity of the left view; calculating the size and position of the cultured objects according to the disparity of the left view; and grabbing the cultured objects according to the position and size of the cultured objects. The present application can automatically acquire optical binocular images of underwater scenes, determine the distance and size of various underwater cultured object targets, and implement grabbing, thereby improving the efficiency and intelligent level of aquatic product fishing.

Description

Technical Field

[0001] This invention belongs to the field of underwater detection technology, specifically relating to a method and device for detecting and capturing underwater aquatic organisms. Background Technology

[0002] With the increasing demand for seafood, aquaculture is developing towards intelligent and refined operations. Underwater robotics plays a crucial role in promoting the automation of aquaculture. Underwater target detection and automatic grasping of farmed organisms, as key technologies for the classification and harvesting of farmed organisms, have significant research value and commercial prospects.

[0003] Due to the poor optical imaging in underwater environments, existing underwater target detection methods still suffer from low detection accuracy. Summary of the Invention

[0004] To address the shortcomings of existing technologies, this invention provides a method and apparatus for detecting and grasping underwater aquatic organisms. It can automatically acquire optical binocular images of underwater scenes, determine the distance and size of various underwater aquatic organism targets, and grasp them, thereby improving the efficiency and intelligence level of aquatic harvesting.

[0005] To address the shortcomings of existing technologies, the technical solution provided by this invention is as follows:

[0006] A method for detecting and capturing underwater aquaculture organisms, comprising,

[0007] Obtain the left and right views taken by two cameras mounted in a parallel binocular configuration;

[0008] Restore the left and right views;

[0009] Detect the aquatic organisms in the restored left and right views to obtain the target bounding boxes of the aquatic organisms;

[0010] Perform stereo matching on the target bounding box area of ​​the cultured organism to obtain the parallax of the left view;

[0011] Calculate the size and position of the farmed animals based on the parallax of the left view;

[0012] Grab the farmed animals based on their location and size.

[0013] Preferably, the method further includes correcting the left and right views before restoring them, including...

[0014] The camera's intrinsic and extrinsic parameters are obtained through calibration:

[0015] Construct the likelihood function:

[0016]

[0017] Where, θ ij Let x represent the coordinates of the i-th corner point of the j-th calibration map, where j = 1, 2, ..., x, and x is the total number of calibration maps. This represents the total number of corner points in a single calibration map. For θ ij The estimated value of R; ij and T ij M represents the rotation and translation vectors of the i-th corner point of the j-th calibration image, respectively; j Let M be the three-dimensional coordinate matrix of the corner points in the j-th calibration image; D = [k1, k2, σ1, σ2, k3] is the distortion parameter vector of the non-ideal lens model, and M is the three-dimensional coordinate matrix of the corner points in the j-th calibration image. I Z is the camera's intrinsic parameter matrix; Z is the planar homography matrix. This is an estimate of Z;

[0018] By setting the distortion coefficient and reprojection error threshold, the likelihood function is solved to obtain the camera's intrinsic and extrinsic parameters.

[0019] Based on the camera's intrinsic and extrinsic parameters and the binocular stereo correction theory, coplanar correction and row alignment correction are performed on the left and right views.

[0020] Preferably, the restoration of the left and right views includes,

[0021]

[0022] Among them, the left and right views before restoration are denoted as the original images, J c (x) represents the pixel value of pixel x in the c channel of the restored image. c (x) represents the pixel value of pixel x in channel c of the original image, A c Let c be the background light intensity of channel c, where c∈(R,G,B), and R, G, and B are the red, green, and blue channels in the RGB color channel. Let be the transmittance of channel c at pixel x;

[0023] The process for determining the background light intensity is as follows:

[0024] Arrange the pixel values ​​of the dark channel image of the fused fog lines in ascending order, and select the top 10% of pixels. Then, among these pixels in the original image, find the pixel with the lowest pixel value in each of the red, green, and blue channels. Use the lowest pixel value in each of the red, green, and blue channels as the background light value A for the red, green, and blue channels. R A G A B ;

[0025] The process of calculating the transmittance of channel c at pixel x is as follows:

[0026] Construct the minimization function:

[0027]

[0028] Where δ is the balance coefficient, N x Let x be a local region centered at pixel x, and y be N. x Pixels within; Let be the maximum transmittance of the red channel at pixel x; σ(x) is the value obtained from the fog line. Standard deviation; Let x be the transmittance of the red channel at pixel x. I represents the transmittance of the red channel at pixel y; R (x), I R (y) represents the pixel values ​​of pixel x and pixel y in the red channel of the original image, respectively;

[0029] The calculation formula is as follows:

[0030]

[0031] Among them, t R (x) represents the initial transmittance of the red channel at pixel x; t Rm This represents the lower threshold of the transmittance of the red channel at pixel x.

[0032] t Rm The calculation formula is as follows:

[0033]

[0034] Among them, I G (x), I B (x) represents the pixel value of pixel x in the green and blue channels of the original image, respectively; β RG β represents the ratio of the attenuation coefficients of red light to green light. RB This represents the ratio of the attenuation coefficients of red light to blue light.

[0035] Set t R (x), obtained by iteratively solving the minimization function. The transmittance of the green channel at pixel x is then calculated using the following formula. Transmittance of the blue channel at pixel x

[0036]

[0037] Where, β R β G β B These are the attenuation coefficients of red, green, and blue light in water, respectively.

[0038] Preferably, the YOLOv5s convolutional neural network model is used to detect the aquatic animals in the restored left and right views to obtain the target bounding boxes of the aquatic animals;

[0039] The YOLOv5s convolutional neural network model includes a backbone network, a feature extraction network, and a prediction network.

[0040] The backbone network uses CSPNet and Focus as the base network, the sampling layer is the C3CBAM module, and the RFB module is used to extract the information of the restored left and right views.

[0041] The feature extraction network constructs FPN and PAN feature pyramids for feature fusion, and uses bilinear interpolation and C3CBAM modules for splicing, adding a CBAM attention mechanism on the smallest scale information;

[0042] The prediction network uses an anchor-based decoupled detection head to output three different resolutions for multi-scale prediction.

[0043] Preferably, the step of stereo matching the target frame region of the cultured organism to obtain the parallax of the left view includes:

[0044] Based on the target bounding box region of the cultured animals in the left view, update the target bounding box region of the cultured animals in the right view according to the intersection-union ratio of the target bounding box regions of the cultured animals in the left and right views;

[0045] Calculate the matching cost of pixels in the target bounding box region of the left view, including,

[0046] C(p,d)=α*β*ρ(C census (p,d),λ ce )+(1-β)*ρ(C AD (p,d),λ AD )+(1-α)*ρ(C grad (p,d),λ grad )

[0047] Where C(p,d) is the matching cost of pixel p in the target box region of the left view.

[0048] d is any value among the pixel differences between pixel p and all pixels in the target area of ​​the right view except pixel p.

[0049] C census (p,d), C AD (p,d), C grad(p, d) represent the cost of pixel p in the target bounding box region of the left view calculated by the census algorithm, AD algorithm, and gradient algorithm, respectively; λ ce , λ AD , λ grad These represent the control parameters for the census algorithm, AD algorithm, and gradient algorithm, respectively.

[0050] α is the adaptive weight parameter, and β is the fixed weight.

[0051] ρ(C census (p,d),λ ce ), ρ(C AD (p,d),λ AD ), ρ(C grad (p,d),λ grad The calculation formulas for ) are as follows:

[0052]

[0053]

[0054]

[0055] Among them, C grad The formula for calculating (p,d) is as follows:

[0056] C grad (p,d)=max(0,L(p R )-L max (p R ),L min (p L )-L(p L ))

[0057] Wherein, L(p L L(p) represents the grayscale value of the left view pixel. R ) represents the pixel grayscale value of the right view, L max (p R ) represents the maximum grayscale value of the target area in the right view, L min (p L L(p) represents the minimum grayscale value of the target area in the left view. L ), L(p R The calculation formula for ) is as follows:

[0058]

[0059]

[0060] Where a and b are weight values, It is the normalized value of the grayscale value of the pixel point p in the m and n directions of the target box region in the left view. It is the normalized value of the grayscale value of the pixel point p in the m and n directions of the target box area in the right view;

[0061] Calculate the matching cost for each pixel in the target box region of the left view to obtain the cost matrix of the target box region of the left view;

[0062] Calculate the optimal arm length τ, aggregate the cost matrix of the target box region in the left view, and obtain the disparity of the target box region in the left view;

[0063] The formula for calculating the optimal arm length τ is as follows:

[0064]

[0065]

[0066] Where τ0 is the initial value of the preset pixel threshold, τ1 is the cutoff value of the preset pixel threshold, L is the preset arm length threshold, and l pq I represents the pixel difference between pixel p and pixel q on the horizontal support arm. p ,I q ,I q′ These are the pixel values ​​of three pixels, p, q, and q′, respectively; q′ is the pixel preceding pixel q on the horizontal support arm.

[0067] Preferably, the step of calculating the size and position of the farmed animals based on the parallax of the left view includes,

[0068]

[0069] Where Xw and Yw represent the world coordinates of the pixel on the horizontal and vertical axes of the world coordinate system, respectively; Zw represents the depth of the pixel; m and n are the horizontal and vertical coordinates of the pixel in the left view, respectively; κ is the focal length of the camera; and b is the baseline distance. The disparity of pixels in the left view;

[0070] Create the minimum bounding rectangle of the farmed organisms, and calculate the dimensions of the farmed organisms and the distance between the farmed organisms and the camera:

[0071]

[0072] Where LY represents the length of the cultured organism; W represents the width of the cultured organism; H represents the height of the cultured organism; and Dis represents the distance between the cultured organism and the camera. The minimum bounding rectangle R of the cultured organism is given by [the following]. N Maximum and minimum parallax within; r1 l r2The minimum bounding rectangle R of the cultured organism is given by [the following]. N The side length.

[0073] Preferably, the step of grasping the farmed organisms based on their location and size includes,

[0074] Calculate the weight of the farmed animals based on their size, and then calculate the clamping force of the grippers based on the weight of the farmed animals.

[0075] The end-effector pose of the multi-axis robotic arm is determined based on the position of the farmed animal. The joint angles of the multi-axis robotic arm are then determined based on the end-effector pose. The multi-axis robotic arm is driven to move based on the joint angles, bringing the gripper to the farmed animal. Based on the determined gripper force, the gripper is triggered to grasp the farmed animal.

[0076] An underwater aquaculture detection and grasping device includes two cameras, a power supply, a controller, a multi-axis robotic arm, and grippers;

[0077] The two cameras are installed in a parallel binocular configuration to acquire images of the underwater aquaculture organisms. The images of the underwater aquaculture organisms are recorded as the left view and the right view, and uploaded to the controller.

[0078] The power source is used to provide power to the camera, controller, multi-axis robotic arm, and gripper;

[0079] The controller is used to: restore the left and right views; detect the farmed animals in the restored left and right views to obtain the target bounding box of the farmed animals; perform stereo matching on the target bounding box area of ​​the farmed animals to obtain the parallax of the left view; calculate the size and position of the farmed animals based on the parallax of the left view; calculate the weight of the farmed animals based on the size of the farmed animals; calculate the gripping force of the gripper based on the weight of the farmed animals; obtain the end-effector pose of the multi-axis robotic arm based on the position of the farmed animals; obtain the joint angles of the multi-axis robotic arm based on the end-effector pose of the multi-axis robotic arm; drive the multi-axis robotic arm to move and move the gripper to the farmed animals based on the joint angles of the multi-axis robotic arm; and trigger the gripper to grasp the farmed animals based on the obtained gripping force.

[0080] Preferably, the controller is used to restore the left and right views according to the following steps, including:

[0081]

[0082] Among them, the left and right views before restoration are denoted as the original images, J c (x) represents the pixel value of pixel x in the c channel of the restored image. c (x) represents the pixel value of pixel x in channel c of the original image, A cLet c be the background light intensity of channel c, where c∈(R,G,B), and R, G, and B are the red, green, and blue channels in the RGB color channel. Let be the transmittance of channel c at pixel x;

[0083] The process for determining the background light intensity is as follows:

[0084] Arrange the pixel values ​​of the dark channel image of the fused fog lines in ascending order, and select the top 10% of pixels. Then, among these pixels in the original image, find the pixel with the lowest pixel value in each of the red, green, and blue channels. Use the lowest pixel value in each of the red, green, and blue channels as the background light value A for the red, green, and blue channels. R A G A B ;

[0085] The process of calculating the transmittance of channel c at pixel x is as follows:

[0086] Construct the minimization function:

[0087]

[0088] Where δ is the balance coefficient, N x Let x be a local region centered at pixel x, and y be N. x Pixels within; Let be the maximum transmittance of the red channel at pixel x; σ(x) is the value obtained from the fog line. Standard deviation; Let x be the transmittance of the red channel at pixel x. I represents the transmittance of the red channel at pixel y; R (x), I R (y) represents the pixel values ​​of pixel x and pixel y in the red channel of the original image, respectively;

[0089] The calculation formula is as follows:

[0090]

[0091] Among them, t R (x) represents the initial transmittance of the red channel at pixel x; t Rm This represents the lower threshold of the transmittance of the red channel at pixel x.

[0092] t Rm The calculation formula is as follows:

[0093]

[0094] Among them, I G (x), I B(x) represents the pixel value of pixel x in the green and blue channels of the original image, respectively; β RG β represents the ratio of the attenuation coefficients of red light to green light. RB This represents the ratio of the attenuation coefficients of red light to blue light.

[0095] Set t R (x), obtained by iteratively solving the minimization function. The transmittance of the green channel at pixel x is then calculated using the following formula. Transmittance of the blue channel at pixel x

[0096]

[0097] Where, β R β G β B These are the attenuation coefficients of red, green, and blue light in water, respectively.

[0098] Preferably, the controller is used to perform stereo matching on the target frame area of ​​the farmed organism to obtain the parallax of the left view by following steps, including:

[0099] Based on the target bounding box region of the cultured animals in the left view, update the target bounding box region of the cultured animals in the right view according to the intersection-union ratio of the target bounding box regions of the cultured animals in the left and right views;

[0100] Calculate the matching cost of pixels in the target bounding box region of the left view, including,

[0101] C(p,d)=α*β*ρ(C census (p,d),λ ce )+(1-β)*ρ(C AD (p,d),λ AD )+(1-α)*ρ(C grad (p,d),λ grad )

[0102] Where C(p,d) is the matching cost of pixel p in the target box region of the left view.

[0103] d is any value among the pixel differences between pixel p and all pixels in the target area of ​​the right view except pixel p.

[0104] C census (p,d), C AD (p,d), C grad (p, d) represent the cost of pixel p in the target bounding box region of the left view calculated by the census algorithm, AD algorithm, and gradient algorithm, respectively; λ ce , λ AD , λgrad These represent the control parameters for the census algorithm, AD algorithm, and gradient algorithm, respectively.

[0105] α is the adaptive weight parameter, and β is the fixed weight.

[0106] ρ(C census (p,d),λ ce ), ρ(C AD (p,d),λ AD ), ρ(C grad (p,d),λ grad The calculation formulas for ) are as follows:

[0107]

[0108]

[0109]

[0110] Among them, C grad The formula for calculating (p,d) is as follows:

[0111] C grad (p,d)=max(0,L(p R )-L max (p R ),L min (p L )-L(p L ))

[0112] Wherein, L(p L L(p) represents the grayscale value of the left view pixel. R ) represents the pixel grayscale value of the right view, L max (p R ) represents the maximum grayscale value of the target area in the right view, L min (p L L(p) represents the minimum grayscale value of the target area in the left view. L ), L(p R The calculation formula for ) is as follows:

[0113]

[0114]

[0115] Where a and b are weight values, It is the normalized value of the grayscale value of the pixel point p in the m and n directions of the target box region in the left view. It is the normalized value of the grayscale value of the pixel point p in the m and n directions of the target box area in the right view;

[0116] Calculate the matching cost for each pixel in the target box region of the left view to obtain the cost matrix of the target box region of the left view;

[0117] Calculate the optimal arm length τ, aggregate the cost matrix of the target box region in the left view, and obtain the disparity of the target box region in the left view;

[0118] The formula for calculating the optimal arm length τ is as follows:

[0119]

[0120]

[0121] Where τ0 is the initial value of the preset pixel threshold, τ1 is the cutoff value of the preset pixel threshold, L is the preset arm length threshold, and l pq I represents the pixel difference between pixel p and pixel q on the horizontal support arm. p ,I q ,I q′ These are the pixel values ​​of three pixels, p, q, and q′, respectively; q′ is the pixel preceding pixel q on the horizontal support arm.

[0122] The beneficial effects of this invention are:

[0123] This invention can automatically acquire optical binocular images of underwater scenes, optimize the estimation methods for image background light and transmittance, preprocess underwater imaging to improve the clarity of underwater images, apply the convolutional neural network algorithm widely used in deep learning to the underwater detection environment in nearshore areas, and provide a more suitable and efficient model for underwater target detection. It can also determine the distance and size of various underwater aquaculture targets and implement capture, thereby improving the efficiency and intelligence level of aquaculture harvesting. Attached Figure Description

[0124] Figure 1 Flowchart of the underwater aquaculture detection and capture method provided by the present invention;

[0125] Figure 2 This is a diagram of the network structure for detecting aquatic organisms provided by the present invention;

[0126] Figure 3 Add a mechanism diagram to the CBAM attention mechanism provided by this invention;

[0127] Figure 4 A structural diagram of the underwater aquaculture detection and grasping device provided by the present invention;

[0128] The components include: 1. gripper; 2. multi-axis robotic arm; 3. storage tank; 4. light source; 5. camera; and 6. underwater drive. Detailed Implementation

[0129] The present invention will be further described below with reference to the embodiments. The following embodiments are only used to more clearly illustrate the technical solution of the present invention, and should not be used to limit the scope of protection of the present invention.

[0130] Example 1

[0131] A method for detecting and capturing underwater aquaculture organisms, see [link to relevant documentation]. Figure 1 The specific implementation steps are as follows:

[0132] I. Binocular Camera Calibration and Image Correction

[0133] Before using the detection and grasping device for the first time, two visible light cameras (referred to as the left camera and the right camera) installed in a parallel binocular configuration with a baseline distance of 6-20cm need to be calibrated to obtain the camera's internal and external parameters.

[0134] The device was placed in water, and more than 20 underwater calibration board images were obtained using a checkerboard calibration board with 12×9 grids and individual grids measuring 3cm×3cm. The checkerboard calibration board was placed within the field of view of the left and right cameras, and the board was rotated at four angles (up, down, left, and right) while being fixed in place for shooting.

[0135] The corner points of the multiple sets of left and right views are extracted in two steps: first, the outer rectangle containing all corner points is manually selected for each calibration board image, i.e., four rectangular corner points; then all other corner points are automatically obtained to avoid corner points appearing outside the calibration board in the fully automatic detection results, which would lead to calibration errors.

[0136] The maximum likelihood function constructed by Zhang Zhengyou's calibration method for solving the camera's intrinsic and extrinsic parameters is as follows:

[0137]

[0138] Where, θ ij Let x represent the coordinates of the i-th corner point of the j-th calibration map, where j = 1, 2, ..., x, and x is the total number of calibration maps. This represents the total number of corner points in a single calibration map. For θ ij The estimated value of R; ij and T ij M represents the rotation and translation vectors of the i-th corner point of the j-th calibration image, respectively; j Let M be the three-dimensional coordinate matrix of the corner points in the j-th calibration image; D = [k1, k2, σ1, σ2, k3] is the distortion parameter vector of the non-ideal lens model, and M is the three-dimensional coordinate matrix of the corner points in the j-th calibration image. I This is the camera's internal parameter matrix.

[0139] Because distortion exists and is difficult to completely eliminate, the selection of feature points (i.e., checkerboard corner points) on the calibration board also has errors, thus affecting the accuracy of intrinsic parameter calculation, manifested as an increase in reprojection error. This invention utilizes the reprojection error to construct a cost function; minimizing this cost function can optimize the camera's projection matrix. Considering the impact of lens distortion on intrinsic parameter accuracy, a reprojection error threshold ξ is set. th The reprojection error is greater than ξ. th The calibration plate image is recalculated, and the geometric transformation error is introduced as a cost function into the likelihood function:

[0140]

[0141] Where Z is the plane homography matrix, Let ξ be the estimated value of Z. Set distortion coefficients (these distortion coefficients are only set to optimize the reprojection error and are not used as the final image distortion coefficients; their adjustment range is (-1, 1)). For each image where the average reprojection error is greater than ξ... th The calibration diagram is re-solved for M. I And recalculate the reprojection error ξ, ξ≤ξ th M at that time I This is the corrected camera intrinsic parameter matrix. The matrix R, composed of the rotation vectors of the corner points in the calibration map, and the matrix T, composed of the translation vectors, are the camera extrinsic parameters.

[0142] Using the obtained camera intrinsic and extrinsic parameters and Bouguet's camera correction theory, coplanar correction and row alignment correction are performed on the left and right image pairs to minimize the reprojection distortion of the pixels, thereby obtaining the corrected image, which is then used as the input image for subsequent image processing.

[0143] II. Image Acquisition

[0144] The detection and grasping device is installed on the underwater robot to capture underwater video in real time. The left and right cameras capture 6 frames of images per second, which are recorded as the left view and the right view, respectively. The left and right views are coplanar and row alignment corrected by the camera intrinsic and extrinsic parameters obtained in step one and the Bouguet camera correction theory (stereo matching camera correction theory), and then marked as the left and right views to be restored.

[0145] III. Image Restoration

[0146] In underwater optical imaging, image quality is often poor due to light scattering in the water, absorption by suspended particles, and insufficient underwater illumination. Common problems include image noise, low contrast, blurred target details and edges, and color distortion. To obtain more accurate target detection and stereo matching results, the left and right views to be restored need to be restored separately to obtain images with accurate color information and clear target details and outlines. The original images described in this section refer to all corrected left and right views to be restored.

[0147] Based on the underwater imaging model and dark channel prior (DCP) dehazing algorithm proposed by Jaffe-McGlamery, assuming that the water transmittance is determined only by the distance between the aquaculture target and the camera, and that the scattering and absorption coefficients of light in the water are constant, the underwater imaging model can be written as:

[0148] I c (x)=J c (x)t(x)+A c (1-t c (x)) (3)

[0149] Among them, J c (x) represents the pixel value of pixel x in the c channel of the restored image. c (x) represents the pixel value of pixel x in channel c of the original image, A c Let t be the background light intensity of channel c, where c∈(R,G,B), and R, G, and B are the red, green, and blue channels in the RGB color channel. c (x) represents the transmittance of channel c at pixel x.

[0150] After determining the background light intensity and water transmittance, the image can be restored.

[0151] For background light, a background light estimation method based on fog line theory is adopted. Since the degree of fog in different regions of the image is inconsistent, an image may contain multiple fog lines; multiple regions may belong to the same fog line, and all fog lines can be represented by the following formula:

[0152] H(x)=(1-I R (x))-max(I B (x),I G (x)) (4)

[0153] Where H(x) represents the intensity of the fog line at pixel x, I R (x), I B (x), I G (x) represents the pixel value of pixel x in the R, B, and G channels of the original image, respectively.

[0154] When the target is at infinity, the red channel decays to 0, and the pixel value at this point is the fog line value. The pixel values ​​of the original image (dark channel image with fog lines blended) with the red channel decayed to 0 are sorted from smallest to largest. The top 10% of pixels (usually pixels in areas with a large depth of field) are selected. From these pixels in the original image, the pixel with the lowest value in each color space is found, and its pixel value is used as the background light value A for that channel. This yields the three-channel background light value A. R A G A B The background light value of the image should not be greater than 0.1H(x).

[0155] Because different water bodies have different environments, the dissolved substances, suspended solids, bottom sediment, and aquatic organisms in the water vary, resulting in different absorption and scattering coefficients for light. In seawater, red light has the greatest absorption while blue light has the least. Considering the color channel, and based on the different attenuations of light of different wavelengths in different water bodies, to reduce the error in transmittance calculation caused by the error in the attenuation coefficient, an attenuation comparison formula is introduced, modifying equation (3) as follows:

[0156]

[0157] In the formula, t R (x) represents the water transmittance of the red channel, β RG The ratio of the attenuation coefficients of red light to green light. β RB The ratio of the attenuation coefficients of red light to blue light. β R β G β B J represents the attenuation coefficients of red, green, and blue light in water, respectively. R (x), J B (x), J G (x) represents the pixel value of pixel x in the R, G, and B channels of the restored image, respectively.

[0158] Existing techniques employ a method of clustering pixel values ​​to find unfuzzy clusters to solve for transmittance: first, the k-means algorithm is used to cluster the RGB channel pixel values ​​of the image into multiple fuzzy clusters; then, each pixel in the image is replaced by the center of each cluster. Assuming that at least one pixel in each cluster is an unfuzzy pixel, the initial transmittance of the RGB channel at pixel x is obtained.

[0159] However, in underwater images, the attenuation coefficients of the RGB channels are inconsistent, and it may be impossible to find unblurred pixels within each blurred cluster. Therefore, this invention uses the red channel as a benchmark, considers the attenuation coefficients of different color channels, and defines a lower limit threshold t for the transmittance of the red channel at pixel x. Rm :

[0160]

[0161] Simultaneously, based on the characteristic that red light attenuates faster with increasing distance, 1-t R (x) is used as the attenuation constraint threshold to obtain the maximum transmittance of the red channel at pixel x:

[0162]

[0163] Furthermore, considering that existing fuzzy clustering methods target global pixels in an image, if a certain pixel class has few pixels and significant noise, the restoration effect of that pixel class will be greatly affected. To improve this situation, the optimal red channel transmittance at pixel x is obtained. Introducing a smoothing term to construct a minimization function:

[0164]

[0165] Where δ is a data item and smoothing terms The balance coefficient, N x Let x be a local region centered at pixel x, and y be N. x The pixels within; σ(x) is calculated based on the fog line. Standard deviation; I represents the transmittance of the red channel at pixel y; R (x), I R (y) represents the pixel values ​​of pixel x and pixel y in the red channel of the original image, respectively;

[0166] The transmittance of the R channel at pixel x is determined using existing methods. And use it as the initial transmittance t of the red channel at pixel x. R Substituting the initial value of (x) into formulas 6-8, we iterate through formula 8 to find the minimum value, thus obtaining the transmittance of the red channel at pixel x. Substitute the following formula to restore the underwater image:

[0167]

[0168] in Let x be the transmittance of the green channel at pixel x. Let x be the transmittance of the blue channel at pixel x.

[0169] The left and right views described below are the corresponding images after the images have been dehazed.

[0170] IV. Target Detection of Underwater Aquaculture

[0171] The detection of aquatic aquaculture targets requires high real-time performance. Therefore, this invention uses the YOLOv5s (a 5s version of YouOnly Look Once) model as the base network and adds an attention detection mechanism and upsampling method to improve the accuracy of small target detection, enabling the detection of various aquaculture targets such as sea cucumbers, sea urchins, starfish, and scallops.

[0172] The YOLOv5s network structure is mainly divided into four parts: input, backbone, neck (feature extraction), and prediction. The input uses Mosaic data augmentation to enrich the background of the detected target. The backbone uses CSPNet (Cross Stage Partial Network) with module replacements and Focus (special downsampling) as the baseline network. The neck uses FPN (Feature Pyramid Network) + PAN (Path Aggregation Network) for feature fusion. The prediction part uses YOLO detection prediction (YOLO detection prediction in parallel), outputting three different resolutions for multi-scale prediction.

[0173] (1) Model training phase

[0174] To adapt to the target size of farmed animals, Mosaic16 (4*4 random cropping and splicing) augmentation is introduced instead of Mosaic augmentation for data augmentation at the input end: 16 random images are read from the training set, and operations such as flipping, scaling, and color gamut changes are performed. Then, 1 / 16 parts of each of the 16 images are cut and spliced ​​together. These parts all contain the target to be detected, which can increase the target density and speed up the training. It is suitable for multi-target detection learning in video frames.

[0175] (2) Actual testing stage

[0176] Aquaculture detection network structure such as Figure 2 As shown, its implementation process includes multiple parts.

[0177] a) Backbone backbone network

[0178] The adaptively scaled left and right views (hereinafter collectively referred to as feature maps) are input into the slicing layer. The slicing layer performs block slicing operations on the feature maps and then connects them. The sampling layer C3 module is modified into a C3CBAM module (Convolutional Block Attention Module, a concentrated convolutional block combined with an attention mechanism). Subsequently, convolution and the modified sampling operation C3CBAM are alternately performed. This feature map is then input into the RFB (Receptive Field Block) module, which replaces the original SPP (Spatial Pyramid Pooling) module, to extract deep information. In this process, feature maps of different scales are fed into the Neck network.

[0179] The C3CBAM module refers to the module built upon the C3 module. Figure 3 In addition to the attention mechanism, a CBAM attention module is added to adjust the model's focus in the channel and spatial dimensions. This means that the attention detection mechanism, built upon the characteristics of human vision, pays more attention to the detection of small objects, increasing the machine vision receptive field and covering more parts of the object to be identified.

[0180] The RFB module refers to the mechanism that simulates the receptive field of the human eye and matches the binocular vision. At the same time, it enhances the feature extraction capability of the network by passing the input feature map through convolutional layers composed of multiple different convolutional kernels and then concatenating them to obtain a feature image with greater visual perception capability.

[0181] b) Neck Feature Extraction

[0182] Construct feature pyramids for FPN and PAN. FPN receives feature images of different scales at different levels, and then... Figure 2 The bilinear interpolation and C3CBAM modules shown are concatenated to obtain a high-level semantic feature map. PAN then performs a simplified reverse operation of this process to obtain localization information. Similar to the baseline network, during processing, information at different scales is fed into the Head network, and an additional CBAM attention mechanism is added on top of the smallest scale information during this process.

[0183] Bilinear interpolation refers to establishing a Cartesian coordinate system MN in the image, with the image center as the origin, and performing linear interpolation once in the m and n directions respectively, defining the four endpoints of the image: Q. 11 = (m1, n1), Q 12 = (m1, n2), Q 21 = (m2, n1), Q 22 = (m2, n2), we get:

[0184]

[0185] Where f(m,n) is the pixel value at sampling point (m,n), and f(Q) is the pixel value at sampling point (m,n). 11 f(Q) 21 f(Q) 12 f(Q) 22 The values ​​are the pixel values ​​at the four endpoints, and the sampling point is any pixel in the image.

[0186] The CBAM module added separately at the bottom layer is the existing attention mechanism module. It is a redistribution of attention weights after the feature map has passed through the C3CBAM module, in order to better match the feature detection capability at low scale with the small target of farmed animals. Its effect is similar to that of the C3CBAM module.

[0187] c) Head Prediction Network

[0188] The anchor-based decoupled detection head performs target detection on different scales of information fed from the Neck network. The decoupled detection head accelerates network convergence and improves accuracy by reducing some feature channels and adding two parallel branches. This process can be described as follows:

[0189] Step 1: Perform forward inference on the feature map, extract the features of the target object, provide an initial prediction box, and provide the category confidence score;

[0190] Step 2: Execute the Non-Maximum Suppression (NMS) algorithm to remove redundant bounding boxes. The execution logic is as follows: obtain information on all bounding boxes under the current target category, sort the information from highest to lowest confidence level, record the current maximum confidence level, calculate the intersection-union ratio (IUR) of the maximum confidence level with all remaining values, and remove bounding boxes with IUR values ​​greater than a preset threshold. Then, repeat the first two steps (sorting information from highest to lowest confidence level, recording the current maximum confidence level, calculating the IUR of the maximum confidence level with all remaining values, and removing bounding boxes with IUR values ​​greater than a preset threshold) until all bounding boxes meet the requirements.

[0191] Step 3: Traverse all predicted bounding boxes of target objects in the detection result list, record the predicted bounding boxes of target objects as the target bounding boxes of target objects, and mark the target category, target number and quantity information.

[0192] V. Obtaining parallax through stereo matching of the target area

[0193] Align the detection results of the left and right views containing target category, target number and quantity information, and compare the detection boxes of targets with the same number in the left and right views in turn. Using the left view box as a reference, perform stereo matching on the selected area to reduce the amount of stereo matching calculations for the whole image.

[0194] The comparison method is as follows: The left view is selected as the baseline image. The intersection-union ratio (IUU) of the selected areas in the left and right views is detected. The higher the degree of overlap between the target boxes in the right and left views, the higher the value (excluding values ​​1 and 0). If no relevant target box exists in the left view, it is counted as 1, and the corresponding information in the right view is deleted; this box is not used for stereo matching. If no relevant target box exists in the right view, it is counted as 0, and the relevant information in the left view is directly considered as the result of the formal stereo matching, meaning that stereo matching for this set of views is complete. A threshold of 0.98 is used. If the value is less than this value but not zero, the relevant information of the target box corresponding to the right view is deleted, and a new target box is generated from the overlapping area for stereo matching. If the value is greater than this threshold, no further operation is performed, and the stereo matching process begins.

[0195] To address the characteristics of underwater targets—small size, minimal color difference, and partial overlap—AD-Census employs a horizontal arm construction method to perform stereo matching on the target bounding box regions in the left and right views. The stereo matching process is as follows:

[0196] Step 1: Calculate the matching cost for the target bounding box regions in the left and right views, measuring the correlation between candidate pixels and matching pixels. Based on the matching cost calculation formula in AD-Census, a gradient matching algorithm is integrated to improve anti-interference capabilities and constrain the smoothness of image information. The gradient matching algorithm can be used to stabilize image noise, prevent excessively smooth edge regions, and improve matching accuracy. Considering the directionality of the gradient, a Cartesian coordinate system MN is established, with the horizontal leftward direction defined as m and the vertical upward direction as n, and their intersection point as 0. Weights are then added to the grayscale values ​​in the m and n directions and normalized to obtain more numerically reasonable grayscale pixel values. The final matching cost of the gradient transformation is as follows:

[0197] C grad (p,d)=max(0,L(p R )-L max (p R ),L min (p L )-L(p L (11)

[0198] Where p is a pixel in the target area of ​​the left view (hereinafter referred to as the left view), d is any value among the pixel differences between pixel p and all pixels in the target area of ​​the right view except for pixel p, and L(p L L(p) represents the grayscale value of the left view pixel. R ) represents the pixel grayscale value of the right view, L max (p R ) represents the maximum grayscale pixel value of the target bounding box area in the right view, L min (p L L(p) represents the minimum grayscale pixel value of the target bounding box area in the left view; L), L(p R The calculation formula for ) is as follows:

[0199]

[0200]

[0201] Where a and b are user-defined weight values. It is the normalized value of the grayscale value of the pixel point p in the m and n directions of the target box region in the left view. It is the normalized value of the grayscale value of the pixel point p in the m and n directions of the target box area in the right view.

[0202] Thus, the final formula for calculating the matching cost is obtained:

[0203] C(p,d)=α*β*ρ(C census (p,d),λ ce )+(1-β)*ρ(C AD (p,d),λ AD )+(1-α)*ρ(C grad (p,d),λ grad (14)

[0204] Where C(p,d) is the matching cost of pixel p in the target box region of the left view, C census (p,d), C AD (p,d), C grad (p,d) represent the cost of pixel p in the left view target bounding box region calculated by the census algorithm, AD algorithm, and gradient algorithm, respectively; λ ce λ represents the control parameters of the census algorithm. AD λ represents the control parameters of the AD algorithm. grad These represent the control parameters of the gradient algorithm;

[0205]

[0206]

[0207]

[0208] Among them, when C census (p,d) and λ ce When both are positive, ρ(C) census (p,d),λ ce The value range of this function is [0,1], and its cost is C. census The larger (p,d) is, the larger the function value is, ρ(C AD (p,d),λ AD ), ρ(Cgrad (p,d),λ grad Similarly, this function can be used to normalize any value to the range [0,1]. An adaptive weighting parameter is defined by a fixed weight β, a correction parameter ε, and a control parameter γ for the adaptive weighting parameter. L The shortest arm length L of pixel p in the four directions (up, down, left, right) from the boundary of the target selection area. min L is used to control this, where ι is the cutoff value, i.e., the minimum value of α. That is, when the pixel with pixel value p is close to the center of the image, L... min The time will be longer, so the weights of AD and gradient cost are smaller, while the weight of census is larger, in order to control the cost and make full use of the texture information of the image.

[0209] Equation (14) calculates the matching cost for each pixel in the target bounding box selection area in the left and right views. The cost of all pixels will generate a set of cost matrices for the left and right target bounding boxes.

[0210] Step 2: Use adaptive nonlinear constraints based on the intensity difference between pixels to construct a support window and obtain the optimal arm length τ:

[0211]

[0212]

[0213] Where τ0 is the initial value of the preset pixel threshold, τ1 is the cutoff value of the preset pixel threshold, and L is the preset arm length threshold. pq I represents the pixel difference between pixel p and pixel q on the horizontal support arm. p ,I q ,I q′ These are the pixel values ​​of three pixels, p, q, and q′, respectively; q′ is the pixel preceding pixel q on the horizontal support arm.

[0214] After constructing the optimal arm length τ, consistent with traditional methods, a four-round aggregation operation is performed on the generated cost matrix to obtain the disparity of the target bounding box region of the farmed animals in the left and right views. In the first and third rounds, the cumulative cost values ​​of all pixels in the horizontal arm are stored first, then accumulated along the vertical arm direction, and finally averaged. In the other two rounds, the accumulation order is switched.

[0215] This aggregation method utilizes two supportable regions, significantly reducing matching errors in disparity discontinuities and resulting in a smoother disparity map, thereby improving the accuracy of textureless and disparity discontinuities.

[0216] Step 3: Perform scanline optimization on the parallax of the target bounding box area of ​​the farmed animals in the left and right views to further improve the accuracy of parallax and reduce matching errors. Optimization is performed in four directions: up, down, left, and right, followed by an arithmetic average.

[0217] Step 4: Perform disparity optimization sequentially as follows to obtain the optimal disparity map:

[0218] (1) Outlier Detection: Perform left-right consistency detection to detect pixels with erroneous disparity in the disparity map and classify them into two categories: occluded pixels and mismatched pixels.

[0219] (2) Iterative Region Voting: This step is used to remove some mismatched pixels.

[0220] (3) Proper Interpolation: For pixels with disparity errors, a search is performed in 16 directions. The minimum disparity is selected as the value of the occluded pixel, and the disparity of the pixel with the shortest color distance is selected as the disparity of the mismatched pixel, in order to eliminate outliers and improve the disparity error rate.

[0221] (4) Depth Discontinuity Adjustment: Further optimizes the disparity value in the discontinuous region by fine-tuning the pixel values ​​on the edge and selecting the disparity value on the left and right sides that has the lower cost, which can significantly reduce the disparity error in the discontinuous region.

[0222] (5) Sub-pixel Enhancement: Further optimize the disparity value.

[0223] VI. Binocular Measurement

[0224] Based on the disparity map in the left view and the calibration parameters of the two cameras, the position information of the target object in the world coordinate system is calculated; the distance between the target object and the camera is measured; and the dimensions of the target object are calculated. The specific steps are as follows:

[0225] Step 1: Using the left view as the reference image, calculate the world coordinates of the target in the reference image based on the calibrated parameters of the two visible light cameras. Xw and Yw represent the world coordinates of the pixel on the horizontal and vertical axes of the world coordinate system, respectively, and Zw represents the depth of the pixel. The world coordinates of the target object are expressed as:

[0226]

[0227] Where k is the camera's focal length, b is the baseline distance, and k and b are the camera intrinsic parameters obtained during the dual-target localization process. Let represent the disparity of a pixel in the left view, and m and n represent the horizontal and vertical coordinates of the pixels in the target bounding box region of the left view, respectively.

[0228] Step 2: Calculate the distance between the binocular camera and the target object, as well as the size of the target object, based on the parallax of the target object.

[0229] Based on the target bounding box regions obtained in the target detection, the connected regions within all target bounding box regions are obtained, and the connected regions are labeled with targets T1,…,T in order from top to bottom and from left to right. N Create a tiltable bounding box, i.e., an outer rectangle, for each connected region and display it in the results window.

[0230] Based on the disparity map of the reference image obtained during target localization, histogram statistics are performed on the disparity values ​​of pixels in each connected region to obtain the disparity values ​​at the maximum peak and the nearest trough. and For disparity maps The system performs connectivity checks on the pixels within the specified range, compares the number of pixels in each connected region, and creates a minimum bounding rectangle R for the connected region with the highest number of pixels. N The minimum bounding rectangle R N The minimum bounding rectangle of the farmed organism is used as the basis for calculating the width and height of the target. The point with the maximum disparity of a pixel in each connected region is the closest point, and the depth value of that pixel is recorded as the target distance. The point with the minimum disparity is the farthest point, and the difference between the two is the maximum possible length of each target.

[0231] The formulas for calculating the length, width, height, and distance of the target object are as follows:

[0232]

[0233] in, The minimum bounding rectangle R of the cultured organism is given by [the following]. N Maximum and minimum parallax within; r1 l r2 The minimum bounding rectangle R of the cultured organism is given by [the following]. N The side length.

[0234] When the distance and size of the target object meet the preset grasping requirements, proceed to the next grasping step.

[0235] VII. Capture Control

[0236] The main body of the grasping system is a robotic arm, with a gripper device installed at the end of the robotic arm. It grasps the target object by screening its features.

[0237] Step 1: Design of the underwater robot's grasping device. Autonomous grasping by the underwater robot requires addressing the challenges posed by the underwater operating environment. Considering the corrosiveness of seawater, corrosion-resistant industrial aluminum was selected to construct the robot's main frame. To overcome the effects of gravity during underwater movement, acrylic chambers and buoyancy blocks were used to provide buoyancy and maintain the robot's balance. Zero-buoyancy cables were used for power supply and signal communication with the surface.

[0238] The gripper is mounted at the end of the robotic arm to perform the grasping function. Considering ease of installation and use, simple structure, and precision, a pneumatic gripper is selected. The selection of the pneumatic gripper is calculated based on the grasping requirements. The gripping force is calculated using the following formula:

[0239]

[0240] The target object's mass ψ and gravitational acceleration g = 9.81 m / s² are selected. 2 With a safety factor S = 3 and a friction coefficient μ = 0.2, the clamping force of the gripper is calculated to be F = 29.43 N. N is a unit of mechanical strength; 1 N = 1 kg·m / s². 2 .

[0241] Step 2: Robot kinematics modeling. A six-axis robotic arm is selected as the robot body. The robot is controlled to descend to a suitable depth. Based on the position of the farmed animal, the pose of the six-axis robotic arm's end-effector coordinate system is solved. Based on the robot's end-effector pose, the joint angles of the robotic arm are calculated. The gripper is driven to the corresponding position. Based on the calculated gripper force, the gripper is triggered to grab the farmed animal and place it in the storage compartment behind it.

[0242] For scenarios with multiple objects, target categories are filtered. For similar targets, the feature points of farmed animals are identified and compared in step four. Farmed animals with more feature points are prioritized for capture. When the number of feature points of farmed animals is close, the target distance is determined, and targets closer to the robot's end effector are prioritized for capture.

[0243] Example 2

[0244] An underwater aquaculture organism detection and grasping device, employing the underwater aquaculture organism detection and grasping method described in Example 1, see [link to example]. Figure 4 It includes an underwater drive 6, two cameras 5, a light source 4, a power supply, a controller, a multi-axis robotic arm 2, a gripper 1, and a storage tank 3;

[0245] Two cameras 5, a light source 4, a power supply, a controller, a multi-axis robotic arm 2, a storage tank 3, and other components are mounted on an underwater actuator 6, which drives these components to move over a wide range underwater. The underwater actuator 6 can be selected from existing mature underwater robots.

[0246] Two visible light cameras 5 are mounted on the underwater actuator 6 in a parallel binocular configuration with a baseline distance of 6-20cm. They are used to acquire images of underwater aquaculture organisms, which are recorded as the left view and the right view, respectively, and uploaded to the controller.

[0247] Light source 4 includes two light sources, such as LED light sources, which are installed near the two cameras 5 to improve underwater brightness and improve image quality.

[0248] The power supply is installed inside the underwater drive 6 and is used to provide power to the underwater drive 6, camera 5, light source 4, controller, multi-axis robotic arm 2, gripper 1, and storage tank 3.

[0249] The controller is mounted on the underwater drive 6, preferably internally, and is electrically connected to the underwater drive 6, two cameras 5, a light source 4, a power supply, a multi-axis robotic arm 2, a gripper 1, and a storage tank 3. Its functions include: reconstructing the left and right views; detecting the aquatic organisms in the reconstructed left and right views to obtain the target frame of the aquatic organisms; performing stereo matching on the target frame area of ​​the aquatic organisms to obtain the parallax of the left view; calculating the size and position of the aquatic organisms based on the parallax of the left view; and calculating the weight of the aquatic organisms based on their size. The gripping force of gripper 1 is calculated based on the weight of the farmed animal; the end-effector pose of multi-axis robotic arm 2 is determined based on the position of the farmed animal; the joint angles of multi-axis robotic arm 2 are determined based on the end-effector pose of multi-axis robotic arm 2; the multi-axis robotic arm 2 is driven to move based on the joint angles of multi-axis robotic arm 2, thereby moving gripper 1 to the farmed animal; based on the determined gripping force of gripper 1, gripper 1 is triggered to grab the farmed animal; after grabbing the farmed animal, multi-axis robotic arm 2 is controlled to move gripper 1 to the opening of storage compartment 3, storage compartment 3 is opened, and gripper 1 is triggered to drop the farmed animal into storage compartment 3.

[0250] The multi-axis robotic arm 2 is mounted on the underwater actuator 6. It moves according to the control signals obtained by the controller for each joint angle of the multi-axis robotic arm 2, and drives the gripper 1 to the aquaculture site. After grabbing the aquaculture site, it moves the gripper 1 to the opening of the storage tank 3 according to the controller's instructions.

[0251] The gripper 1 is installed at the front end of the multi-axis robotic arm 2 and grabs the farmed animals and throws them into the storage compartment 3 according to the trigger command issued by the controller.

[0252] The storage tank 3 is mounted on the underwater drive 6, close to the multi-axis robotic arm 2, and is used to store the captured aquatic organisms and to open and close according to the instructions of the controller.

[0253] The outer surfaces of the underwater drive, two cameras, light source, power supply, controller, multi-axis robotic arm, grippers, storage tank, etc., need to be waterproofed and corrosion-resistant.

[0254] The controller is used to restore the left and right views by following these steps:

[0255]

[0256] Among them, the left and right views before restoration are denoted as the original images, J c (x) represents the pixel value of pixel x in the c channel of the restored image. c (x) represents the pixel value of pixel x in channel c of the original image, A c Let c be the background light intensity of channel c, where c∈(R,G,B), and R, G, and B are the red, green, and blue channels in the RGB color channel. Let be the transmittance of channel c at pixel x;

[0257] The process for determining the background light intensity is as follows:

[0258] Arrange the pixel values ​​of the pixels in the dark channel image of the fused fog lines in ascending order, and select the top 10% of pixels. Then, among these pixels in the original images of the left and right views, find the pixel with the lowest pixel value in each of the red, green, and blue channels. Use the lowest pixel value in each of the red, green, and blue channels as the background light value A for the red, green, and blue channels. R A G A B ;

[0259] The process of calculating the transmittance of channel c at pixel x is as follows:

[0260] Construct the minimization function:

[0261]

[0262] Where δ is the balance coefficient, N x Let x be a local region centered at pixel x, and y be N. x Pixels within; Let be the maximum transmittance of the red channel at pixel x; σ(x) is the value obtained from the fog line. Standard deviation; Let x be the transmittance of the red channel at pixel x. I represents the transmittance of the red channel at pixel y; R (x), I R (y) represents the pixel values ​​of pixel x and pixel y in the red channel of the original image, respectively;

[0263] The calculation formula is as follows:

[0264]

[0265] Among them, tR (x) represents the initial transmittance of the red channel at pixel x; t Rm This represents the lower threshold of the transmittance of the red channel at pixel x.

[0266] t Rm The calculation formula is as follows:

[0267]

[0268] Among them, I G (x), I B (x) represents the pixel value of pixel x in the green and blue channels of the original image, respectively; β RG β represents the ratio of the attenuation coefficients of red light to green light. RB This represents the ratio of the attenuation coefficients of red light to blue light.

[0269] Set t R (x), obtained by iteratively solving the minimization function. The transmittance of the green channel at pixel x is then calculated using the following formula. Transmittance of the blue channel at pixel x

[0270]

[0271] Where, β R β G β B These are the attenuation coefficients of red, green, and blue light in water, respectively.

[0272] The controller is used to perform stereo matching on the target frame area of ​​the farmed organism to obtain the parallax of the left view by following the steps below, including:

[0273] Based on the target bounding box region of the cultured animals in the left view, update the target bounding box region of the cultured animals in the right view according to the intersection-union ratio of the target bounding box regions of the cultured animals in the left and right views;

[0274] Calculate the matching cost of pixels in the target bounding box region of the left view, including,

[0275] C(p,d)=α*β*ρ(C census (p,d),λ ce )+(1-β)*ρ(C AD (p,d),λ AD )+(1-α)*ρ(C grad (p,d),λ grad )

[0276] Where C(p,d) is the matching cost of pixel p in the target box region of the left view.

[0277] d is any value among the pixel differences between pixel p and all pixels in the target area of ​​the right view except pixel p.

[0278] C census (p,d), C AD (p,d), C grad (p, d) represent the cost of pixel p in the target bounding box region of the left view calculated by the census algorithm, AD algorithm, and gradient algorithm, respectively; λ ce , λ AD , λ grad These represent the control parameters for the census algorithm, AD algorithm, and gradient algorithm, respectively.

[0279] α is the adaptive weight parameter, and β is the fixed weight.

[0280] ρ(C census (p,d),λ ce ), ρ(C AD (p,d),λ AD ), ρ(C grad (p,d),λ grad The calculation formulas for ) are as follows:

[0281]

[0282]

[0283]

[0284] Among them, C grad The formula for calculating (p,d) is as follows:

[0285] C grad (p,d)=max(0,L(p L )-L max (p R ),L min (p R )-L(p L ))

[0286] Wherein, L(p L L(p) represents the grayscale value of the left view pixel. R ) represents the pixel grayscale value of the right view, L max (p R ) represents the maximum grayscale value of the target area in the right view, L min (p L L(p) represents the minimum grayscale value of the target area in the left view. L ), L(p R The calculation formula for ) is as follows:

[0287]

[0288]

[0289] Where a and b are weight values, It is the normalized value of the grayscale value of the pixel point p in the m and n directions of the target box region in the left view. It is the normalized value of the grayscale value of the pixel point p in the m and n directions of the target box area in the right view;

[0290] Calculate the matching cost for each pixel in the target box region of the left view to obtain the cost matrix of the target box region of the left view;

[0291] Calculate the optimal arm length τ, aggregate the cost matrix of the target box region in the left view, and obtain the disparity of the target box region in the left view;

[0292] The formula for calculating the optimal arm length τ is as follows:

[0293]

[0294]

[0295] Where τ0 is the initial value of the preset pixel threshold, τ1 is the cutoff value of the preset pixel threshold, L is the preset arm length threshold, I is the pixel difference between the pixel point p and the pixel point q to be matched on the horizontal support arm, and L is the pixel difference between the pixel point p and the pixel point q to be matched on the horizontal support arm. p ,I q ,I q′ These are the pixel values ​​of three pixels, p, q, and q′, respectively; q′ is the pixel preceding pixel q on the horizontal support arm.

[0296] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0297] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0298] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0299] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0300] The embodiments of the present invention have been described above with reference to the accompanying drawings. However, the present invention is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of the present invention without departing from the spirit and scope of the claims. All of these forms are within the protection scope of the present invention.

[0301] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely illustrative of the principles of the invention. Various changes and modifications can be made to the invention without departing from its spirit and scope, and all such changes and modifications fall within the scope of the present invention as claimed. The scope of protection of this invention is defined by the appended claims and their equivalents.

Claims

1. A method for detecting and capturing underwater aquaculture organisms, characterized in that, include, Obtain the left and right views taken by two cameras mounted in a parallel binocular configuration; Restore the left and right views, including: The left and right views before restoration are referred to as the original images. For pixels In the restored image Pixel values ​​of the channel, For pixels In the original image Pixel values ​​of the channel, for Background light intensity of the channel, , , , These are the red, green, and blue channels in the RGB color channel. For pixels Place The transmittance of the channel; The process for determining the background light intensity is as follows: Arrange the pixel values ​​of the dark channel image of the merged fog lines in ascending order, and select the top 10% of pixels. Then, among these pixels in the original image, find the pixel with the lowest pixel value in each of the red, green, and blue channels. Use the lowest pixel value in each of the red, green, and blue channels as the background light value for the red, green, and blue channels, respectively. , , ; pixel Place The process of determining the channel transmittance is as follows: Construct the minimization function: in, For balance coefficient, In pixels A local area block centered on the core. for Pixels within; For pixels Maximum transmittance of the red channel; To be obtained from fog lines Standard deviation; For pixels Transmittance of the red channel; For pixels Transmittance of the red channel; , Each pixel Pixel Pixel values ​​in the red channel of the original image; The calculation formula is as follows: in, For pixels Initial transmittance of the red channel; For pixels The lower limit threshold of transmittance in the red channel. The calculation formula is as follows: in, , Each pixel Pixel values ​​in the green and blue channels of the original image; The ratio of the attenuation coefficients of red light to green light. This represents the ratio of the attenuation coefficients of red light to blue light. set up The minimized function is obtained by iteratively solving the problem. And obtain the pixel points according to the following formula. Transmittance of the green channel Pixel transmittance of the blue channel : ， in, , , These are the attenuation coefficients of red, green, and blue light in water, respectively. Detect the aquatic organisms in the restored left and right views to obtain the target bounding boxes of the aquatic organisms; Perform stereo matching on the target bounding box area of ​​the cultured organism to obtain the parallax of the left view; Calculate the size and position of the farmed animals based on the parallax of the left view; Grab the farmed animals based on their location and size.

2. The method for detecting and capturing underwater aquaculture organisms according to claim 1, characterized in that, This also includes correcting the left and right views before restoring them, including... The camera's intrinsic and extrinsic parameters are obtained through calibration: Construct the likelihood function: in, Indicates the first The first calibration map The coordinates of the corner points , For the total number of calibration maps, , This represents the total number of corner points in a single calibration map. for The estimated value; and The first The first calibration map Rotation and translation vectors of each corner point; For the first The three-dimensional coordinate matrix of the corner points in the calibration diagram; This represents the distortion parameter vector for a non-ideal lens model. This is the camera's internal parameter matrix; It is a planar homography matrix. for The estimated value; By setting the distortion coefficient and reprojection error threshold, the likelihood function is solved to obtain the camera's intrinsic and extrinsic parameters. Based on the camera's intrinsic and extrinsic parameters and the binocular stereo correction theory, coplanar correction and row alignment correction are performed on the left and right views.

3. The method for detecting and capturing underwater aquaculture organisms according to claim 1, characterized in that, The YOLOv5s convolutional neural network model was used to detect the aquatic organisms in the restored left and right views and obtain the target bounding boxes of the aquatic organisms. The YOLOv5s convolutional neural network model includes a backbone network, a feature extraction network, and a prediction network. The backbone network uses CSPNet and Focus as the base network, the sampling layer is the C3CBAM module, and the RFB module is used to extract the information of the restored left and right views. The feature extraction network constructs FPN and PAN feature pyramids for feature fusion, and uses bilinear interpolation and C3CBAM modules for splicing, adding a CBAM attention mechanism on the smallest scale information; The prediction network uses an anchor-based decoupled detection head to output three different resolutions for multi-scale prediction.

4. The method for detecting and capturing underwater aquaculture organisms according to claim 1, characterized in that, The stereo matching of the target frame region of the cultured organism to obtain the parallax of the left view includes, Based on the target bounding box region of the cultured animals in the left view, update the target bounding box region of the cultured animals in the right view according to the intersection-union ratio of the target bounding box regions of the cultured animals in the left and right views; Calculate the matching cost of pixels in the target bounding box region of the left view. include, in, For pixels in the target area of ​​the left view The matching cost, It is a pixel. Except for pixels in the target area of ​​the right view Any value among the pixel differences of all other pixels, , , These represent the pixels in the target bounding box region of the left view calculated by the census algorithm, AD algorithm, and gradient algorithm, respectively. The value of the substitute; , , These represent the control parameters for the census algorithm, AD algorithm, and gradient algorithm, respectively. For adaptive weight parameters, For fixed weights, , , The calculation formulas are as follows: in, The calculation formula is as follows: in, This represents the pixel grayscale value of the left view. This represents the pixel grayscale value of the right view. This indicates the maximum grayscale value of the target area in the right view. This indicates the minimum grayscale value of the target area in the left view. , The calculation formula is as follows: in, and For weight values, , The target area pixels in the left view of and Normalized value of grayscale value of direction pixel , It refers to the pixel points of the target box area in the right view. of and Normalized value of grayscale value of directional pixel; Calculate the matching cost for each pixel in the target box region of the left view to obtain the cost matrix of the target box region of the left view; Calculate the optimal arm length The cost matrix of the target box region in the left view is aggregated to obtain the disparity of the target box region in the left view; Optimal arm length The calculation formula is as follows: in, This is the initial value for the preset pixel threshold. This is the cutoff value for the preset pixel threshold. The preset arm length threshold, For the pixels to be matched on the horizontal support arm With pixels The pixel difference between them They are respectively The pixel values ​​of the three pixels; For pixels on the horizontal support arm The previous pixel.

5. The method for detecting and capturing underwater aquaculture organisms according to claim 1, characterized in that, The calculation of the size and position of the farmed animals based on the parallax of the left view includes, in, , These represent the world coordinates of the pixel on the horizontal and vertical axes of the world coordinate system, respectively. Represents the depth of a scene at a pixel level. , These are the x and y coordinates of the pixel in the left view, respectively. For the camera's focal length, Baseline distance, The disparity of pixels in the left view; Create the minimum bounding rectangle of the farmed organisms, and calculate the dimensions of the farmed organisms and the distance between the farmed organisms and the camera: in, Indicates the length of the farmed organism; Indicates the width of the farmed organism; Indicates the height of the farmed organisms; Indicates the distance between the farmed animals and the camera; , These are the minimum bounding rectangles of the cultured organism. The maximum and minimum parallax within; , These are the minimum bounding rectangles of the cultured organism. The side length.

6. The method for detecting and capturing underwater aquaculture organisms according to claim 1, characterized in that, The process of grasping the farmed organisms based on their location and size includes, Calculate the weight of the farmed animals based on their size, and then calculate the clamping force of the grippers based on the weight of the farmed animals. The end-effector pose of the multi-axis robotic arm is determined based on the position of the farmed animal. The joint angles of the multi-axis robotic arm are then determined based on the end-effector pose. The multi-axis robotic arm is driven to move based on the joint angles, bringing the gripper to the farmed animal. Based on the determined gripper force, the gripper is triggered to grasp the farmed animal.

7. A device for detecting and grasping underwater aquaculture organisms, characterized in that, Includes two cameras, power supply, controller, multi-axis robotic arm, and gripper; The two cameras are installed in a parallel binocular configuration to acquire images of the underwater aquaculture organisms. The images of the underwater aquaculture organisms are recorded as the left view and the right view, and uploaded to the controller. The power source is used to provide power to the camera, controller, multi-axis robotic arm, and gripper; The controller is used to: restore the left and right views; detect the farmed animals in the restored left and right views to obtain the target bounding box of the farmed animals; perform stereo matching on the target bounding box area of ​​the farmed animals to obtain the parallax of the left view; calculate the size and position of the farmed animals based on the parallax of the left view; calculate the weight of the farmed animals based on the size of the farmed animals; calculate the gripping force of the gripper based on the weight of the farmed animals; obtain the end-effector pose of the multi-axis robotic arm based on the position of the farmed animals; obtain the joint angles of the multi-axis robotic arm based on the end-effector pose of the multi-axis robotic arm; drive the multi-axis robotic arm to move and move the gripper to the farmed animals based on the joint angles of the multi-axis robotic arm; and trigger the gripper to grasp the farmed animals based on the obtained gripping force. The controller is used to restore the left and right views according to the following steps, including: The left and right views before restoration are referred to as the original images. For pixels In the restored image Pixel values ​​of the channel, For pixels In the original image Pixel values ​​of the channel, for Background light intensity of the channel, , , , These are the red, green, and blue channels in the RGB color channel. For pixels Place The transmittance of the channel; The process for determining the background light intensity is as follows: Arrange the pixel values ​​of the dark channel image of the merged fog lines in ascending order, and select the top 10% of pixels. Then, among these pixels in the original image, find the pixel with the lowest pixel value in each of the red, green, and blue channels. Use the lowest pixel value in each of the red, green, and blue channels as the background light value for the red, green, and blue channels, respectively. , , ; pixel Place The process of determining the channel transmittance is as follows: Construct the minimization function: in, For balance coefficient, In pixels A local area block centered on the core. for Pixels within; For pixels Maximum transmittance of the red channel; To be obtained from fog lines Standard deviation; For pixels Transmittance of the red channel; For pixels Transmittance of the red channel; , Each pixel Pixel Pixel values ​​in the red channel of the original image; The calculation formula is as follows: in, For pixels Initial transmittance of the red channel; For pixels The lower limit threshold of transmittance in the red channel. The calculation formula is as follows: in, , Each pixel Pixel values ​​in the green and blue channels of the original image; The ratio of the attenuation coefficients of red light to green light. This represents the ratio of the attenuation coefficients of red light to blue light. set up The minimized function is obtained by iteratively solving the problem. And obtain the pixel points according to the following formula. Transmittance of the green channel Pixel transmittance of the blue channel : ， in, , , These are the attenuation coefficients of red, green, and blue light in water, respectively.

8. The underwater aquaculture organism detection and grasping device according to claim 7, characterized in that, The controller is used to perform stereo matching on the target frame area of ​​the farmed organism to obtain the parallax of the left view according to the following steps, including: Based on the target bounding box region of the cultured animals in the left view, update the target bounding box region of the cultured animals in the right view according to the intersection-union ratio of the target bounding box regions of the cultured animals in the left and right views; Calculate the matching cost of pixels in the target bounding box region of the left view. include, in, For pixels in the target area of ​​the left view The matching cost, It is a pixel. Except for pixels in the target area of ​​the right view Any value among the pixel differences of all other pixels, , , These represent the pixels in the target bounding box region of the left view calculated by the census algorithm, AD algorithm, and gradient algorithm, respectively. The value of the substitute; , , These represent the control parameters for the census algorithm, AD algorithm, and gradient algorithm, respectively. For adaptive weight parameters, For fixed weights, , , The calculation formulas are as follows: in, The calculation formula is as follows: in, This represents the pixel grayscale value of the left view. This represents the pixel grayscale value of the right view. This indicates the maximum grayscale value of the target area in the right view. This indicates the minimum grayscale value of the target area in the left view. , The calculation formula is as follows: in, and For weight values, , The target area pixels in the left view of and Normalized value of grayscale value of direction pixel , It refers to the pixel points of the target box area in the right view. of and Normalized value of grayscale value of directional pixel; Calculate the matching cost for each pixel in the target box region of the left view to obtain the cost matrix of the target box region of the left view; Calculate the optimal arm length The cost matrix of the target box region in the left view is aggregated to obtain the disparity of the target box region in the left view; Optimal arm length The calculation formula is as follows: in, This is the initial value for the preset pixel threshold. This is the cutoff value for the preset pixel threshold. The preset arm length threshold, For the pixels to be matched on the horizontal support arm With pixels The pixel difference between them They are respectively The pixel values ​​of the three pixels; For pixels on the horizontal support arm The previous pixel.

Images