Fish size estimation system and method in complex underwater environment based on key point detection

Abstract

The present invention discloses a system and method for estimating fish size in a complex underwater environment based on key point detection. The system comprises an image acquisition module that acquires a calibration plate image and left and right images of the fish; a calibration parameter acquisition module that acquires calibration parameters; a distortion correction module that performs distortion correction on the left and right images; a depth information calculation module that calculates depth information corresponding to pixels in the left image; a data set construction module that annotates the fish body and constructs a fish key point detection data set; an image enhancement module that enhances the left image; a key point detection model construction module that constructs a key point detection model; a model training module that trains the key point detection model to identify fish targets and key points in the image; a key point movement module that moves each key point; a fish size calculation module that obtains depth information of the moved key points, converts pixel coordinates into world coordinates, and calculates the distance between each key point to obtain the fish size. The present invention achieves accurate estimation of fish size.

Description

Technical Field

[0001] The present invention relates to the technical field of fish size estimation, and in particular to a system and method for estimating fish size in a complex underwater environment based on key point detection. Background Art

[0002] Currently, fish size estimation methods based on machine vision can be divided into two categories. One category relies on two-dimensional size estimation technology, which obtains size information by fixing the swimming direction of the fish and taking two-dimensional images of the fish with a monocular camera. The other category relies on stereo vision technology, which obtains size information by taking three-dimensional images with a binocular camera.

[0003] Among the current two-dimensional size estimation techniques, some currently estimate the size by placing a reference object of known size next to the fish and establishing a proportional mapping between the pixel distance and the actual distance. This method must ensure that the fish and the reference object are in the same plane and that the fish's body cannot be bent, twisted, or deformed. Others fix the distance between the fish and the camera, use image segmentation technology to extract the fish's outline, and then calculate its characteristic size. These methods have specific requirements for ambient lighting and fish posture, and estimation based solely on two-dimensional feature information is unstable, making it difficult to apply in actual aquaculture scenarios.

[0004] Among those relying on stereo vision technology, some currently build binocular camera systems to collect three-dimensional information of fish bodies to generate point clouds, and combine them with deep learning RCNN segmentation models to accurately extract the three-dimensional contours of fish. Finally, the fish features are extracted and the size information is calculated based on the obtained point cloud data of a single fish. Although this method produces stable measurement results, the segmentation model has a large amount of computation, the required data is manually labeled at a high cost, the point cloud data processing process is complex, and the equipment requirements are high, making it difficult to popularize and apply in real aquaculture environments. Moreover, it is mostly concentrated in experimental environments with clear water quality and controllable lighting. In actual aquaculture scenarios, water quality fluctuations are often not so clear, and lighting conditions will also change with changes in weather factors, resulting in insufficient estimation accuracy and stability.

[0005] Therefore, how to accurately estimate fish size in a complex aquaculture environment with turbid water and variable light has become an urgent problem to be solved. Summary of the Invention

[0006] In order to overcome the defects and shortcomings of the existing technology, the present invention provides a fish size estimation system and method in a complex underwater environment based on key point detection. The present invention can accurately identify fish bodies that are not bent, overlapped, have no occlusion in key parts, and are parallel to the camera. It solves the problem of measurement error caused by posture deviation, realizes the precise calculation of fish size only through mathematical calculation, reduces the parameters of the overall model, and meets the demand for low-latency inference in a real aquaculture environment.

[0007] In order to achieve the above object, the present invention adopts the following technical solutions:

[0008] The present invention provides a fish size estimation system in a complex underwater environment based on key point detection, comprising:

[0009] Image acquisition module, used to collect the calibration plate image and the left and right images of the fish swimming freely underwater;

[0010] A calibration parameter acquisition module, used to obtain calibration parameters based on the calibration plate image;

[0011] A distortion correction module, used to perform distortion correction on the left and right images based on calibration parameters;

[0012] Depth information calculation module, used to calculate the depth information corresponding to each pixel of the left image;

[0013] The dataset construction module is used to annotate the fish in the left image, construct a fish key point detection dataset, and divide the dataset;

[0014] An image enhancement module, used for performing image enhancement on the left image;

[0015] The key point detection model construction module is used to build a key point detection model. It extracts fish features based on the multi-layer DRMB-Block module and the Conv downsampling module. The CGLU-SAM module adjusts the feature weights after feature fusion and splicing through the gating mechanism and spatial attention mechanism, generates a spatial weight map, realizes feature selection and regional focusing, and outputs fish targets and key points through multiple detection heads.

[0016] A model training module is used to input the enhanced left image into a key point detection model for training, and identify fish targets and key points in the image, wherein the key points include the head, tail, upper width point, and lower width point of the fish;

[0017] The key point moving module is used to move the head key point toward the tail key point, the tail key point toward the head key point, the upper width point key point toward the lower width point, and the lower width point toward the upper width point;

[0018] The fish size calculation module is used to obtain the depth information of the key points after movement, convert the pixel coordinates of the key points after movement into world coordinates, calculate the three-dimensional Euclidean distance between the head and tail to obtain the length of the fish, and calculate the three-dimensional Euclidean distance between the upper wide point and the lower wide point to obtain the width of the fish.

[0019] As a preferred technical solution, the data set construction module marks the fish bodies in the captured left image that have no bends, no overlaps, no occlusions on key parts and are parallel to the camera.

[0020] As a preferred technical solution, the image enhancement module uses a limited contrast histogram equalization algorithm to enhance the left image.

[0021] As a preferred technical solution, the key point detection model includes a backbone network and a head network;

[0022] In the backbone network, the image resolution of the enhanced left image is unified and then input into the Conv downsampling module for downsampling to obtain feature F1. Feature F1 passes through multiple layers of Conv downsampling modules and DRMB-Block modules to obtain feature F9. Among them, the second DRMB-Block module outputs feature F5, and the third DRMB-Block module outputs feature F7. Feature F9 passes through the SPPF module and CGLU-SAM module to obtain feature F11.

[0023] In the head network, feature F11 is upsampled by EUCB and concatenated with feature F7 to obtain feature F13. Feature F13 is processed by the CGLU-SAM module and the DRMB-Block module to obtain feature F15. Feature F15 is upsampled by EUCB and concatenated with feature F5 to obtain feature F17. Feature F17 is processed by the CGLU-SAM module and the DRMB-Block module to obtain feature F19. Feature F19 is input to the P3 detection head to obtain large-sized fish targets and key points.

[0024] After downsampling, feature F19 is concatenated with feature F15 to obtain feature F21. Feature F21 is processed through the CGLU-SAM module and the DRMB-Block module to obtain feature F23. Feature F23 is input to the P4 detection head to obtain medium-sized fish targets and key points.

[0025] After downsampling, feature F23 is concatenated with feature F11 to obtain feature F25. Feature F25 passes through the CGLU-SAM module and the DRMB-Block module and is input into the P5 detection head to obtain small-sized fish targets and key points.

[0026] As an optimal technical solution, the DRMB-Block module doubles the feature channel through the convolution layer, and divides the features into two feature branches according to the channel. The first feature branch extracts features through deep convolution, and adjusts the channel dimension through the convolution layer to obtain feature OF1. The second feature branch is copied to obtain two identical features, one of which is convolution, normalization, convolution, activation, and convolution to obtain feature OF2, and the other feature is extracted through deep convolution and ordinary convolution to obtain feature OF3. Feature OF4 obtained by adding feature OF2 and feature OF3 is spliced ​​with feature OF1, and the spliced ​​features are subjected to a convolution to adjust the channel dimension.

[0027] As a preferred technical solution, in EUCB upsampling, the nearest neighbor interpolation is used to expand the feature map resolution, and each channel is independently deep convolution is performed. The obtained features pass through a batch normalization layer and an activation layer, and the channel order is reorganized through channel shuffling, and the features are integrated through convolution.

[0028] As a preferred technical solution, the CGLU-SAM module adjusts the feature weights after feature fusion and splicing through a gating mechanism and a spatial attention mechanism, specifically including:

[0029] The CGLU-SAM module uses a gating mechanism to split the input features into two parts according to the channel dimension. The two features pass through two different linear transformation branches. One branch generates gating weights after passing through a deep convolution and activation layer. The gating weights are multiplied element-wise with the other branch. The channel feature strength is adjusted based on the gating weights. After passing through the linear layer, a residual connection is made with the input features, and the resulting features enter the spatial attention module.

[0030] In the spatial attention module, after the convolution operation, the global contextual spatial information is extracted through global average pooling and global maximum pooling respectively. After splicing the features extracted by global average pooling and global maximum pooling, the output feature map is obtained through convolution and activation operations.

[0031] As a preferred technical solution, pixel coordinates are converted into world coordinates, which can be specifically expressed as:

[0032] ;

[0033] in, Indicates the Z-axis distance between the target and the camera coordinate source point. 、 Respectively represent the focal length of the left camera and the focal length of the right camera, represents the principal point coordinates, represents the pixel coordinates of the projection point, represents the rotation matrix, represents the translation matrix, Represents world coordinates.

[0034] As a preferred technical solution, the pixel coordinates of the projection point are specifically expressed as:

[0035] ;

[0036] ;

[0037] The movement amount is expressed as:

[0038] ;

[0039] ;

[0040] in, Indicates the percentage of the distance from the source point to the target point that needs to be moved;

[0041] The straight-line distance from the source point to the target point is expressed as:

[0042] ;

[0043] ;

[0044] in, represents the original pixel coordinates, Indicates the pixel coordinates of the target point.

[0045] The present invention also provides a method for estimating fish size in a complex underwater environment based on key point detection, comprising the following steps:

[0046] Collect the calibration plate image and the left and right images of the fish swimming freely underwater;

[0047] Obtaining calibration parameters based on the calibration plate image;

[0048] Perform distortion correction on the left and right images based on the calibration parameters;

[0049] Calculate the depth information corresponding to each pixel of the left image;

[0050] Label the fish in the left image, construct a fish key point detection dataset, and divide the dataset;

[0051] Perform image enhancement on the left image;

[0052] A key point detection model is constructed, using a multi-layer DRMB-Block module and a Conv downsampling module to extract fish features. The CGLU-SAM module uses a gating mechanism and a spatial attention mechanism to adjust the feature weights after feature fusion and splicing, generating a spatial weight map to achieve feature selection and regional focusing. The model then outputs fish targets and key points through multiple detection heads.

[0053] The enhanced left image is input into the key point detection model for training to identify the fish target and key points in the image, wherein the key points include the head, tail, upper width point and lower width point of the fish;

[0054] Move the head key point toward the tail key point, the tail key point toward the head key point, the upper width point key point toward the lower width point, and the lower width point key point toward the upper width point;

[0055] Get the depth information of the key point after movement, convert the pixel coordinates of the key point after movement into world coordinates, calculate the 3D Euclidean distance between the head and tail to get the length of the fish, and calculate the 3D Euclidean distance between the upper wide point and the lower wide point to get the width of the fish.

[0056] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0057] (1) The present invention adopts the limited contrast histogram equalization algorithm for image enhancement, which solves the problem of image blurring caused by illumination changes and water eutrophication in real aquaculture environments, improves image contrast and clarity, and ensures that the edges of fish outlines can still be accurately captured under different illumination and water quality conditions.

[0058] (2) The present invention constructs the DRMB-Block module as the feature extraction module of the network. It realizes efficient and hierarchical feature extraction through the collaborative architecture of multi-branch feature decoupling, deep convolution lightweighting, residual connection and feature fusion, thereby improving the robustness of the model in complex environments.

[0059] (3) The present invention adopts EUCB upsampling to avoid the problems of weak detail recovery and blurred edges in traditional upsampling. By adding depthwise convolution to speed up efficiency and channel shuffling to capture intra-channel dependencies, the present invention retains and enhances feature details during the upsampling process while maintaining low computational latency.

[0060] (4) The convolutional gated spatial attention mechanism CGLU-SAM module of the present invention dynamically filters important channel information through a gating mechanism, suppresses redundant features, focuses on key areas of fish edges by generating a spatial weight map, reduces background noise interference, and solves the problem of feature information redundancy in multi-scale feature fusion and feature splicing in the SPPF module, improves the model's feature extraction capability in complex environments, and ensures the accuracy and stability of fish size estimation.

[0061] (5) The present invention solves the problem of incorrect depth information of key points caused by blurred fish edges in complex environments by projecting the key points of the fish edge in an adaptive direction. The three-dimensional Euclidean distance of the head, tail, upper width point and lower width point is accurately calculated through the three-dimensional information of the key points, thereby achieving accurate estimation of the fish size. BRIEF DESCRIPTION OF THE DRAWINGS

[0062] Figure 1 Schematic diagram of the overall architecture of the fish size estimation system in complex underwater environments based on key point detection of the present invention;

[0063] Figure 2 Schematic diagram of the overall network architecture of the key point detection model of the present invention;

[0064] Figure 3 Schematic diagram of the network architecture for implementing the DRMB-Block module of the present invention;

[0065] Figure 4 Schematic diagram of the network architecture for implementing EUCB upsampling in the present invention;

[0066] Figure 5 Schematic diagram of the network architecture for implementing the CGLU-SAM module of the present invention;

[0067] Figure 6 The figure is a flow chart of the method for estimating fish size in a complex underwater environment based on key point detection according to the present invention. DETAILED DESCRIPTION

[0068] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0069] Example 1

[0070] like Figure 1 As shown, this embodiment provides a fish size estimation system for complex underwater environments based on key point detection, including: an image acquisition module, a calibration parameter acquisition module, a distortion correction module, a depth information calculation module, a data set construction module, an image enhancement module, a key point detection model construction module, a model training module, a key point movement module, and a fish size calculation module;

[0071] The image acquisition module is used to collect the calibration plate image and the left and right images of the fish swimming freely underwater;

[0072] The calibration parameter acquisition module is used to obtain calibration parameters based on the calibration plate image;

[0073] The distortion correction module is used to perform distortion correction on the left and right images based on the calibration parameters;

[0074] The depth information calculation module is used to calculate the depth information corresponding to each pixel of the left image;

[0075] The dataset construction module is used to annotate the fish bodies in the left image. Specifically, it annotates the fish bodies in the left image that are not bent, overlapped, have no occlusion of key parts, and are parallel to the camera. It constructs a fish key point detection dataset and divides the dataset.

[0076] The image enhancement module is used to enhance the left image, specifically using a limited contrast histogram equalization algorithm to enhance the left image;

[0077] The key point detection model construction module is used to build a key point detection model. It extracts fish features based on the multi-layer DRMB-Block module and the Conv downsampling module. The CGLU-SAM module adjusts the feature weights after feature fusion and splicing through the gating mechanism and spatial attention mechanism, generates a spatial weight map, realizes feature selection and regional focusing, and outputs fish targets and key points through multiple detection heads.

[0078] like Figure 2 As shown, the key point detection model includes a backbone network and a head network

[0079] The enhanced left image is input into the key point detection model. In the backbone network, the original image resolution is first adjusted to 1024×1024, and then input into the Conv downsampling module for the first downsampling, reducing the spatial size by 1 times, and the number of channels remains unchanged to obtain feature F1. Subsequently, feature F1 will pass through the second Conv downsampling module and the first DRMB-Block module. Such Conv downsampling modules and DRMB modules will appear alternately 4 times for feature extraction. The feature F9 output by the fourth DRMB-Block module enters the SPPF module (fast spatial pyramid pooling module) for multi-scale extraction. The obtained feature F10 is input into the convolutional gated spatial attention mechanism CGLU-SAM module for feature selection and regional focusing to obtain feature F11. The SPPF module of this embodiment performs pooling operations of multiple scales on the feature map (such as 5×5, 3×3, and 1×1 pooling) and splices these pooled feature maps together. In this way, the model can extract useful contextual information from different spatial scales, thereby improving the detection ability of objects of different sizes.

[0080] Feature F11 is input into the first EUCB module of the head network for upsampling. The upsampled feature F12 is concatenated with the feature F7 output by the backbone network. The obtained feature F13 is input into the CGLU-SAM module for feature selection and regional focusing to obtain feature F14. Feature F14 is input into the DRMB-Block module for feature extraction to obtain feature F15. Feature F15 is upsampled by EUCB to obtain feature F16. Feature F16 is concatenated with feature F5 to obtain feature F17. Feature F17 is input into the CGLU-SAM module for feature selection and regional focusing to obtain feature F18. Feature F18 is input into the DRMB-Block module to obtain feature F19. Feature F19 is input into the P3 detection head and the downsampling module at the same time. The P3 detection head is used to detect large-sized fish targets and key points. The downsampling module further extracts features to obtain feature F20, which is concatenated with feature F15 to obtain feature F21. Feature F21 first enters the CGLU-SAM module for feature selection and regional focusing to obtain feature F22. Feature F22 is then input into the DRMB-Block module to obtain feature F23. Feature F23 is simultaneously input into the P4 detection head and the downsampling module. The P4 detection head detects medium-sized fish targets and key points. The downsampling module further refines the features to obtain feature F24, which is concatenated with feature F11 to obtain feature F25. Feature F25 is again subjected to enhanced regional focusing by the CGLU-SAM module to obtain feature F26. Feature F26 is then processed by the DRMB-Block module and ultimately output to the P5 detection head, which focuses on detecting small-sized fish targets and key points. The overall network structure ensures the accurate recognition and positioning of fish targets of different scales through the synergistic effect of multi-level feature fusion.

[0081] like Figure 3 As shown in the figure, after the feature map enters the DRMB-Block module, it will be disassembled according to the logic of branch processing, fusion, and input. First, the channel is doubled through the convolution layer, and the features are divided into two feature branches according to the channel. The first feature branch extracts features through deep convolution, and adjusts the channel dimension through the convolution layer to obtain feature OF1. The second feature branch is copied to obtain two identical features, one of which is convolved, normalized, convolved, activated, and convolved to obtain feature OF2. The other feature is extracted through deep convolution and ordinary convolution to obtain feature OF3. Feature OF4 obtained by adding feature OF2 and feature OF3 is concatenated with feature OF1, and finally the channel dimension is adjusted through a convolution.

[0082] In this embodiment, the first feature branch focuses on shallow fine-grained details, such as texture, edges, and other basic features, through lightweight deep convolution plus ordinary convolution. The second feature branch focuses on deep feature fusion and nonlinear enhancement through multiple convolutions, normalization, activation, and residual structure, processing more complex semantic features. Combining the feature processing of the two feature branches enables the network to have better feature extraction capabilities.

[0083] like Figure 4 As shown in the figure, in EUCB upsampling, the feature map resolution is first doubled using nearest neighbor interpolation, and then a 3×3 depthwise convolution is performed independently on each channel to reduce the number of parameters and extract spatial features. The obtained features pass through a batch normalization layer and an activation layer, and then the channel order is reorganized by channel shuffling to break the isolation between groups of grouped convolution and enhance cross-channel information interaction. Finally, a 1×1 convolution is performed to integrate the features.

[0084] In this embodiment, the specific steps of channel shuffling are as follows:

[0085] Input Tensor , the dimension is ,in, is the batch size, is the number of channels, is the characteristic height, is the feature width, and the number of groups is , the number of channels in each group is ;

[0086] Group the input tensor Grouping, the channels are divided into Groups, each containing channels, expressed as:

[0087] ;

[0088] Transpose swaps the group dimension and the channel dimension so that the channels in the original group are rearranged, expressed as:

[0089] ;

[0090] Flattening a tensor restores the tensor to its original four-dimensional shape, expressed as:

[0091] ;

[0092] like Figure 5As shown in the figure, the CGLU-SAM module consists of convolutional gating and spatial attention. The convolutional gating mechanism dynamically filters important channel information and suppresses redundant features, while the spatial attention focuses on key areas around the edges of fish to reduce background noise interference. Features first enter the convolutional gating module, where they are split into two parts according to the channel dimension. These two features undergo two different linear transformation branches, one of which generates gating weights after undergoing a depthwise convolution and activation layer. These gating weights are multiplied element-wise with the other branch to dynamically adjust the channel feature strength using the gating weights. After passing through the linear layer, a residual connection is made with the initial input features. The resulting features enter the spatial attention module to focus on important areas of the image. First, they undergo a convolution, then global average pooling and global maximum pooling are used to extract global contextual spatial information. After concatenating the two pooled features, they undergo another convolution and activation to obtain the final feature map. After the features pass through convolutional gating and spatial attention, the network's ability to extract important features in both channels and space is improved.

[0093] The model training module is used to input the enhanced left image into the key point detection model for training, and identify the fish targets and key points in the image. The key points include the head, tail, upper width point, and lower width point of the fish;

[0094] The key point moving module is used to move the head key point toward the tail key point, the tail key point toward the head key point, the upper width point key point toward the lower width point, and the lower width point toward the upper width point;

[0095] The fish size calculation module is used to obtain the depth information of key points after movement, convert pixel coordinates into world coordinates, calculate the 3D Euclidean distance between the head and tail to obtain the length of the fish, and calculate the 3D Euclidean distance between the upper and lower wide points to obtain the width of the fish. Specifically, it includes:

[0096] First, calculate the straight-line distance from the source point to the target point for subsequent unitization, expressed as:

[0097] ;

[0098] ;

[0099] in, represents the original pixel coordinates, Indicates the pixel coordinates of the target point;

[0100] The movement amount is calculated as:

[0101] ;

[0102] ;

[0103] in, Indicates the percentage of the distance from the source point to the target point that needs to be moved, preferably 0.01, which is 1% of the distance from the source point to the target point;

[0104] Calculate the pixel coordinates of the projection point, expressed as:

[0105] ;

[0106] ;

[0107] The conversion formula between the pixel coordinates of the projection point and the world coordinates is:

[0108] ;

[0109] in, Indicates the Z-axis distance (depth) between the target and the camera coordinate source point. is the pixel coordinate, 、 Respectively represent the focal length of the left camera, the focal length of the right camera, and the coordinates of the principal point is the camera internal parameter, rotation matrix and translation matrices is the camera extrinsic parameter, the camera intrinsic parameter and camera extrinsic parameter are obtained through camera calibration, Represents world coordinates;

[0110] After obtaining the world coordinates of the head and tail projection points, the length of the fish is calculated using the three-dimensional Euclidean distance, which is expressed as:

[0111] ;

[0112] in, is the world coordinate of the head projection key point, The world coordinates of the tail projection key point;

[0113] After obtaining the world coordinates of the projection points of the upper and lower wide points, the width of the fish is calculated using the three-dimensional Euclidean distance, which is expressed as:

[0114] ;

[0115] in, The world coordinates of the key point of the upper wide point projection, Project the keypoint world coordinates for the lower wide point.

[0116] This embodiment enhances the image by introducing a restricted contrast histogram equalization algorithm, constructs a key point detection model based on the YOLOv11 network model, constructs a DRMB-Block module as the main feature extraction module of the network, enhances the network's ability to extract fish features under complex lighting and water quality conditions, introduces the EUCB model for feature upsampling, and efficiently enhances the feature map without increasing computational overhead. The CGLU-SAM module adaptively adjusts the feature weights after feature fusion and splicing through a gating mechanism and a spatial attention mechanism, thereby improving the model's adaptability to complex environments. The key point detection model realizes accurate, efficient, and stable fish key point detection in complex underwater environments. Specifically, it detects the four key points of the identified fish: the head, tail, upper width point, and lower width point. The three-dimensional Euclidean distance between the key points is calculated, and accurate fish size estimation is achieved by converting pixel distance into actual distance.

[0117] Example 2

[0118] like Figure 6 As shown, this embodiment provides a method for estimating fish size in a complex underwater environment based on key point detection, comprising the following steps:

[0119] S1: Use the binocular camera and calibration plate installed underwater to capture images of the calibration plate at different angles and distances, and use Zhang's calibration method to obtain calibration parameters;

[0120] In this embodiment, the calibration parameters include the left camera focal length, right camera focal length, principal point coordinates, radial distortion coefficient, tangential distortion, rotation matrix, and translation matrix of the binocular camera;

[0121] S2: Use a binocular camera to capture left and right images of free-swimming fish in a real outdoor aquaculture scene, and use the calibrated parameters to correct the distortion of the left and right images;

[0122] S3: The NEURAL depth perception module of the ZED binocular camera uses the captured left and right images to calculate the depth information corresponding to each pixel in the left image;

[0123] In this embodiment, the stereo matching results are obtained based on the NEURAL depth perception module, which is represented as a two-dimensional matrix of 1920×1080 (image resolution). The depth value of each pixel corresponds to the pixel. After obtaining the depth information, the pixel coordinates are converted to world coordinates. After obtaining the world coordinates of the corresponding key points, the three-dimensional Euclidean distance is used to calculate the actual length and width of the fish.

[0124] In this embodiment, because the depth information corresponding to each pixel cannot be obtained from the left image alone, the left image and the right image are needed together to calculate the depth corresponding to each pixel;

[0125] S4: Annotate the fish bodies in the left image that are not bent, overlapped, have no occlusion of key parts, and are parallel to the camera. Build a fish key point detection dataset and divide the dataset into 70% of the images as a training set, 15% of the images as a validation set, and 15% of the images as a test set.

[0126] S5: The contrast-limited histogram equalization algorithm (CLAHE) is used to enhance the left image to improve the image quality;

[0127] In this embodiment, the left image enhancement is aimed at adjusting the contrast, thereby enhancing the clarity of the edge contour of the fish body;

[0128] In this embodiment, the contrast-limited histogram equalization algorithm avoids noise amplification and over-enhancement by limiting local contrast and performing block processing. First, the image is divided into non-overlapping sub-blocks. The grayscale histogram of each sub-block is calculated. A contrast-limited threshold is set to trim excessive pixel counts and evenly redistribute excess pixels to avoid local over-enhancement. The processed histogram is then equalized to generate a grayscale mapping function within the sub-block. To eliminate the blocking effect caused by sub-block splicing, bilinear interpolation is used to smooth the transition between the edges of adjacent sub-blocks. The equalization results are then blended based on the weights of the pixels in the adjacent sub-blocks. Finally, all sub-blocks are merged to obtain the fully enhanced image.

[0129] S6: Input the enhanced left image into the key point detection model for training, and optimize the model parameters through the loss function;

[0130] like Figure 2 As shown, the enhanced left image is input into the key point detection model. In the backbone network, the original image resolution is first adjusted to 1024×1024, and then input into the Conv downsampling module for the first downsampling, reducing the spatial size by 1 times, and the number of channels remains unchanged to obtain feature F1. Subsequently, feature F1 will pass through the second Conv downsampling module and the first DRMB-Block module. Such Conv downsampling modules and DRMB modules will appear alternately 4 times for feature extraction. The feature F9 output by the fourth DRMB-Block module enters the SPPF module (fast spatial pyramid pooling module) for multi-scale extraction. The obtained feature F10 is input into the convolutional gated spatial attention mechanism CGLU-SAM module for feature selection and regional focusing to obtain feature F11. The SPPF module of this embodiment performs pooling operations of multiple scales on the feature map (such as 5×5, 3×3, and 1×1 pooling) and splices these pooled feature maps together. In this way, the model can extract useful contextual information from different spatial scales, thereby improving the detection ability of objects of different sizes.

[0131] Feature F11 is input into the first EUCB module of the head network for upsampling. The upsampled feature F12 is concatenated with the feature F7 output by the backbone network. The obtained feature F13 is input into the CGLU-SAM module for feature selection and regional focusing to obtain feature F14. Feature F14 is input into the DRMB-Block module for feature extraction to obtain feature F15. Feature F15 is upsampled by EUCB to obtain feature F16. Feature F16 is concatenated with feature F5 to obtain feature F17. Feature F17 is input into the CGLU-SAM module for feature selection and regional focusing to obtain feature F18. Feature F18 is input into the DRMB-Block module to obtain feature F19. Feature F19 is input into the P3 detection head and the downsampling module at the same time. The P3 detection head is used to detect large-sized fish targets and key points. The downsampling module further extracts features to obtain feature F20, which is concatenated with feature F15 to obtain feature F21. Feature F21 first enters the CGLU-SAM module for feature selection and regional focusing to obtain feature F22. Feature F22 is then input into the DRMB-Block module to obtain feature F23. Feature F23 is simultaneously input into the P4 detection head and the downsampling module. The P4 detection head detects medium-sized fish targets and key points. The downsampling module further refines the features to obtain feature F24, which is concatenated with feature F11 to obtain feature F25. Feature F25 is again subjected to enhanced regional focusing by the CGLU-SAM module to obtain feature F26. Feature F26 is then processed by the DRMB-Block module and ultimately output to the P5 detection head, which focuses on detecting small-sized fish targets and key points. The overall network structure ensures the accurate recognition and positioning of fish targets of different scales through the synergistic effect of multi-level feature fusion.

[0132] like Figure 3 As shown in the figure, after the feature map enters the DRMB-Block module, it will be disassembled according to the logic of branch processing, fusion, and input. First, the channel is doubled through the convolution layer, and the features are divided into two feature branches according to the channel. The first feature branch extracts features through deep convolution, and adjusts the channel dimension through the convolution layer to obtain feature OF1. The second feature branch is copied to obtain two identical features, one of which is convolved, normalized, convolved, activated, and convolved to obtain feature OF2. The other feature is extracted through deep convolution and ordinary convolution to obtain feature OF3. Feature OF4 obtained by adding feature OF2 and feature OF3 is concatenated with feature OF1, and finally the channel dimension is adjusted through a convolution.

[0133] In this embodiment, the first feature branch focuses on shallow fine-grained details, such as texture, edges, and other basic features, through lightweight deep convolution plus ordinary convolution. The second feature branch focuses on deep feature fusion and nonlinear enhancement through multiple convolutions, normalization, activation, and residual structure, processing more complex semantic features. Combining the feature processing of the two feature branches enables the network to have better feature extraction capabilities.

[0134] like Figure 4 As shown in the figure, in EUCB upsampling, the feature map resolution is first doubled using nearest neighbor interpolation, and then a 3×3 depthwise convolution is performed independently on each channel to reduce the number of parameters and extract spatial features. The obtained features pass through a batch normalization layer and an activation layer, and then the channel order is reorganized by channel shuffling to break the isolation between groups of grouped convolution and enhance cross-channel information interaction. Finally, a 1×1 convolution is performed to integrate the features.

[0135] In this embodiment, the specific steps of channel shuffling are as follows:

[0136] Input Tensor , the dimension is ,in, is the batch size, is the number of channels, is the characteristic height, is the feature width, and the number of groups is , the number of channels in each group is ;

[0137] Group the input tensor Grouping, the channels are divided into Groups, each containing channels, expressed as:

[0138] ;

[0139] Transpose swaps the group dimension and the channel dimension so that the channels in the original group are rearranged, expressed as:

[0140] ;

[0141] Flattening a tensor restores the tensor to its original four-dimensional shape, expressed as:

[0142] ;

[0143] like Figure 5As shown in the figure, the CGLU-SAM module consists of convolutional gating and spatial attention. The convolutional gating mechanism dynamically filters important channel information and suppresses redundant features, while the spatial attention focuses on key areas around the edges of fish to reduce background noise interference. Features first enter the convolutional gating module, where they are split into two parts according to the channel dimension. These two features undergo two different linear transformation branches, one of which generates gating weights after undergoing a depthwise convolution and activation layer. These gating weights are multiplied element-wise with the other branch to dynamically adjust the channel feature strength using the gating weights. After passing through the linear layer, a residual connection is made with the initial input features. The resulting features enter the spatial attention module to focus on important areas of the image. First, they undergo a convolution, then global average pooling and global maximum pooling are used to extract global contextual spatial information. After concatenating the two pooled features, they undergo another convolution and activation to obtain the final feature map. After the features pass through convolutional gating and spatial attention, the network's ability to extract important features in both channels and space is improved.

[0144] S7: Input the enhanced left image into the trained key point detection model for inference, identify the fish body outline with no bends, no overlaps, no occlusion of key parts, and the fish body parallel to the camera, and detect the four key points of the identified fish: head, tail, upper wide point, and lower wide point;

[0145] S8: According to the key point detection results, the head key point is moved toward the tail key point, the tail key point is moved toward the head key point, the upper wide point key point is moved toward the lower wide point, and the lower wide point is moved toward the upper wide point. Preferably, the distance moved by the head and tail key points is 1% of the pixel distance from the head key point to the tail key point, and the distance moved by the upper wide point and lower wide point key points is 1% of the pixel distance from the upper wide point key point to the lower wide point.

[0146] S9: Obtain the depth information of the key points after movement, convert the pixel coordinates into world coordinates and combine them with the Euclidean distance formula to calculate the three-dimensional Euclidean distance of the head, tail, upper wide point and lower wide point to obtain the actual size of the fish.

[0147] In this embodiment, due to the large changes in light and turbid water in real aquaculture scenes, there may be a problem of loss of depth information of the fish edge. In order to prevent model misjudgment, this embodiment constructs a key point projection method, reads the depth information based on the projected key point information, and converts its pixel coordinates into world coordinates. The three-dimensional Euclidean distance between the head and tail is calculated as the fish length, and the three-dimensional Euclidean distance between the upper width and the lower width is calculated as the fish width. The specific steps include:

[0148] First, calculate the straight-line distance from the source point to the target point for subsequent unitization, expressed as:

[0149] ;

[0150] ;

[0151] in, represents the original pixel coordinates, Indicates the pixel coordinates of the target point;

[0152] The movement amount is calculated as:

[0153] ;

[0154] ;

[0155] in, Indicates the percentage of the distance from the source point to the target point that needs to be moved, preferably 0.01, which is 1% of the distance from the source point to the target point;

[0156] Calculate the pixel coordinates of the projection point, expressed as:

[0157] ;

[0158] ;

[0159] The conversion formula between the pixel coordinates of the projection point and the world coordinates is:

[0160] ;

[0161] in, Indicates the Z-axis distance (depth) between the target and the camera coordinate source point. is the pixel coordinate, 、 Respectively represent the focal length of the left camera, the focal length of the right camera, and the coordinates of the principal point is the camera internal parameter, rotation matrix and translation matrices is the camera extrinsic parameter, the camera intrinsic parameter and camera extrinsic parameter are obtained through camera calibration, Represents world coordinates;

[0162] After obtaining the world coordinates of the head and tail projection points, the length of the fish is calculated using the three-dimensional Euclidean distance, which is expressed as:

[0163] ;

[0164] in, is the world coordinate of the head projection key point, The world coordinates of the tail projection key point;

[0165] After obtaining the world coordinates of the projection points of the upper and lower wide points, the width of the fish is calculated using the three-dimensional Euclidean distance, which is expressed as:

[0166] ;

[0167] in, The world coordinates of the key point of the upper wide point projection, Project the keypoint world coordinates for the lower wide point.

[0168] The above embodiments are preferred implementation modes of the present invention, but the implementation modes of the present invention are not limited to the above embodiments. Any other changes, modifications, substitutions, combinations, and simplifications that do not deviate from the spirit and principles of the present invention should be considered as equivalent replacement methods and are included in the scope of protection of the present invention.

Claims

1. A fish size estimation system in complex underwater environments based on key point detection, characterized in that: include: Image acquisition module, used to collect the calibration plate image and the left and right images of the fish swimming freely underwater; A calibration parameter acquisition module, used to obtain calibration parameters based on the calibration plate image; A distortion correction module, used to perform distortion correction on the left and right images based on calibration parameters; Depth information calculation module, used to calculate the depth information corresponding to each pixel of the left image; The dataset construction module is used to annotate the fish in the left image, construct a fish key point detection dataset, and divide the dataset; An image enhancement module, used for performing image enhancement on the left image; The key point detection model construction module is used to build a key point detection model. It extracts fish features based on the multi-layer DRMB-Block module and the Conv downsampling module. The CGLU-SAM module adjusts the feature weights after feature fusion and splicing through the gating mechanism and spatial attention mechanism, generates a spatial weight map, realizes feature selection and regional focusing, and outputs fish targets and key points through multiple detection heads. A model training module is used to input the enhanced left image into a key point detection model for training, and identify fish targets and key points in the image, wherein the key points include the head, tail, upper width point, and lower width point of the fish; The key point moving module is used to move the head key point toward the tail key point, the tail key point toward the head key point, the upper width point key point toward the lower width point, and the lower width point toward the upper width point; The fish size calculation module is used to obtain the depth information of the key points after movement, convert the pixel coordinates of the key points after movement into world coordinates, calculate the three-dimensional Euclidean distance between the head and tail to obtain the length of the fish, and calculate the three-dimensional Euclidean distance between the upper wide point and the lower wide point to obtain the width of the fish.

2. The fish size estimation system in complex underwater environment based on key point detection according to claim 1 is characterized in that: The dataset construction module annotates the fish bodies in the captured left image that have no bending, no overlap, no occlusion of key parts, and the fish bodies are parallel to the camera.

3. The fish size estimation system in complex underwater environment based on key point detection according to claim 1 is characterized in that: The image enhancement module uses the limited contrast histogram equalization algorithm to enhance the left image.

4. The fish size estimation system in complex underwater environment based on key point detection according to claim 1 is characterized in that: The key point detection model includes a backbone network and a head network; In the backbone network, the image resolution of the enhanced left image is unified and then input into the Conv downsampling module for downsampling to obtain feature F1. Feature F1 passes through multiple layers of Conv downsampling modules and DRMB-Block modules to obtain feature F9. Among them, the second DRMB-Block module outputs feature F5, and the third DRMB-Block module outputs feature F7. Feature F9 passes through the SPPF module and CGLU-SAM module to obtain feature F11. In the head network, feature F11 is upsampled by EUCB and concatenated with feature F7 to obtain feature F13. Feature F13 is processed by the CGLU-SAM module and the DRMB-Block module to obtain feature F15. Feature F15 is upsampled by EUCB and concatenated with feature F5 to obtain feature F17. Feature F17 is processed by the CGLU-SAM module and the DRMB-Block module to obtain feature F19. Feature F19 is input to the P3 detection head to obtain large-sized fish targets and key points. After downsampling, feature F19 is concatenated with feature F15 to obtain feature F21. Feature F21 is processed through the CGLU-SAM module and the DRMB-Block module to obtain feature F23. Feature F23 is input to the P4 detection head to obtain medium-sized fish targets and key points. After downsampling, feature F23 is concatenated with feature F11 to obtain feature F25. Feature F25 passes through the CGLU-SAM module and the DRMB-Block module and is input into the P5 detection head to obtain small-sized fish targets and key points.

5. The fish size estimation system in complex underwater environment based on key point detection according to claim 4 is characterized in that: The DRMB-Block module doubles the channel of the feature through the convolution layer and divides the feature into two feature branches according to the channel. The first feature branch extracts features through deep convolution and adjusts the channel dimension through the convolution layer to obtain feature OF1. The second feature branch is copied to obtain two identical features. One feature is convolved, normalized, convolved, activated, and convolved to obtain feature OF2. The other feature is extracted through deep convolution and ordinary convolution to obtain feature OF3. Feature OF4 obtained by adding features OF2 and OF3 is spliced ​​with feature OF1, and the spliced ​​features are subjected to a convolution to adjust the channel dimension.

6. The fish size estimation system in complex underwater environment based on key point detection according to claim 4 is characterized in that: In EUCB upsampling, the nearest neighbor interpolation is used to expand the feature map resolution, and each channel is independently deep convolution is performed. The obtained features pass through a batch normalization layer and an activation layer, and the channel order is reorganized through channel shuffling, and the features are integrated through convolution.

7. The fish size estimation system in complex underwater environment based on key point detection according to claim 1 is characterized in that: The CGLU-SAM module adjusts the feature weights after feature fusion and splicing through a gating mechanism and a spatial attention mechanism, specifically: The CGLU-SAM module uses a gating mechanism to split the input features into two parts according to the channel dimension. The two features pass through two different linear transformation branches. One branch generates gating weights after passing through a deep convolution and activation layer. The gating weights are multiplied element-wise with the other branch. The channel feature strength is adjusted based on the gating weights. After passing through the linear layer, a residual connection is made with the input features, and the resulting features enter the spatial attention module. In the spatial attention module, after the convolution operation, the global contextual spatial information is extracted through global average pooling and global maximum pooling respectively. After splicing the features extracted by global average pooling and global maximum pooling, the output feature map is obtained through convolution and activation operations.

8. The fish size estimation system in complex underwater environment based on key point detection according to claim 1 is characterized in that: Convert the pixel coordinates of the key point after movement into world coordinates, specifically expressed as: in, Indicates the Z-axis distance between the target and the camera coordinate source point. 、 Respectively represent the focal length of the left camera and the focal length of the right camera, represents the principal point coordinates, represents the pixel coordinates of the projection point, represents the rotation matrix, represents the translation matrix, Represents world coordinates.

9. The fish size estimation system in complex underwater environment based on key point detection according to claim 8, characterized in that: The pixel coordinates of the projection point are specifically expressed as: The movement amount is expressed as: in, Indicates the percentage of the distance from the source point to the target point that needs to be moved; The straight-line distance from the source point to the target point is expressed as: in, represents the original pixel coordinates, Indicates the pixel coordinates of the target point.

10. A method for estimating fish size in complex underwater environments based on key point detection, characterized in that: The steps include: Collect the calibration plate image and the left and right images of the fish swimming freely underwater; Obtaining calibration parameters based on the calibration plate image; Perform distortion correction on the left and right images based on the calibration parameters; Calculate the depth information corresponding to each pixel of the left image; Label the fish in the left image, construct a fish key point detection dataset, and divide the dataset; Perform image enhancement on the left image; A key point detection model is constructed, using a multi-layer DRMB-Block module and a Conv downsampling module to extract fish features. The CGLU-SAM module uses a gating mechanism and a spatial attention mechanism to adjust the feature weights after feature fusion and splicing, generating a spatial weight map to achieve feature selection and regional focusing. The model then outputs fish targets and key points through multiple detection heads. The enhanced left image is input into the key point detection model for training to identify the fish target and key points in the image, wherein the key points include the head, tail, upper width point and lower width point of the fish; Move the head key point toward the tail key point, the tail key point toward the head key point, the upper width point key point toward the lower width point, and the lower width point key point toward the upper width point; Get the depth information of the key point after movement, convert the pixel coordinates of the key point after movement into world coordinates, calculate the 3D Euclidean distance between the head and tail to get the length of the fish, and calculate the 3D Euclidean distance between the upper wide point and the lower wide point to get the width of the fish.

Images