Intelligent device and method for investigating fish species and quantity

Abstract

The application discloses a device and method for intelligently investigating fish species and quantity. The device comprises an adaptive sonar detection module, a three-dimensional image acquisition module, an image preprocessing module, a fish detection module, a main body segmentation module and an identification and classification module. The sonar module integrates a three-frequency transducer, which can dynamically switch frequencies according to the depth of the fish school, and realize target tracking counting and quantity estimation in combination with an algorithm; the image acquisition module uses a depth camera in combination with a light attenuation compensation algorithm to construct a three-dimensional image model of the fish school; the preprocessing module fuses acoustic and optical features to expand the data set; the detection and segmentation module accurately locates the fish and removes impurities; and the identification and classification module realizes species identification based on transfer learning and supports incremental learning of new fish species. Through the cooperation of multiple modules, the method solves the problems of insufficient detection accuracy and poor image quality in traditional investigation, realizes efficient and accurate investigation of fish species and quantity, and provides technical support for fishery resource management and ecological protection.

Description

Technical Field

[0001] This invention relates to the field of fish survey technology, specifically to an intelligent device and method for surveying fish species and quantities. Background Technology

[0002] In fields such as fisheries resource surveys, aquatic ecological monitoring, and aquatic organism research, accurately obtaining information on fish species and quantities is crucial for assessing the status of fisheries resources and maintaining aquatic ecological balance. However, current fish survey methods have many shortcomings and are unable to efficiently and accurately meet practical needs.

[0003] Traditional fish surveys primarily rely on manual fishing sampling, which has significant limitations. On the one hand, due to the range of fish activity and their habitat habits (such as some fish being nocturnal and preferring to inhabit specific water layers or complex aquatic structures), manual fishing cannot provide comprehensive coverage, and the samples often fail to accurately reflect the overall distribution and quantity of fish in the water. On the other hand, frequent fishing directly damages the aquatic ecosystem, interfering with fish survival and reproduction, which contradicts the concept of ecological protection and is particularly unsuitable for surveys of rare and endangered fish species.

[0004] With technological advancements, sonar detection has been increasingly applied to fish surveys, but conventional sonar systems suffer from functional limitations. Most sonar systems use single-frequency transducers, making it difficult to adapt to the needs of detecting fish schools at different depths. In shallow waters, while high-frequency sonar offers high resolution, its short propagation distance prevents comprehensive coverage; in deep waters, low-frequency sonar has a long propagation range but is prone to losing fish details due to insufficient resolution, leading to significant errors in target identification and counting. Furthermore, in complex aquatic environments (such as turbid waters and underwater topography), echo signals are easily interfered with, making it difficult for traditional sonar image processing algorithms to effectively extract fish targets. Multi-target tracking is prone to trajectory loss and misjudgment, hindering accurate fish population counting.

[0005] Image recognition technology faces numerous challenges in assisting fish surveys. The unique underwater environment, with its uneven lighting, water scattering, and interference from impurities, results in poor quality raw images. Existing image preprocessing methods often employ simple denoising and enhancement operations, failing to fully integrate multi-dimensional image features (such as grayscale, color, and texture), making it difficult to reconstruct clear and effective fish target information. In fish detection and classification, general image recognition models exhibit poor adaptability to underwater fish morphology and texture, lacking optimization for underwater scenarios. Model training relies heavily on labeled data, making it prone to misclassification of new fish species or individuals with morphological variations. The lack of a complete and efficient processing workflow makes it difficult to guarantee the accuracy of fish species identification and the precision of quantity statistics.

[0006] In summary, existing fish survey technologies have shortcomings in multi-scenario adaptation, resistance to interference in complex environments, module collaboration, and accurate identification and counting, failing to meet the refined needs of dynamic monitoring of fishery resources and ecological research. Therefore, there is an urgent need to develop an intelligent system integrating multi-band adaptive sonar, multi-dimensional image preprocessing, intelligent detection and segmentation, and transfer learning recognition to achieve efficient and accurate surveys of fish species and quantities, providing strong technical support for fishery resource management and aquatic ecological protection. Summary of the Invention

[0007] The technical problem to be solved by the present invention is to provide an intelligent device and method for investigating fish species and quantities, so as to solve the problems of insufficient accuracy of single-band sonar detection, poor underwater image preprocessing effect and weak adaptability of recognition model in traditional fish surveys. Through the synergy of multi-band sonar adaptive detection, multi-dimensional image preprocessing, intelligent detection and segmentation and transfer learning recognition, efficient and accurate investigation of fish species and quantities can be achieved.

[0008] To address the aforementioned technical problems, embodiments of the present invention provide the following technical solutions:

[0009] An intelligent device for surveying fish species and quantities includes:

[0010] The adaptive sonar detection module integrates three-band transducers of 30kHz, 200kHz, and 400kHz, dynamically switches frequencies according to the depth of the fish school, and generates echo intensity feature vectors. It extracts targets through sonar image processing algorithms, uses the nearest neighbor algorithm combined with the extended Kalman filter algorithm to track and count multiple targets, counts the volume / area density of the water area swept by the sonar, and estimates the number of fish by combining the water area.

[0011] The 3D image acquisition module uses a depth camera to collect water depth and turbidity data in real time and simultaneously mark the image time sequence, and capture images of fish schools, covering multiple morphological scenes not limited to the side of the fish body and bending posture, to form a 3D image model of the fish school; at the same time, when capturing fish schools, the supplementary light is used to dynamically adjust the light intensity according to the water depth, and the light attenuation compensation algorithm is used to adapt to the underwater light environment.

[0012] The image preprocessing module associates the three-dimensional image model of the fish school with the acoustic signals output by sonar detection, constructs a high-dimensional image dataset of the fish school with acoustic and optical capabilities, and expands the dataset through data augmentation methods, including but not limited to rotation, scaling, and cropping.

[0013] The fish detection module is used to input the preprocessed high-dimensional image dataset into the fish detection model, call the detection algorithm to detect fish in the image, calculate the coordinate position of the fish detection box in the image, and the corresponding prediction score.

[0014] The main body segmentation module, after obtaining the location of the fish target rectangle, completes the target cropping and enters the fish segmentation model for main body segmentation. After segmentation, underwater impurities that interfere with the image are removed, so that the image only contains the main recognition object, the fish target.

[0015] The identification and classification module inputs the segmented fish images into the transfer learning identification model to classify and identify the fish species.

[0016] Preferably, when the depth camera acquires water depth and turbidity data in real time and simultaneously marks the image time sequence to capture images of the fish school to form a three-dimensional image model of the fish school, it adopts the following steps:

[0017] The depth information of the fish school is obtained by using Time-of-Flight (TOF) technology and fused with the pixel information of the collected fish school image to generate fish school image data with depth channels.

[0018] Based on a multi-frame image sequence, the SFM structure-of-motion (SOFM) algorithm is used, combined with the time sequence of the labeled images, to construct a dynamic 3D image model of a school of fish over a certain period of time, showing the posture changes of the school of fish during swimming.

[0019] By adjusting the field of view and resolution parameters of the depth camera, it is ensured that depth data of fish schools can be collected at a point cloud density of no less than a certain level at different underwater distances.

[0020] Preferably, the adaptation to the underwater light environment through the light attenuation compensation algorithm specifically includes:

[0021] The acquired fish images were subjected to illumination equalization processing. An adaptive histogram equalization algorithm was used to adjust the image contrast based on the light intensity distribution after light attenuation compensation.

[0022] An underwater optical attenuation compensation network is introduced. An optical attenuation compensation model is constructed based on the U-Net architecture. The input is the original image and the optical attenuation coefficient, and the output is the image after optical attenuation compensation. The formula is as follows:

[0023]

[0024] Calculate the optical attenuation compensation enhancement value, combine the image features before and after optical attenuation compensation, and fuse the compensated image features with the original image features through a feature fusion layer;

[0025] in, This represents the light attenuation compensation enhancement value at pixel position (i,j) on channel c. α represents the light intensity value at pixel position (i,j) on channel c in the original image. i,j This represents the local light attenuation estimation factor at pixel location (i,j). The color shift correction factor represents the pixel position (i,j) on channel c. The compensation perturbation factor represents the pixel position (i,j) on channel c. This represents the estimated attenuation value at pixel position (i,j) on channel c. This represents the local average value of the estimated attenuation value in the neighborhood centered at pixel position (i,j) on channel c.

[0026] Preferably, the three-dimensional image model of the fish school is associated with the acoustic signal output from the sonar detection to construct a high-dimensional image dataset of the fish school combining acoustic and optical methods, specifically as follows:

[0027] Bandpass filtering was performed on the sonar echo signals in the 30kHz, 200kHz and 400kHz bands respectively to remove environmental noise and frequency aliasing interference.

[0028] A GPS timing module is used to timestamp and synchronize sonar signals and optical images to ensure data alignment at the same time.

[0029] The sonar detection depth and optical acquisition depth are calibrated using an extended Kalman filter algorithm.

[0030] Extract the time-frequency and spatial features of sonar signals, as well as the geometric and texture features of three-dimensional images;

[0031] The extracted acoustic and optical features are weighted and fused, and the high-dimensional features after fusion are then reduced in dimensionality.

[0032] The fused dataset is hierarchically labeled with behavioral patterns, fish species attributes, and environmental parameters. Cross-modal consistency testing, sample equalization processing, and incremental learning validation are performed on the dataset.

[0033] Preferably, the data augmentation method, which is not limited to rotation, scaling, and cropping, is used to expand the dataset, specifically as follows:

[0034] The fish image after subject segmentation is rotated, with the rotation angle ranging from -15° to 15°, to simulate different swimming postures of fish underwater.

[0035] The images are scaled, with the scaling ratio controlled between 0.8 and 1.2 times, to accommodate changes in fish size at different shooting distances;

[0036] Using a random cropping method, the region containing the complete fish body is cropped from the original image, and the cropping size is no less than 70% of the original image;

[0037] Mosaic data augmentation is introduced, which randomly stitches together multiple different fish images to simulate a scene with multiple schools of fish.

[0038] Preferably, the adaptive sonar detection module calculates the volume / area density of the water area swept by the sonar, and estimates the fish population by combining this with the water area, specifically as follows:

[0039] When the sonar scans the water area with a fan-shaped beam at a scanning angle of θ, a maximum detection range of R, and a depth of h, the scanned volume is... Horizontal scan area

[0040] Divide the number of tracked targets N by the scanned volume / area to obtain the volume density ρ. V =N / V or surface density ρ S =N / S;

[0041] Given the total area S of the water body total At that time, the total number N total =ρ S ·S toal Alternatively, the total volume can be estimated by combining the average depth calculation.

[0042] Preferably, the fish detection module calls a detection algorithm to detect fish in the image, calculates the coordinates of the fish detection boxes in the image, and the corresponding prediction scores, specifically as follows:

[0043] The Faster R-CNN detection algorithm is used to extract multi-scale visual features from preprocessed images to capture the morphology, texture, and edge information of fish.

[0044] Anchor boxes are generated based on feature maps, and their positions are adjusted through regression calculations to determine the coordinates of the fish detection boxes.

[0045] The classifier is used to calculate the class probability of the target in each detection box, and the predicted score and the corresponding fish category label are output.

[0046] Preferably, after obtaining the location of the fish target bounding box, the main body segmentation module performs target cropping and then enters the fish segmentation model for main body segmentation. The segmented image is free of underwater impurities interfering with the image. Specifically:

[0047] Based on the bounding box coordinates output by the fish detection module, the preprocessed composite image is cropped to extract the region of interest (ROI) containing the fish target, and the background area outside the bounding box is removed to narrow down the scope of subsequent processing.

[0048] The cropped ROI image is input into a U-Net-based fish segmentation model. The encoder-decoder structure classifies the pixels in the image and generates masks for the fish body and background. The model's pre-training weights are derived from an underwater biological dataset.

[0049] The ROI image is binarized based on segmentation masking, and small noise areas are eliminated by combining morphological operations. Finally, the main pixels of the fish are retained and interference from underwater suspended objects and aquatic plants is removed, so that the output image contains only the fish target and its outline details.

[0050] Preferably, the identification and classification module inputs the segmented fish image into the transfer learning identification model to classify and identify the fish species, specifically as follows:

[0051] The transfer learning model is constructed based on the pre-trained deep learning network ResNet, where the pre-trained weights come from ImageNet or underwater fish datasets, using their learned general visual features as basic parameters.

[0052] Image feature extraction and adaptation: The fish image after subject segmentation is adjusted to the model input size. High-level semantic features are extracted through the model's convolutional and pooling layers. Batch normalization and dropout regularization are introduced in the feature extraction stage.

[0053] The classification and output process maps the extracted features through the fully connected layers and softmax classifier of the model to generate the probability distribution of each fish category, and outputs the label of the fish category with the highest probability and the corresponding confidence score. When a new fish species is detected, the model supports incremental learning and achieves adaptive recognition of the new category by updating the classification layer weights online.

[0054] This invention also proposes an intelligent method for investigating fish species and quantities, comprising the following steps:

[0055] By integrating a 30kHz, 200kHz, and 400kHz three-band transducer, the frequency is dynamically switched according to the depth of the fish school to generate an echo intensity feature vector; the target is extracted by sonar image processing algorithm, and multiple targets are tracked and counted by the nearest neighbor algorithm combined with the extended Kalman filter algorithm. The volume / area density of the water area swept by the sonar is statistically analyzed, and the number of fish is estimated by combining the water area.

[0056] A depth camera is used to collect water depth and turbidity data in real time and simultaneously mark the image time sequence, and capture images of fish schools, covering multiple morphological scenes, including but not limited to the side of the fish body and bending posture, to form a three-dimensional image model of the fish school; at the same time, when capturing fish schools, a supplementary light is used to dynamically adjust the light intensity according to the water depth, and an underwater light environment is adapted through a light attenuation compensation algorithm.

[0057] The three-dimensional image model of the fish school is associated with the acoustic signal output by sonar detection to construct a high-dimensional image dataset of the fish school with acoustic and optical capabilities. The dataset is then expanded using data augmentation methods, not limited to rotation, scaling, and cropping.

[0058] The preprocessed high-dimensional image dataset is input into the fish detection model, the detection algorithm is called to detect fish in the image, and the coordinate position of the fish detection box in the image and the corresponding prediction score are calculated.

[0059] After obtaining the bounding box position of the fish target, the target is cropped and then entered into the fish segmentation model for main body segmentation. After segmentation, underwater impurities that interfere with the image are removed, so that the image only contains the main recognition object, the fish target.

[0060] The segmented fish images are input into the YOLOv7 deep learning model to classify and identify fish species.

[0061] The beneficial effects of the above-described technical solution of the present invention are as follows:

[0062] 1. This invention integrates a three-band transducer, which can dynamically switch frequencies according to the depth of the fish school. Combined with sonar image processing and multi-target tracking algorithms, it can accurately extract the outline of the fish school target and count its number. This solves the problem that traditional single-band sonar cannot simultaneously achieve both detection resolution and distance in waters at different depths, and enables efficient estimation of fish numbers.

[0063] 2. This invention uses a depth camera to collect water depth and turbidity data in real time and marks the image time sequence to capture images of fish schools, covering various morphological scenes such as the side of the fish and their curved postures, forming a comprehensive three-dimensional image model of the fish school, which facilitates a comprehensive understanding of the fish school's morphological information. Combined with a light attenuation compensation algorithm to adapt to the underwater lighting environment, it ensures clear and stable fish school images at different water depths. Data datasets are expanded through data augmentation methods such as rotation, scaling, and cropping to improve data diversity.

[0064] 3. This invention uses the Faster R-CNN algorithm and U-Net model to achieve accurate detection and segmentation of fish targets, remove underwater impurities and interference, so that the image retains only the main fish, providing a clean target area for species identification, and solving the problem that fish targets are easily interfered with in complex underwater backgrounds.

[0065] 4. This invention is based on transfer learning to build a model, uses pre-trained weights to extract high-level semantic features, and combines batch normalization and incremental learning mechanisms. This not only improves the accuracy of fish species identification, but also enables the system to adaptively learn the features of new fish species, giving the system continuous expansion capabilities and meeting the needs of dynamic monitoring of fishery resources. Attached Figure Description

[0066] Figure 1 This is a schematic diagram of the principle of an intelligent device for investigating fish species and quantities according to the present invention;

[0067] Figure 2 This is a flowchart of an intelligent method for investigating fish species and quantities according to the present invention. Detailed Implementation

[0068] To make the technical problems, technical solutions and advantages of the present invention clearer, a detailed description will be given below in conjunction with the accompanying drawings and specific embodiments.

[0069] like Figure 1 As shown, the present invention proposes an intelligent device for investigating the species and quantity of fish, comprising:

[0070] The adaptive sonar detection module 101 integrates a three-band transducer of 30kHz, 200kHz and 400kHz, dynamically switches the frequency according to the depth of the fish school, and generates an echo intensity feature vector; it extracts targets through sonar image processing algorithms, uses the nearest neighbor algorithm combined with the extended Kalman filter algorithm to track and count multiple targets, counts the volume / area density of the water area swept by the sonar, and estimates the number of fish by combining the water area.

[0071] The 3D image acquisition module 102 uses a depth camera to acquire water depth and turbidity data in real time and simultaneously mark the image time sequence, and capture images of fish schools, covering multiple morphological scenes not limited to the side of the fish body and bending posture, to form a 3D image model of the fish school; at the same time, when capturing images of the fish school, the supplementary light is used to dynamically adjust the light intensity according to the water depth, and the light attenuation compensation algorithm is used to adapt to the underwater light environment.

[0072] Image preprocessing module 103 associates the three-dimensional image model of the fish school with the acoustic signal output by sonar detection, constructs a high-dimensional image dataset of the fish school with acoustic and optical methods, and expands the dataset through data augmentation methods not limited to rotation, scaling, and cropping.

[0073] The fish detection module 104 is used to input the preprocessed high-dimensional image dataset into the fish detection model, call the detection algorithm to detect fish in the image, calculate the coordinate position of the fish detection box in the image, and the corresponding prediction score.

[0074] The main body segmentation module 105, after obtaining the position of the rectangular box of the fish target, completes the cropping of the target and enters the fish segmentation model for main body segmentation. After segmentation, underwater impurities that interfere with the image are removed, so that the image only contains the main recognition object, the fish target.

[0075] The identification and classification module 106 inputs the fish image after subject segmentation into the transfer learning identification model to classify and identify the fish species.

[0076] In this embodiment, the echo intensity monitored by the adaptive sonar detection module 101 is calculated as follows:

[0077] I = 10log 10 (P)

[0078] Among them, P is the sonar reception power, and the frequency switching threshold is:

[0079]

[0080] 30 kHz (low frequency): suitable for deep water areas (h > 20 m), with strong penetration ability, long detection distance, but low resolution; 200 kHz (medium frequency): suitable for medium-depth water areas (10 m < h ≤ 20 m), balancing detection distance and resolution; 400 kHz (high frequency): suitable for shallow water areas (h ≤ 10 m), with high resolution, capable of capturing small fish schools or fine contours.

[0081] Among them, h is the depth of the fish school, c is the propagation speed of the sonar signal in water, Δt is the time delay from sonar signal emission to reception, f switch is the acoustic wave emission frequency, and the sonar detection module switches frequencies according to the above function according to the detected depth.

[0082] In this embodiment, the adaptive sonar detection module 101 extracts the target through a sonar image processing algorithm, specifically by extracting the fish school target contour through the Otsu threshold segmentation or Canny edge detection algorithm, where:

[0083] Otsu threshold segmentation automatically determines the optimal segmentation threshold by calculating the between-class variance of the image gray histogram, binarizes the sonar image, and separates the fish school target from background noise, which is suitable for water area environments with relatively uniform gray distributions.

[0084] Canny edge detection extracts the edge contour of the fish school through Gaussian filtering for denoising, calculating the gradient magnitude and direction, non-maximum suppression, and double-threshold screening, which is suitable for target boundary recognition under complex backgrounds.

[0085] Then, a method combining the nearest neighbor algorithm and the extended Kalman filter is adopted to achieve continuous tracking of the fish school target across frames, including:

[0086] For the target in the first-frame sonar image, calculate the spatial distance and feature similarity between the targets in adjacent frames through the nearest neighbor algorithm, and establish an initial tracking trajectory.

[0087] Then, use the extended Kalman filter algorithm to predict the next-frame target position based on the target motion model (such as a uniform or uniformly accelerated model), and then correct the prediction error through the measured data to update the target trajectory, solving the problem of trajectory loss in scenarios such as multi-target occlusion and cross-motion.

[0088] Finally, count the stable tracking target trajectories to generate the fish school quantity data within a unit time.

[0089] In this embodiment, the adaptive sonar detection module 101 calculates the volume / area density of the water area swept by the sonar, and estimates the number of fish by combining the water area. Specifically:

[0090] When the sonar scans the water area with a fan-shaped beam at a scanning angle of θ, a maximum detection range of R, and a depth of h, the scanned volume is... Horizontal scan area

[0091] Divide the number of tracked targets N by the scanned volume / area to obtain the volume density ρ. V =N / V or surface density ρ S =N / S;

[0092] Given the total area S of the water body total At that time, the total number N total =ρ S ·S toal Alternatively, the total volume can be estimated by combining the average depth calculation.

[0093] The 3D image acquisition module 102 uses a depth camera to acquire water depth and turbidity data in real time and simultaneously mark the image time sequence, and capture images of fish schools, covering multiple morphological scenes, including but not limited to the side of the fish body and bending posture, to form a 3D image model of the fish school; at the same time, when capturing images of the fish school, the supplementary light is used to dynamically adjust the light intensity according to the water depth, and the light attenuation compensation algorithm is used to adapt to the underwater light environment.

[0094] A depth camera is used to acquire water depth and turbidity data in real time and simultaneously label the image time sequence. When capturing images of fish schools, Time-of-Flight (TOF) technology is used to obtain the depth information of the fish school, which is then fused with the pixel information of the acquired fish school images to generate fish school image data with depth channels. Secondly, based on a multi-frame image sequence, the Structure for Motion Recovery (SFM) algorithm is used, combined with the labeled image time sequence, to construct a dynamic 3D image model of the fish school over a certain period of time, showing the posture changes of the fish school during swimming. Finally, by adjusting the field of view and resolution parameters of the depth camera, it is ensured that the depth data of the fish school can be acquired at a point cloud density of no less than a certain level at different underwater distances.

[0095] The underwater lighting environment is adapted through a light attenuation compensation algorithm. Specifically: First, the acquired fish images undergo illumination equalization processing using an adaptive histogram equalization algorithm, adjusting image contrast based on the light intensity distribution after light attenuation compensation. Second, an underwater light attenuation compensation network is introduced, constructing a light attenuation compensation model based on the U-Net architecture. The original image and light attenuation coefficient are input, and the image after light attenuation compensation is output using the formula:

[0096]

[0097] Calculate the optical attenuation compensation enhancement value, combine the image features before and after optical attenuation compensation, and fuse the compensated image features with the original image features through a feature fusion layer;

[0098] in, This represents the light attenuation compensation enhancement value at pixel position (i,j) on channel c. α represents the light intensity value at pixel position (i,j) on channel c in the original image. i,j This represents the local light attenuation estimation factor at pixel location (i,j). The color shift correction factor represents the pixel position (i,j) on channel c. The compensation perturbation factor represents the pixel position (i,j) on channel c. This represents the estimated attenuation value at pixel position (i,j) on channel c. This represents the local average value of the estimated attenuation value in the neighborhood centered at pixel position (i,j) on channel c.

[0099] In this embodiment, illumination equalization processing is performed on a frame of raw image data containing a clownfish collected by an underwater remotely operated vehicle (ROV) in a coral reef area at a depth of 30 meters in the East China Sea. Specifically, the image is divided into multiple non-overlapping 8x8 pixel sub-regions. For example, for the sub-region in the m-th row and n-th column of the image, its grayscale histogram is calculated, and a cropping threshold is set. This threshold is obtained by dividing the total number of pixels in the sub-region by the grayscale level 256, i.e., cropping threshold = (8*8) / 256 = 0.25. The frequency portion of the histogram exceeding 0.25 is cropped, and the number of cropped pixels (0.1) is recalculated. The gray levels in the histogram are evenly distributed to limit excessive amplification of local contrast. Then, the cumulative distribution function is calculated for the cropped and redistributed histogram, and a new gray-level mapping value is obtained for each pixel based on this function. For each pixel in the image, its final gray-level value is calculated by bilinear interpolation from the gray-level mapping values ​​of its four neighboring sub-regions, thus completing the illumination equalization. Next, the illumination-equalized image is input into a light attenuation compensation model built on the U-Net architecture. This model calculates a light attenuation compensation enhancement value for each pixel in each color channel. The calculation process is based on the formula. Define, where, This represents the light attenuation compensation enhancement value for the pixel located at coordinates (i,j) on color channel c. This value is a unitless scalar. This is the light intensity value of the original input image at channel c and position (i,j), with a value range of [0,255]. The subscripts i and j represent the row and column coordinates of the pixel in the image, respectively, and the superscript (c) represents the color channel, which can be one of the three channels: R, G, and B. The core operation logic in the formula is that the numerator is estimated by the local light attenuation factor α. i,j For the original light intensity Perform initial magnification, then apply a color shift correction factor. Additive corrections are performed, and the denominator is a regularization term, which compensates for the perturbation factor. Ensure the denominator is not zero, and estimate the decay value. Its neighborhood mean The absolute value of the difference is used to perceive local details of the image. When the attenuation characteristics of a pixel differ significantly from those of its neighbors, the denominator is increased, thereby suppressing the compensation intensity and avoiding over-enhanced artifacts in edge and texture regions. The absolute value operation ensures that the final output enhancement value is non-negative. The advantage of the formula is that it introduces a comparison with the attenuation characteristics of the neighborhood. This allows the compensation process to have local adaptability, enabling strong color and brightness restoration in smooth areas while providing gentle enhancement in textured areas, effectively preserving image details. To obtain the parameters in the formula, we will take the calculation of the pixel with coordinates (100, 150) in the red channel (R channel) as an example. First, we will directly obtain the light intensity value of that point through the image acquisition device. With a value of 50, the encoder-decoder structure of the U-Net model generates multiple feature maps, one of which corresponds to the output local optical attenuation estimation factor α. i,j This factor is negatively correlated with the local brightness of a pixel. It is assumed that the model outputs α based on the low-brightness characteristics of the point and its neighborhood. 100,150 The value is 0.3, set with reference to a large amount of underwater image data. Its range was experimentally determined to be between [0, 1.5]. The other output feature map corresponds to the color shift correction factor. Because the underwater environment causes the most severe attenuation of red light, the model will output a large positive offset for the R channel. Let the model output... The value is set to 20. This value is determined based on the average attenuation rate of different color channels. In experiments, the β value of the R channel is typically between [10, 50], which is the compensation perturbation factor. This value, output by the network, is used to increase computational stability. It is typically very small, set to ensure the gradient of the denominator term remains stable during training. Through training on tens of thousands of sample images, its value range is limited to [0.01, 0.2]. Here, the model output... The estimated attenuation value is 0.05. Also directly output by the network, it represents the model's estimate of the degree of light attenuation at that point, and its value is normalized to the [0,1] interval. Assuming the model output... It is 0.7, for Then calculation is required. The arithmetic mean of the estimated attenuation values ​​of all 25 pixels within a 5x5 neighborhood is shown in Table 1 below, representing a portion of the estimated attenuation values ​​for this 5x5 neighborhood.

[0100] Table 1. Estimated Attenuation Values ​​of Sample Pixel Neighborhood

[0101] coordinate Estimated attenuation value of R channel coordinate Estimated attenuation value of R channel (98,148) 0.72 (100,150) 0.70 (98,149) 0.71 (100,151) 0.68 (98,150) 0.70 (100,152) 0.69 (99,148) 0.73 (101,148) 0.75 (99,149) 0.72 (101,149) 0.73 (99,150) 0.71 (101,150) 0.72 (99,151) 0.69 (101,151) 0.70 (99,152) 0.69 (101,152) 0.71 (100,148) 0.74 (102,150) 0.74 (100,149) 0.72 (102,151) 0.72

[0102] As shown in Table 1, the local average value is calculated by summing the attenuation values ​​of all 25 points in the neighborhood (only a few examples are shown in the table) and then dividing by 25. The value is 0.71. Substitute all the above parameter values ​​into the formula for calculation. The result shows that the light attenuation compensation enhancement value for pixel (100, 150) in the R channel is 82.15. This value will be used as an input for subsequent feature fusion. Finally, the compensated and enhanced image features calculated for all pixels are fused with the original image features after illumination equalization. Specifically, the compensation enhancement value is... Compared with the original light intensity value Weighted summation is performed, with weight coefficients set based on a balance between preserving original image details and enhancing compensation effects. Through experimental testing, the original image weight w is set... 0rig The compensation image weight w is 0.4. comp If the value is 0.6, then the final output pixel value is Results outside the range [0, 255] are truncated to obtain the final image after optical attenuation compensation.

[0103] Image preprocessing module 103 associates the three-dimensional image model of the fish school with the acoustic signal output from sonar detection to construct a high-dimensional image dataset of the fish school using both acoustic and optical methods, specifically:

[0104] Bandpass filtering was performed on the sonar echo signals in the 30kHz, 200kHz and 400kHz bands respectively to remove environmental noise and frequency aliasing interference.

[0105] A GPS timing module is used to timestamp and synchronize sonar signals and optical images to ensure data alignment at the same time.

[0106] The sonar detection depth and optical acquisition depth are calibrated using an extended Kalman filter algorithm.

[0107] Extract the time-frequency and spatial features of sonar signals, as well as the geometric and texture features of three-dimensional images;

[0108] The extracted acoustic and optical features are weighted and fused, and the high-dimensional features after fusion are then reduced in dimensionality.

[0109] The merged dataset was hierarchically labeled with behavioral patterns, fish species attributes, and environmental parameters. Cross-modal consistency testing, sample equalization, and incremental learning validation were then performed on the dataset.

[0110] Then, the dataset was expanded using data augmentation methods, not limited to rotation, scaling, and cropping. These methods included: rotating the fish images after subject segmentation, with rotation angles ranging from -15° to 15° to simulate different swimming postures of fish underwater; scaling the images, with scaling ratios controlled between 0.8 and 1.2 times to accommodate changes in fish size at different shooting distances; using random cropping to crop regions containing complete fish bodies from the original images, with cropping sizes no less than 70% of the original image; and introducing Mosaic data augmentation, randomly stitching together multiple different fish images to simulate a multi-fish school scene.

[0111] In this embodiment, the fish detection module 104 calls a detection algorithm to detect fish in the image, calculates the coordinates of the fish detection box in the image, and the corresponding prediction score, specifically:

[0112] Multi-scale visual features of preprocessed images are extracted using the Faster R-CNN detection algorithm to capture the morphology, texture, and edge information of fish. The Faster R-CNN detection algorithm, relying on its backbone network containing convolutional and pooling layers, performs feature extraction operations on the preprocessed composite image. The convolutional layers use multiple convolutional kernels of different sizes (e.g., 3×3, 5×5) to slide and compute at different positions in the image, generating feature maps containing information on fish morphology (e.g., the curved features of the fish's outline), texture (e.g., the repetitive patterns formed by fish scales), and edges (e.g., the gray-scale transition boundaries between the fish and the background). Pooling layers (max pooling or average pooling) downsample the feature maps, reducing feature dimensionality and computational cost while preserving key features, and enhancing the translation invariance of features to ensure stable feature extraction even when the fish is located in different positions within the image. The Feature Pyramid (FPN) structure of Faster R-CNN is used to fuse the feature maps output from different levels of the backbone network (e.g., shallow feature maps retain more detailed texture, while deep feature maps contain more abstract semantic information). By performing operations such as upsampling and lateral connections, multi-scale feature maps are generated, enabling the algorithm to effectively capture features of fish of different sizes. For small fish, shallow high-resolution feature maps are used to identify subtle textures; for large fish, deep feature maps are used to understand the overall morphology, improving the algorithm's adaptability to detecting fish of different sizes.

[0113] Anchor boxes are generated based on feature maps, and their positions are adjusted through regression calculations to determine the coordinates of the fish detection boxes. Based on the fused multi-scale feature maps, multiple anchor boxes are generated at each pixel location in the feature maps according to preset anchor box sizes (e.g., 16×16, 32×32, 64×64, etc., to adapt to different fish sizes) and aspect ratios (e.g., 1:1, 1:2, 2:1, matching common fish morphologies), covering areas in the image where fish may exist. For each anchor box, its offset from the real fish bounding box is predicted using a regression model (including horizontal offset Δx, vertical offset Δy, width scaling factor Δω, and height scaling factor Δh). These offsets are used to adjust the position and size of the anchor boxes, making the adjusted detection boxes closer to the boundaries of real fish targets. The calculation formula can be expressed as:

[0114] x new =x anchor +Δx·ω anchor

[0115] y new =y anchor +Δy·h anchor

[0116] ω new =ω anchor ·exp(Δω)

[0117] hnew =h anchor ·exp(Δh)

[0118] Among them, (x anchor y anchor ) represents the coordinates of the top left corner of the anchor frame, (ω) anchor h anchor (x) represents the width and height of the anchor frame. new y new (ω) represents the coordinates of the top-left corner of the adjusted detection box. new h new This is to adjust the width and height of the detection frame to achieve accurate determination of the detection frame coordinates.

[0119] A classifier is used to calculate the category probability of the target within each detection box, outputting a predicted score and the corresponding fish category label. Specifically, for each detection box after regression adjustment, the feature vector of its corresponding feature map region is extracted and input into the classifier (usually a fully connected layer with a softmax function). Based on the feature differences between fish and non-fish, and between different fish species learned during pre-training, the classifier calculates the probability that the target within the detection box belongs to each category. For example, for a detection box containing crucian carp, the classifier will output the probability value of it belonging to the "crucian carp" category, as well as the probability values ​​of belonging to other fish categories (such as carp, grass carp, etc.) and non-fish categories. A probability threshold is set (e.g., 0.5, which can be adjusted according to actual detection needs), and detection boxes with a predicted probability greater than this threshold are selected, their corresponding category probabilities are used as the predicted score, and the corresponding fish category label (e.g., "crucian carp", "carp") is output. Detection boxes with predicted scores lower than the threshold are judged as false detections or background and are filtered out. Finally, the coordinates, predicted scores, and corresponding category labels of the fish detection boxes in the image are obtained, providing accurate target region information for subsequent fish segmentation and species identification.

[0120] In this embodiment, after obtaining the position of the fish target rectangle, the main body segmentation module 105 completes the target cropping and enters the fish segmentation model for main body segmentation. After segmentation, underwater impurities interfering with the image are removed. Specifically:

[0121] Based on the bounding box coordinates output by the fish detection module, the preprocessed composite image is cropped to extract the Region of Interest (ROI) containing the fish target, and the background area outside the bounding box is removed to reduce the scope of subsequent processing. The bounding box coordinates (x1, y1, x2, y2) output by the fish detection module need to be mapped back from the feature map scale of the detection model to the original pixel scale of the preprocessed image (because the detection model involves downsampling, the coordinate correspondence needs to be restored through deconvolution or interpolation). To avoid losing the fish body edges during cropping, the coordinates are elastically expanded using the following formula:

[0122]

[0123] Where W and H are the width and height of the preprocessed image, and α is the elastic expansion ratio coefficient, which controls the extent to which the rectangle expands outward from the boundary, and its value is usually between 0 and 1. (x1, y1) represents the coordinates of the upper left vertex of the detected fish target rectangle, which is used to locate the starting position of the rectangle in the image; (x2, y2) represents the coordinates of the lower right vertex of the detected fish target rectangle. The coordinates of the top-left vertex of the new rectangle are obtained after the elastic expansion calculation. It is the coordinate of the lower right corner vertex of the new rectangle obtained after elastic expansion calculation.

[0124] This operation ensures that the details of the fish's body edges (such as the tail and fins) are fully preserved, avoiding segmentation loss due to errors in the detection box boundaries.

[0125] Then, the cropped ROI image is input into a U-Net-based fish segmentation model. An encoder-decoder structure classifies the pixels in the image, generating masks for the fish body and background. The model's pre-training weights are derived from an underwater biological dataset. The encoder-decoder structure is as follows:

[0126] 1. Encoder Feature Extraction:

[0127] A 5-layer downsampling encoder is used, with each layer containing convolution (3×3 kernel, stride 2), batch normalization, and ReLU activation. The weights are initialized using pre-trained weights from an underwater biological dataset. The first layer extracts low-level features such as edges and textures (e.g., fish scale texture, fish body outline), and the fifth layer extracts high-level semantic features (e.g., overall fish morphology, category differentiation features).

[0128] 2. Decoder Feature Recovery:

[0129] The decoder gradually restores image resolution through upsampling convolutions (2×2 kernels, stride 2), while fusing feature maps from corresponding encoder layers (via skip connections) to address feature loss caused by scattering in underwater images. For example, fusing features from layers 1 and 5 can restore fish details while enhancing semantic classification capabilities.

[0130] The ROI image is binarized based on segmentation masks, and morphological operations are combined to eliminate small noise areas. Ultimately, the main fish pixels are preserved while removing interference from underwater suspended matter and aquatic plants, ensuring the output image contains only the fish target and its outline details. In the morphological operations, opening operations (erosion + dilation) are first performed, using a 3×3 rectangular kernel as the structuring element to eliminate small noise areas (such as bubbles and aquatic plant fragments; noise removal rate >90% for areas <10 pixels). Then, closing operations (dilation + erosion) are performed to fill in voids inside the fish caused by occlusion (such as small gaps at the mouth and gills), making the fish outline more complete.

[0131] In this embodiment, the identification and classification module 106 inputs the fish image after subject segmentation into the transfer learning identification model to classify and identify the fish species, specifically as follows:

[0132] The transfer learning model is constructed based on a pre-trained deep learning network, ResNet, to build a transfer learning recognition model. The pre-trained weights are derived from ImageNet or underwater fish datasets, utilizing their learned general visual features as basic parameters. When building the transfer learning model for fish recognition, the pre-trained deep learning network ResNet is chosen as the basic architecture. ResNet, with its residual connection structure, effectively alleviates the gradient vanishing problem in deep network training. It has been pre-trained on the large-scale general image dataset ImageNet, learning rich general visual features such as object edges, contours, and textures. Simultaneously, weights pre-trained on underwater fish datasets can also be used. These weights are optimized for the underwater environment and are more suitable for fish recognition scenarios. By using the general visual features learned in these pre-trained weights as basic parameters, the model possesses a certain feature extraction capability from initialization, eliminating the need to learn basic visual patterns from scratch. This significantly reduces the amount of data and computational resources required for model training, accelerates model convergence, and improves the foundation for fish feature recognition.

[0133] Image feature extraction and adaptation involves adjusting the segmented fish image to the model's input size. High-level semantic features are extracted using convolutional and pooling layers. Batch normalization and dropout regularization are introduced during the feature extraction stage. First, since the model has fixed input size requirements, the segmented fish image needs to be adjusted to the corresponding size. This step ensures the image can be successfully input into the model for subsequent processing. Then, the model's convolutional layers perform layer-by-layer feature extraction. The convolutional kernel slides across the image, capturing features at different levels, from simple edges and color information to complex fish morphology and texture combinations. The pooling layers downsample the feature maps obtained from the convolutions, reducing feature dimensionality and computational cost while preserving key features, allowing the model to focus more on global features. Introducing batch normalization in the feature extraction stage can standardize the features of each batch, making the feature distribution more stable, accelerating model training convergence, and improving the model's generalization ability. Dropout regularization randomly "shuts down" some neurons to prevent the model from over-relying on specific features, prevent overfitting, and allow the model to still stably extract effective features when facing images of fish in different underwater environments and with different postures.

[0134] The classification and output process involves mapping extracted features through the model's fully connected layers and a softmax classifier to generate probability distributions for each fish category. The model outputs the label for the fish category with the highest probability and its corresponding confidence score. When a new fish species is detected, the model supports incremental learning, achieving adaptive recognition of the new category by updating the classification layer weights online. After extracting high-level semantic features, the model's fully connected layers integrate and map these features, converting high-dimensional features into dimensions corresponding to the number of fish categories. Next, the softmax classifier processes the output of the fully connected layers, transforming it into probability distributions for each fish category. A higher probability value indicates a higher likelihood that the fish in the image belongs to that category. Then, the category with the highest probability is selected, and the corresponding fish category label, such as "crucian carp" or "carp," is output, along with a confidence score reflecting the reliability of the recognition result. When encountering a new fish species, the model supports incremental learning, eliminating the need to retrain the entire model. Instead, the classification layer weights are updated online, allowing the model to learn the feature patterns of the new fish species, thus adaptively recognizing new categories. This gives the model the advantage of continuously expanding its recognition capabilities and adapting to dynamic changes in fish species.

[0135] like Figure 2 As shown, this invention also proposes an intelligent method for investigating fish species and quantities, comprising the following steps:

[0136] S1. By integrating 30kHz, 200kHz and 400kHz three-band transducers, the frequency is dynamically switched according to the depth of the fish school to generate an echo intensity feature vector; the target is extracted by sonar image processing algorithm, and the nearest neighbor algorithm combined with the extended Kalman filter algorithm is used to track and count multiple targets, and the volume / area density of the water area swept by the sonar is statistically analyzed, and the number of fish is estimated by combining the water area.

[0137] S2. A depth camera is used to collect water depth and turbidity data in real time and simultaneously mark the image time sequence, and capture images of fish schools, covering multiple morphological scenes not limited to the side of the fish body and bending posture, to form a three-dimensional image model of the fish school; at the same time, when capturing fish schools, a supplementary light is used to dynamically adjust the light intensity according to the water depth, and the light attenuation compensation algorithm is used to adapt to the underwater light environment.

[0138] S3. Associate the three-dimensional image model of the fish school with the acoustic signal output by the sonar detection to construct a high-dimensional image dataset of the fish school with acoustic and optical methods, and expand the dataset by data augmentation methods not limited to rotation, scaling, and cropping.

[0139] S4. Input the preprocessed high-dimensional image dataset into the fish detection model, call the detection algorithm to detect fish in the image, calculate the coordinate position of the fish detection box in the image, and the corresponding prediction score.

[0140] S5. After obtaining the rectangular bounding box position of the fish target, the target is cropped and then entered into the fish segmentation model for main body segmentation. After segmentation, underwater impurities that interfere with the image are removed, so that the image only contains the main recognition object, the fish target.

[0141] S6. Input the fish image after subject segmentation into the transfer learning recognition model to classify and identify the fish species.

[0142] The intelligent fish survey device and method provided by this invention achieves accurate surveys of fish species and quantities through multi-module collaboration and technology integration. The working principle is as follows:

[0143] The device integrates low, medium, and high frequency transducers (30kHz, 200kHz, and 400kHz), which can automatically switch frequencies according to the depth of the fish school: low frequency is used in deep water to ensure detection range; medium frequency is used in medium-deep water to balance range and resolution; and high frequency is used in shallow water to capture fish details. The sonar scans the water area with a fan-shaped beam, extracts fish targets through image processing algorithms, counts the number of targets using tracking algorithms, and calculates the density based on the volume or area of ​​the scanned area, thereby estimating the total number of fish in the water.

[0144] A depth camera is used to capture real-time images of the fish school, simultaneously recording data such as water depth and turbidity. Depth information is obtained through time-of-flight technology, and a dynamic 3D model of the fish school is constructed by combining multiple frame image sequences, covering various postures such as the fish's sides and bent bodies. During filming, the supplementary lighting automatically adjusts its brightness according to the water depth. Combined with a light attenuation compensation algorithm, histogram equalization, image enhancement models, and feature fusion, the clarity of underwater images is improved, resolving the problem of uneven lighting.

[0145] The acoustic signals detected by sonar are correlated with the optical images acquired by the camera. First, the sonar signals are denoised and synchronized with the images in time. Then, the depth data of both are calibrated, and acoustic features (time-frequency, spatial) and optical features (geometric, texture) are extracted and fused to construct a high-dimensional dataset. At the same time, the data is expanded by rotating, scaling, cropping images, and stitching multiple images to simulate different fish postures and multi-fish school scenes, thereby enhancing data diversity.

[0146] Multi-scale feature extraction is performed on the preprocessed image using object detection algorithms to generate candidate boxes and adjust their positions, determining the specific coordinates of fish in the image and their probability of belonging to a category. Based on the detection results, regions of interest are cropped and input into a segmentation model for pixel-level classification, removing underwater suspended matter, aquatic plants, and other impurities, retaining only the main fish body to provide a clean image for subsequent recognition.

[0147] A recognition model is built based on a pre-trained deep learning network. It extracts high-level semantic features of fish using learned general visual features and incorporates regularization methods to prevent overfitting. A classifier generates a probability distribution of fish categories, outputting the species with the highest probability and its confidence level. When encountering new fish species, the model supports online parameter updates and adaptive learning of new category features, continuously expanding its recognition capabilities.

[0148] This invention's method automates the analysis from fish population statistics to species identification through a complete process: "sonar detection for quantity estimation → image acquisition and model building → data fusion and enhancement → target detection and localization → subject segmentation and denoising → transfer learning classification." The hardware collaboration between multi-band sonar and depth cameras, cross-modal processing of acoustic and optical data, and an intelligent model with incremental learning capabilities effectively solve problems such as poor depth adaptability, low image quality, and difficulty in identifying new species in traditional surveys, providing an efficient and accurate technical solution for fisheries resource management and ecological monitoring.

[0149] The above description represents the preferred embodiments of the present invention. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principles of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. An intelligent device for surveying fish species and quantities, characterized in that, include: The adaptive sonar detection module integrates three-band transducers of 30kHz, 200kHz, and 400kHz, dynamically switches frequencies according to the depth of the fish school, and generates echo intensity feature vectors. It extracts targets through sonar image processing algorithms, uses the nearest neighbor algorithm combined with the extended Kalman filter algorithm to track and count multiple targets, and counts the volume density or surface density of the water area swept by the sonar, and estimates the number of fish by combining the water area. The 3D image acquisition module uses a depth camera to collect water depth and turbidity data in real time and simultaneously mark the image time sequence, and capture images of fish schools, covering multiple morphological scenes such as the side of the fish body and bending posture, to form a 3D image model of the fish school; at the same time, when capturing fish schools, the supplementary light is used to dynamically adjust the light intensity according to the water depth, and the light attenuation compensation algorithm is used to adapt to the underwater light environment. The image preprocessing module associates the three-dimensional image model of the fish school with the acoustic signals output by sonar detection, constructs a high-dimensional image dataset of the fish school with acoustic and optical capabilities, and expands the dataset through data augmentation methods such as rotation, scaling, and cropping. The fish detection module is used to input the preprocessed high-dimensional image dataset into the fish detection model, call the detection algorithm to detect fish in the image, calculate the coordinate position of the fish detection box in the image, and the corresponding prediction score. The main body segmentation module, after obtaining the location of the fish target rectangle, completes the target cropping and enters the fish segmentation model for main body segmentation. After segmentation, underwater impurities that interfere with the image are removed, so that the image only contains the fish target as the recognition object. The identification and classification module inputs the fish images after subject segmentation into the transfer learning identification model to classify and identify fish species; The adaptation to the underwater light environment through the light attenuation compensation algorithm is specifically as follows: The acquired fish images were subjected to illumination equalization processing. An adaptive histogram equalization algorithm was used to adjust the image contrast based on the light intensity distribution after light attenuation compensation. An underwater optical attenuation compensation network is introduced. An optical attenuation compensation model is constructed based on the U-Net architecture. The input is the original image and the optical attenuation coefficient, and the output is the image after optical attenuation compensation. The formula is as follows: ； Calculate the optical attenuation compensation enhancement value, combine the image features before and after optical attenuation compensation, and fuse the compensated image features with the original image features through a feature fusion layer; in, Representative Channel Upper pixel position The optical attenuation compensation enhancement value, Represents the channels in the original image Upper pixel position The light intensity value, Represents pixel position Local optical attenuation estimation factor at the location, Representative Channel Upper pixel position Color shift correction factor, Representative Channel Upper pixel position The compensation disturbance factor. Representative Channel Upper pixel position The estimated attenuation value, Representative Channel The above pixel position It is the local average of the estimated decay values ​​in the central neighborhood.

2. The intelligent device for investigating fish species and quantities according to claim 1, characterized in that, When the depth camera acquires water depth and turbidity data in real time and simultaneously marks the image time sequence to capture images of fish schools to form a three-dimensional image model of the fish school, it adopts the following steps: The depth information of the fish school is obtained by using Time-of-Flight (TOF) technology and fused with the pixel information of the collected fish school image to generate fish school image data with depth channels. Based on a multi-frame image sequence, the SFM structure-of-motion (SOFM) algorithm is used, combined with the time sequence of the labeled images, to construct a dynamic 3D image model of a school of fish over a certain period of time, showing the posture changes of the school of fish during swimming. By adjusting the field of view and resolution parameters of the depth camera, it is ensured that depth data of fish schools can be collected at a point cloud density of no less than a certain level at different underwater distances.

3. The intelligent device for investigating fish species and quantities according to claim 1, characterized in that, The three-dimensional image model of the fish school is associated with the acoustic signal output from the sonar detection to construct a high-dimensional image dataset of the fish school combining acoustic and optical methods, specifically as follows: Bandpass filtering was performed on the sonar echo signals of the three frequency bands of 30kHz, 200kHz and 400kHz to remove environmental noise and frequency aliasing interference. A GPS timing module is used to timestamp and synchronize sonar signals and optical images to ensure data alignment at the same time. The sonar detection depth and optical acquisition depth are calibrated using an extended Kalman filter algorithm. Extract the time-frequency and spatial features of sonar signals, as well as the geometric and texture features of three-dimensional images; The extracted acoustic and optical features are weighted and fused, and the high-dimensional features after fusion are then reduced in dimensionality. The fused dataset is hierarchically labeled with behavioral patterns, fish species attributes, and environmental parameters. Cross-modal consistency testing, sample equalization processing, and incremental learning validation are performed on the dataset.

4. The intelligent device for investigating fish species and quantities according to claim 1, characterized in that, The data augmentation methods described above, which involve rotation, scaling, and cropping, specifically include: The fish image after subject segmentation is rotated, with the rotation angle ranging from -15° to 15°, to simulate different swimming postures of fish underwater. The images are scaled, with the scaling ratio controlled between 0.8 and 1.2 times, to accommodate changes in fish size at different shooting distances; Using a random cropping method, the region containing the complete fish body is cropped from the original image, with the cropped size not less than 70% of the original image; Mosaic data augmentation is introduced, which randomly stitches together multiple different fish images to simulate a scene with multiple schools of fish.

5. The intelligent device for investigating fish species and quantities according to claim 1, characterized in that, The adaptive sonar detection module calculates the volume density or surface density of the water area swept by the sonar, and estimates the number of fish by combining this with the water area. Specifically: The sonar scans the water area with a fan-shaped beam at a scanning angle of [missing information]. The maximum detection range is Depth is At that time, scan volume Horizontal scan area ; The target number of tracking counts Divide by the scanned volume or area to obtain the volume density. or areal density ; Total area of ​​water bodies At that time, the total number Alternatively, the total volume can be estimated by combining the average depth calculation.

6. The intelligent device for investigating fish species and quantities according to claim 1, characterized in that, The fish detection module calls a detection algorithm to detect fish in the image, calculates the coordinates of the fish detection boxes in the image, and the corresponding prediction scores, specifically: The Faster R-CNN detection algorithm is used to extract multi-scale visual features from preprocessed images to capture the morphology, texture, and edge information of fish. Anchor boxes are generated based on feature maps, and their positions are adjusted through regression calculations to determine the coordinates of the fish detection boxes. The classifier is used to calculate the class probability of the target in each detection box, and the predicted score and the corresponding fish category label are output.

7. The intelligent device for investigating fish species and quantities according to claim 1, characterized in that, The main body segmentation module, after obtaining the location of the fish target bounding box, performs target cropping and then enters the fish segmentation model for main body segmentation. The segmented image is then free of underwater impurities that interfere with the image. Specifically: Based on the bounding box coordinates output by the fish detection module, the preprocessed composite image is cropped to extract the region of interest (ROI) containing the fish target, and the background area outside the bounding box is removed to narrow down the scope of subsequent processing. The cropped ROI image is input into a U-Net-based fish segmentation model. The encoder-decoder structure classifies the pixels in the image and generates masks for the fish body and background. The model's pre-training weights are derived from an underwater biological dataset. The ROI image is binarized based on segmentation masking, and small noise areas are eliminated by combining morphological operations. Finally, the main pixels of the fish are retained and underwater suspended objects and aquatic plants are removed to make the output image contain only the fish target and its outline details.

8. The intelligent device for investigating fish species and quantities according to claim 1, characterized in that, The identification and classification module inputs the segmented fish image into the transfer learning identification model to classify and identify the fish species. Specifically: The transfer learning model is constructed based on the pre-trained deep learning network ResNet, where the pre-trained weights come from ImageNet or underwater fish datasets, using their learned general visual features as basic parameters. Image feature extraction and adaptation: The fish image after subject segmentation is adjusted to the model input size. High-level semantic features are extracted through the model's convolutional and pooling layers. Batch normalization and dropout regularization are introduced in the feature extraction stage. The classification and output process maps the extracted features through the fully connected layers and softmax classifier of the model to generate the probability distribution of each fish category, and outputs the label of the fish category with the highest probability and the corresponding confidence score. When a new fish species is detected, the model supports incremental learning and achieves adaptive recognition of the new category by updating the classification layer weights online.

9. A method for intelligently surveying fish species and quantities, characterized in that, Includes the following steps: By integrating a 30kHz, 200kHz, and 400kHz three-band transducer, the frequency is dynamically switched according to the depth of the fish school to generate an echo intensity feature vector; the target is extracted by sonar image processing algorithm, and multiple targets are tracked and counted by the nearest neighbor algorithm combined with the extended Kalman filter algorithm. The volume density or surface density of the water area swept by the sonar is statistically analyzed, and the number of fish is estimated by combining the water area. A depth camera is used to collect water depth and turbidity data in real time and simultaneously mark the image time sequence. It also captures images of fish schools, covering various scenes of fish body side and bending posture, to form a three-dimensional image model of fish schools. At the same time, when capturing images of fish schools, a supplementary light is used to dynamically adjust the light intensity according to the water depth, and a light attenuation compensation algorithm is used to adapt to the underwater light environment. The three-dimensional image model of the fish school is associated with the acoustic signal output by sonar detection to construct a high-dimensional image dataset of the fish school with acoustic and optical capabilities. The dataset is then expanded by data augmentation methods such as rotation, scaling, and cropping. The preprocessed high-dimensional image dataset is input into the fish detection model, the detection algorithm is called to detect fish in the image, and the coordinates of the fish detection boxes in the image and the corresponding prediction scores are calculated. After obtaining the bounding box position of the fish target, the target is cropped and then entered into the fish segmentation model for subject segmentation. After segmentation, underwater impurities that interfere with the image are removed, so that the image only contains the fish target as the identification object. The fish images after subject segmentation are input into a transfer learning recognition model to classify and identify fish species; The adaptation to the underwater light environment through the light attenuation compensation algorithm is specifically as follows: The acquired fish images were subjected to illumination equalization processing. An adaptive histogram equalization algorithm was used to adjust the image contrast based on the light intensity distribution after light attenuation compensation. An underwater optical attenuation compensation network is introduced. An optical attenuation compensation model is constructed based on the U-Net architecture. The input is the original image and the optical attenuation coefficient, and the output is the image after optical attenuation compensation. The formula is as follows: ； Calculate the optical attenuation compensation enhancement value, combine the image features before and after optical attenuation compensation, and fuse the compensated image features with the original image features through a feature fusion layer; in, Representative Channel Upper pixel position The optical attenuation compensation enhancement value, Represents the channels in the original image Upper pixel position The light intensity value, Represents pixel position Local optical attenuation estimation factor at the location, Representative Channel Upper pixel position Color shift correction factor, Representative Channel Upper pixel position The compensation disturbance factor. Representative Channel Upper pixel position The estimated attenuation value, Representative Channel The above pixel position It is the local average of the estimated decay values ​​in the central neighborhood.

Images