School counting method and system based on contextual aggregation and self-distillation

Abstract

The fish school counting method and system based on context aggregation and self-distillation disclosed by the application comprise: acquiring a fish school image; performing feature extraction on the fish school image to obtain an initial spatial feature map; extracting local context information and global context information from the initial spatial feature map; splicing the local context information and the global context information, and generating a density map through a density regression module, wherein the density regression module comprises multiple level regression blocks and a convolution layer, each regression block performs convolution and transposed convolution on input features, the convolutions of multiple regression blocks are cascaded, and after convolution of the last regression block, the features are convolved through the convolution layer to generate the density map, and after transposed convolution of each regression block, the features are input into a self-distillation module, in the self-distillation module, the features after transposed convolution of deep regression blocks are used to supervise the features after transposed convolution of shallow regression blocks; and obtaining a fish school counting result according to the density map. The accurate counting of the fish school is realized.

Description

Technical Field

[0001] This invention relates to the field of fish counting technology, and in particular to a fish counting method and system based on context aggregation and self-distillation. Background Technology

[0002] The statements in this section are merely background information related to the present invention and do not necessarily constitute prior art.

[0003] Accurate fish counting in aquaculture environments helps aquaculture managers monitor stocking densities at various growth stages and develop precise feeding strategies. Traditional fish counting relies mainly on manual sampling and direct counting, which is not only time-consuming and labor-intensive but may also harm the normal growth of the fish. Non-invasive fish monitoring methods based on machine vision can avoid disrupting the normal growth of fish and have gradually replaced traditional manual counting methods, and are widely used in aquaculture and marine environments.

[0004] Traditional image segmentation methods, such as edge detection and watershed algorithms, can be used to segment fish targets in acquired fish school images. The number of fish can then be estimated by combining the target spot count or quantifying the area occupied by the fish. However, this method suffers from problems such as susceptibility to background noise, difficulty in segmenting adhered or overlapping targets, and reliance on manual thresholding or feature extraction.

[0005] With the rapid development of deep learning technology, deep neural networks have also been widely used in the field of fish counting. Currently, fish counting methods based on deep neural networks in aquaculture are mainly divided into two categories: fish counting based on object detection and fish counting based on density estimation. Fish counting based on object detection uses object detection algorithms based on deep neural networks to identify and locate fish in images, and then estimates the fish count by calculating the number of bounding boxes. However, object detection-based algorithms are mainly suitable for low-density scenes, and it is difficult to accurately detect fish in dense situations where fish frequently overlap, resulting in inaccurate fish counting.

[0006] The density estimation-based fish counting method maps the fish school image to a corresponding density map, and then estimates the total number of fish in the image by integrating the density map. However, existing density estimation-based fish counting methods are often only applicable to aquaculture scenarios where the fish scale changes little and the background is uniform. When applied to complex deep-sea aquaculture fish counting, the following limitations still exist: (1) Deep-sea aquaculture cages cover a wide area, and the underwater camera has a large field of view. Due to camera perspective and the different distances of the camera relative to the fish swimming freely in the cage, the pixel scale of individual fish in the underwater fish school images obtained by deep-sea aquaculture cages varies densely over a wide range, from a few pixels to thousands of pixels. This extreme scale variation makes it difficult for fish counting models to extract general features from the image to accurately count the fish school. Although existing fish counting methods use multi-column deep networks with different convolutional kernel sizes to extract multi-scale features of fish, the features extracted from different columns are highly similar and introduce training burden; (2) Due to the change in light angle and the occlusion of the central column of the net cage, there is obvious uneven background lighting in the underwater fish images obtained from deep-sea aquaculture net cages, which causes some fish to be submerged in local background areas with high similarity. In this case, if only the neighboring information around the pixel is considered, i.e., the local context, it is difficult to identify fish in the low-contrast areas of the image; (3) Existing density estimation-based fish counting methods only use the last layer of the density map label to supervise the network, which easily leads to poor supervision of the shallow network and insufficient representation; resulting in the inability of density estimation-based fish counting methods to achieve accurate counting of fish. Summary of the Invention

[0007] To address the aforementioned problems, this invention proposes a fish swarm counting method and system based on context aggregation and self-distillation. It extracts local and global context information from fish swarm images, generates a density map based on the local and global context information, and introduces a self-distillation strategy when generating the density map, so that the final density map can contain more fish feature information, ensuring the accuracy of fish swarm counting.

[0008] To achieve the above objectives, the present invention adopts the following technical solution:

[0009] Firstly, a fish swarm counting method based on context aggregation and self-distillation is proposed, including:

[0010] Obtain images of the fish school;

[0011] Feature extraction is performed on the fish school image to obtain an initial spatial feature map;

[0012] Extract local and global contextual information from the initial spatial feature map;

[0013] The local and global context information is concatenated and a density map is generated through the density regression module. The density regression module includes multiple cascaded regression blocks and a convolutional layer. Each regression block performs convolution and transpose convolution on the input features. The convolutions of multiple regression blocks are cascaded. The features after the last regression block is convolved by the convolutional layer to generate the density map. The features after the transpose convolution of each regression block are input into the self-distillation module. In the self-distillation module, the features after the transpose convolution of the deep regression blocks are used to supervise the features after the transpose convolution of the shallow regression blocks.

[0014] The fish count results are obtained based on the density map.

[0015] Secondly, a fish counting system based on context aggregation and self-distillation is proposed, including:

[0016] Fish school image acquisition module, used to acquire images of fish schools;

[0017] The fish school image feature extraction module is used to extract features from fish school images to obtain an initial spatial feature map;

[0018] The context information acquisition module is used to extract local and global context information from the initial spatial feature map;

[0019] The density map generation module is used to concatenate local and global context information and generate a density map through the density regression module. The density regression module includes multiple cascaded regression blocks and a convolutional layer. Each regression block performs convolution and transpose convolution on the input features. The convolutions of multiple regression blocks are cascaded. The features after the last regression block is convolved by the convolutional layer to generate a density map. The features after the transpose convolution of each regression block are input into the self-distillation module. In the self-distillation module, the features after the transpose convolution of deep regression blocks are used to supervise the features after the transpose convolution of shallow regression blocks.

[0020] The fish counting module is used to obtain fish count results based on the density map.

[0021] Thirdly, an electronic device is proposed, including a memory and a processor, and computer instructions stored in the memory and running on the processor, wherein the computer instructions, when executed by the processor, complete the steps described in the fish swarm counting method based on context aggregation and self-distillation.

[0022] Fourthly, a computer-readable storage medium is proposed for storing computer instructions, which, when executed by a processor, complete the steps described in the fish swarm counting method based on context aggregation and self-distillation.

[0023] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0024] 1. This invention extracts local and global contextual information from fish school images, generates a density map based on the local and global contextual information, and introduces a self-distillation strategy when generating the density map, so that the final density map can contain more fish feature information and ensure the accuracy of fish school counting.

[0025] 2. This invention extracts local contextual information from the initial spatial feature map through a local multi-scale context aggregation module. This module includes multiple dilated convolutional layers. The initial spatial feature map is input into each dilated convolutional layer, and the output feature map of each dilated convolutional layer is also input into all subsequent dilated convolutional layers. The initial spatial feature map is concatenated with the output feature maps of all dilated convolutional layers to output local contextual information. This local contextual information not only covers the contextual information of a large receptive field but also covers this range in a very dense manner. A global context aggregation module extracts global contextual information from the initial spatial feature map to supplement the local contextual information, thereby enhancing the fish discriminative representation information. After concatenating the global and local contextual information, robust features of fish of different sizes and background lighting can be obtained. When these features are used for fish counting, the accuracy of fish counting is improved.

[0026] 3. This invention introduces a self-distillation strategy, which uses feature maps from deeper layers of the network to supervise feature maps from shallower layers, improving the representation learning of the network model without increasing inference time, and further improving the accuracy of fish counting.

[0027] Advantages of additional aspects of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description

[0028] The accompanying drawings, which form part of this application, are used to provide a further understanding of this application. The illustrative embodiments of this application and their descriptions are used to explain this application and do not constitute an undue limitation of this application.

[0029] Figure 1 The aquaculture cage disclosed in the embodiment;

[0030] Figure 2 This is a schematic diagram of the data acquisition system disclosed in the embodiment;

[0031] Figure 3 The underwater fish image in video 1 disclosed in the embodiment;

[0032] Figure 4 The underwater fish image in video 2 disclosed in the embodiment;

[0033] Figure 5Density map of the underwater fish school image in Video 1 disclosed in the embodiment;

[0034] Figure 6 The above is a diagram of the overall architecture of LGSDNet as disclosed in the embodiments.

[0035] Figure 7 The above is a diagram of the overall architecture of LCMAM as disclosed in the embodiments.

[0036] Figure 8 The above is a diagram of the overall architecture of GCAM disclosed in the embodiments.

[0037] Figure 9 This is a first low-density fish school image disclosed in the embodiment;

[0038] Figure 10 This is an image of a first medium-density fish school disclosed in the embodiment;

[0039] Figure 11 This is a first high-density fish school image disclosed in the embodiment;

[0040] Figure 12 Density map label for the first low-density fish school image disclosed in the embodiment;

[0041] Figure 13 Density map label for the first medium-density fish school image disclosed in the embodiment;

[0042] Figure 14 Density map label for the first high-density fish school image disclosed in the embodiment;

[0043] Figure 15 The first low-density fish count results obtained using a baseline are disclosed in the embodiment;

[0044] Figure 16 The first medium-density fish population count results obtained using a baseline are disclosed in the embodiment;

[0045] Figure 17 The first high-density fish count results obtained using a baseline are disclosed in the embodiment;

[0046] Figure 18 The first low-density fish population counting result obtained using LGSDNet is disclosed in the embodiment;

[0047] Figure 19 The first medium-density fish population counting result obtained using LGSDNet is disclosed in the embodiment;

[0048] Figure 20 The first high-density fish count result obtained using LGSDNet is disclosed in the embodiment;

[0049] Figure 21This is a second low-density fish school image disclosed in the embodiment;

[0050] Figure 22 This is an image of a second medium-density fish school disclosed in the embodiment;

[0051] Figure 23 This is a second high-density fish school image disclosed in the embodiment;

[0052] Figure 24 Density map label for the second low-density fish school image disclosed in the embodiment;

[0053] Figure 25 Density map labels for the second density fish school image disclosed in the embodiment;

[0054] Figure 26 Density map labels for the second high-density fish school image disclosed in the embodiment;

[0055] Figure 27 The second low-density fish population counting results obtained using MAN are disclosed in the embodiment;

[0056] Figure 28 The second medium-density fish population counting results obtained using MAN are disclosed in the embodiment;

[0057] Figure 29 The second high-density fish population counting results obtained using MAN are disclosed in the embodiment;

[0058] Figure 30 The example discloses a second low-density fish population counting result obtained using HDNN;

[0059] Figure 31 The second medium-density fish count result obtained using HDNN is disclosed in the embodiment;

[0060] Figure 32 The second high-density fish count result obtained using HDNN is disclosed in the embodiment;

[0061] Figure 33 The second low-density fish population counting results obtained using CSRNet are disclosed in the embodiment;

[0062] Figure 34 The second medium-density fish population counting results obtained using CSRNet are disclosed in the embodiment.

[0063] Figure 35 The second high-density fish population counting results obtained using CSRNet are disclosed in the embodiment.

[0064] Figure 36 The second low-density fish population counting result obtained using LGSDNet is disclosed in the embodiment;

[0065] Figure 37 The second medium-density fish population counting results obtained using LGSDNet are disclosed in the example.

[0066] Figure 38 The second high-density fish count result obtained using LGSDNet is disclosed in the embodiment;

[0067] Figure 39 The third low-density fish school image disclosed in the embodiment;

[0068] Figure 40 This is an image of a third medium-density fish school disclosed in the embodiment;

[0069] Figure 41 Density map labels for the third low-density fish school image disclosed in the embodiment;

[0070] Figure 42 Density map labels for the third high-density fish school image disclosed in the embodiment;

[0071] Figure 43 The third low-density fish population counting result obtained using LGSDNet is disclosed in the embodiment;

[0072] Figure 44 The third medium-density fish population counting results obtained using LGSDNet are disclosed in the example. Detailed Implementation

[0073] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0074] It should be noted that the following detailed descriptions are illustrative and intended to provide further explanation of this application. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains.

[0075] Example 1

[0076] Fish school counting is a fundamental task in deep-sea aquaculture. However, significant scale variations in fish within underwater images and non-uniform background illumination greatly limit the counting accuracy of fish schools in deep-sea aquaculture. To overcome these problems, this embodiment discloses a fish school counting method based on context aggregation and self-distillation, including:

[0077] Obtain images of the fish school;

[0078] Feature extraction is performed on the fish school image to obtain an initial spatial feature map;

[0079] Extract local and global contextual information from the initial spatial feature map;

[0080] The local and global context information is concatenated and a density map is generated through the density regression module. The density regression module includes multiple cascaded regression blocks and a convolutional layer. Each regression block performs convolution and transpose convolution on the input features. The convolutions of multiple regression blocks are cascaded. The features after the last regression block is convolved by the convolutional layer to generate the density map. The features after the transpose convolution of each regression block are input into the self-distillation module. In the self-distillation module, the features after the transpose convolution of the deep regression blocks are used to supervise the features after the transpose convolution of the shallow regression blocks.

[0081] The fish count results are obtained based on the density map.

[0082] In specific implementation, such as Figure 2 As shown, underwater cameras capture videos of schools of fish, and images of the fish school are obtained by extracting one frame from the video at predetermined frame intervals. The underwater camera captures videos with a resolution of 1920×1080 pixels and a frame rate of 25fps, with a predetermined frame rate of 15 frames per second.

[0083] This embodiment utilizes a deep neural network with local-global context aggregation and self-distillation for fishcounting in deep-sea aquaculture (LGSDNet) to generate density maps from fish swarm images.

[0084] like Figure 6 As shown, LGSDNet takes a fish school image as input and maps it to a high-quality density map in an end-to-end manner. Then, it integrates the density maps to obtain the number of fish. LGSDNet includes a feature extraction module (FEM), a local-global context aggregation module (LGCAM), a density regression module (DRM), and a self-distillation module (SDM).

[0085] In this process, the fish school image is used to extract features. The FEM takes the fish school image as input and the initial spatial feature map as output, and is constructed using the first ten layers of VGG16.

[0086] LGCAM takes the initial spatial feature map output by the FEM as input and aggregates the contextual information of the image from both local and global perspectives for fish diversity feature extraction, thereby capturing robust features of fish of different sizes and background lighting. LGCAM comprises two parallel branches: Local Multi-Scale Context Aggregation Module (LCMAM) and Global Context Aggregation Module (GCAM). LCMAM extracts local contextual information from the initial spatial feature map, while GCAM extracts global contextual information. LCMAM utilizes densely connected dilated convolutions with different receptive fields to extract and aggregate rich multi-scale local contextual information across a large receptive field in the initial spatial feature map. GCAM is used to aggregate global contextual information from the initial spatial feature map to supplement the local context and enhance fish discriminative representation. Finally, the features containing global information and rich local background information are concatenated as the output, resulting in robust features of fish of different sizes and background lighting.

[0087] Due to camera perspective and the varying distances of fish swimming freely in deep-sea aquaculture cages from the camera, fish of similar sizes exhibit significant differences in pixel scale in underwater fish school images. Furthermore, the large number of fish in deep-sea aquaculture schools results in denser scale variations. Therefore, this embodiment proposes LCMAM to utilize densely connected dilated convolutions with different receptive fields to aggregate rich multi-scale local contextual information over a wide receptive field, thereby effectively handling the significant scale variations of fish in the image.

[0088] like Figure 7 As shown, the local multi-scale context aggregation module takes an initial spatial feature map as input and includes multiple dilated convolutional layers. The dilation rates of these layers are all different, and the dilation rate increases with each layer. The initial spatial feature map is input into each dilated convolutional layer, and the output feature map of each layer is also input into all subsequent dilated convolutional layers. The initial spatial feature map is concatenated with the output feature maps of all dilated convolutional layers to output local context information.

[0089] Preferably, LCMAM comprises four dilated convolutional layers. LCMAM densely connects the initial input spatial feature map to four dilated convolutional layers with varying dilation rates, increasing with each layer. The output feature map of each dilated convolutional layer is concatenated with the initial input spatial feature map and the output feature maps of all preceding dilated convolutional layers before being fed into the next dilated convolutional layer. Finally, the initial input spatial feature map is concatenated with the output feature maps of all dilated convolutional layers to obtain the final local context information. In this way, the final integrated local context information not only covers a large receptive field but also covers this field in a very dense manner.

[0090] All dilated convolutions use a 3×3 kernel, with dilation rates of 2, 4, 6, and 8 for different layers. To control the number of network parameters and prevent excessive network growth, a 1×1 kernel convolutional layer is placed after the initial spatial feature map and before each dilated convolutional layer. This layer performs channel compression on the features input to the dilated convolutional layer, reducing the depth of the input feature map to half. The convolutional features are then input into the dilated convolutional layer. The number of channels in each layer is set to {128, 128, 128, 128}. Furthermore, a ReLU activation function follows each convolution.

[0091] LCMAM takes features generated by FEM as input and outputs refined features that contain rich local contextual information with different receptive fields and have the same spatial resolution as the input features.

[0092] This embodiment utilizes an efficient and lightweight Global Context Aggregation (GCAM) module to extract global context information from the initial spatial feature map, thereby capturing the long-range dependencies of all fish species, performing global context modeling, and enhancing fish features.

[0093] The global context aggregation module takes the initial spatial feature map as input, extracts the attention weight map that is irrelevant to the query from the initial spatial feature map, and multiplies the attention weight map with the initial spatial feature map to obtain the initial global context features; after compressing the initial global context features, it restores them to obtain the global context feature map; and after adding the global context feature map with the initial spatial feature map, it outputs the global context information.

[0094] Preferably, the first convolutional layer is used to compress the channels of the initial global context features, the compressed features are normalized, and the second convolutional layer is used to restore the channels of the normalized features to obtain the global context feature map. The number of channels in the global context feature map is the same as the number of channels in the initial global context features.

[0095] like Figure 8As shown, the global context aggregation module includes a global context modeling module, a bottleneck transformation module, and a feature fusion module. The global context modeling module first extracts query-independent attention weight maps from the initial spatial feature map using 1×1 convolutional layers and the SoftMax activation function. Then, by multiplying the weight maps with the corresponding points of the input initial spatial feature map, it models the global context features as a weighted average of all locations, obtaining the initial global context features. The bottleneck transformation module consists of two 1×1 convolutional layers, used to extract the channel dependencies of the global context features. Specifically, to reduce model complexity, a first 1×1 convolutional layer compresses the channels of the input initial global context features from c to c / r (r>1). A second 1×1 convolutional layer restores the channels of the compressed features from c / r back to c, obtaining the global context feature map. Preferably, the hyperparameter r is set to 4. Furthermore, layer normalization after the first 1×1 convolutional layer simplifies optimization and increases the model's generalization ability. Finally, the feature fusion module adds the global context feature map to each element of the input initial spatial feature map to capture long-range dependencies in the image, obtaining the final global context information, which is then output.

[0096] The local and global context information is concatenated and used as the output of LGCAM to obtain robust features of fish of different sizes and background lighting.

[0097] Input the output of LGCAM into the Density Regression Module (DRM) to generate a density map.

[0098] Specifically, the density regression module includes multiple cascaded regression blocks and a convolutional layer. Each regression block performs convolution and transpose convolution on the input features. The convolutions of multiple regression blocks are cascaded. After the last regression block is convolved, the features are convolved by the convolutional layer to generate a density map. The features after the transpose convolution of each regression block are input into the distillation module.

[0099] Preferably, the DRM includes three cascaded regression blocks and a convolutional layer with a 1×1 kernel. Each regression block contains a convolutional layer with a 3×3 kernel and a transposed convolutional layer with a 4×4 kernel. First, the features of the input regression block are extracted through the convolutional layers to obtain convolutional features. Then, the features of the input regression block are upsampled through the transposed convolutional layer to obtain transposed convolutional features. The convolutional layers of the three regression blocks are cascaded. The convolutional features obtained from the last regression block are convolved by the 1×1 kernel to obtain a density map. The transposed convolutional features of each regression block are input into the SDM, which is used to improve the representation learning of the network model. It directly uses the feature maps of deeper layers as targets to guide the learning of early layers of the network without increasing any additional inference time. Preferably, the number of feature channels in the convolutional layers and the transposed convolutional layers in the three cascaded regression blocks are 256, 128, and 64, respectively.

[0100] Existing methods only use density map labels to supervise the last layer of the fish counting network, which leads to insufficient optimization of the shallow layers and poor discriminative features. Inspired by the self-distillation strategy, this embodiment proposes SDM, which uses latent information from deep layers as supervision to guide the learning of shallow layers, and then in turn benefits the deep layers and the final density map prediction results, thereby enhancing the representation learning of the fish counting network without increasing inference time.

[0101] like Figure 6 As shown, LGSDNet extracts three output branches from the three regression blocks of the density regression module, namely three transposed convolutional features, and sends the three transposed convolutional features to the self-distillation module to supervise network optimization. The self-distillation module adjusts the features of all transposed convolutional features to match the channels and scales of the adjusted features. Then, the adjusted features obtained from the shallow regression blocks are used to simulate the adjusted features obtained from the adjacent deep regression blocks, so as to use the transposed convolutional features of the deep regression blocks to supervise the transposed convolutional features of the shallow regression blocks.

[0102] Specifically, in the self-distillation module, the bottleneck layer is first used to adjust the transposed convolutional features of different output branches, including scale adjustment and channel adjustment. Specifically, the 1×1 convolutional layer in the bottleneck layer is used first to match the number of channels of the transposed convolutional features between adjacent regression blocks. Since the size of the transposed convolutional features output by shallow regression blocks differs from that of deep regression blocks, bilinear upsampling is used to match the spatial size of the transposed convolutional features between adjacent regression blocks. Then, mean squared error (MSE) is used as the self-distillation loss function L between branches. dThis is used to measure the difference in adjusted features obtained between adjacent branches, so that the spatial information of the adjusted features obtained by the shallow regression blocks sequentially simulates the spatial information of the adjusted features obtained by the adjacent deep regression blocks. This achieves the goal of using the transposed convolutional features of the deep regression blocks to supervise the transposed convolutional features of the shallow regression blocks. Figure 6 The dashed lines in the diagram represent the simulation direction of knowledge distillation. The overall self-distillation loss function L for the self-distillation module is... SKD Defined as:

[0103]

[0104]

[0105] Where N represents the total number of images in a training batch, and M represents the number of regression blocks, set to 3. F m-1 and F m These represent the transposed convolutional features of the deep regression block and the shallow regression block, respectively.

[0106] The loss function L of the LGSDNet constructed in this embodiment consists of three parts: MSE loss L MSE Structural Similarity Index (SSIM) Loss L SSIM and self-distillation loss L SKD The model was optimized in the direction of minimizing L.

[0107] L = L MSE +L SSIM +L SKD (3)

[0108] MSE loss L MSE It is used to calculate the pixel-level Euclidean distance between the predicted density map and the density map label. The loss function L... MSE Defined as:

[0109]

[0110] Where N represents the total number of images in a training batch; X i and Y i θ represents the input image and the corresponding density map label, respectively; θ represents the model parameters learned by the proposed model, F(X) i ;θ) represents the density map of the LGSDNet prediction output.

[0111] SSIM is used to quantify the similarity of local structures between the predicted density map and the density map labels. The loss function L... SSIM Defined as:

[0112]

[0113]

[0114] Where, μ x and δ x These represent the mean and variance of the predicted density map, respectively; μ y and δ y δ represents the mean and variance of the labels, respectively; xy C1 represents the covariance of the density map label and the predicted density map; C2 and C1 represent the constant parameters for stability calculation.

[0115] Self-distillation loss is used to measure the difference between feature maps of deeper layers and feature maps of earlier layers in LGSDNet, and its definition is shown in Equation (1).

[0116] This embodiment constructs a dataset by acquiring existing fish school images, labeling them, and generating density maps. The dataset is then used to train and validate the constructed LGSDNet. The trained LGSDNet can accurately convert fish school images into density maps. The labeling process is as follows:

[0117] The center of each fish in each existing fish school image was manually labeled using MATLAB code. In each labeled underwater fish school image, if the center of each fish is located at pixel x... i There is an instance of a fish, which can be represented as an impact response function δ(xx). i Therefore, a fish school image H(x) labeled with N fish can be represented as:

[0118]

[0119] Where x represents a pixel in the image, x i This represents the location of the marker point for each fish in the image.

[0120] Then, the adaptive Gaussian kernel Convolving with the function H(x) generates a corresponding true density map label F(x) for each underwater image, expressed as follows:

[0121]

[0122] Where, σ i This represents the variance of the Gaussian kernel applied at position i. The label x represents each fish i The average distance to the k nearest neighbors, where k is set to 3. β is an adjustable parameter, set to 0.3. Figure 1 The original underwater image of the fish school is displayed. Figure 5The corresponding density map obtained by formula (8) is shown. A more distinct region in the density map indicates a higher density of fish in that region. Conversely, a more blurred region indicates a lower fish density.

[0123] During training, Adam was used as the optimizer for the model, with a learning rate of 1e-4 and a batch size of 1. LGSDNet was trained on the training dataset for 100 iterations to obtain sufficient parameter updates, and the parameters that performed best on the validation dataset were saved as the optimal model parameters.

[0124] After generating a density map from the fish images using the trained LGSDNet, the fish count results are obtained through the density map.

[0125] This embodiment also verifies the method disclosed in this embodiment, with experimental data collected from the "Deep Blue No. 1" aquaculture cage located in the Yellow Sea. Figure 1 As shown, the perimeter and height of the net cage are 180 meters and 30 meters, respectively. Adult Atlantic salmon are cultured inside the cage. The experimental data acquisition system is as follows: Figure 2 As shown, an underwater camera was deployed on the central pillar of the fish cage at an angle to the water surface to capture video of the fish swimming from bottom to top. To improve the diversity of the experimental data, underwater video data was collected at regular intervals in two time periods: June 2022 (Video 1) and December 2022 (Video 2). The data in this experiment were collected without affecting the normal growth of adult Atlantic salmon and do not involve animal ethics issues.

[0126] The underwater camera captured videos at a resolution of 1920×1080 pixels at a frame rate of 25fps. One frame was extracted from the video data every 15 frames to construct the dataset. To further reduce computation, the fish school images were resized to 1280×720 pixels. Furthermore, the resized fish school images were randomly cropped and horizontally flipped for data augmentation. Finally, 1200 images were obtained from video 1 and 300 images from video 2. The original underwater fish school images extracted from video 1 and video 2 are shown below. Figure 3 , Figure 4 As shown, due to adjustments in the camera's shooting angle and changes in various factors such as lighting, water turbidity, and the number and size of fish, there are significant differences in the underwater images between Video 1 and Video 2.

[0127] Fish were labeled in 1500 acquired images, and corresponding density maps were generated. Two fish counting datasets, designated Dataset 1 and Dataset 2, were constructed using underwater images obtained at different acquisition times and their corresponding density maps. Dataset 1 contains 1200 images and corresponding density map labels. The number of adult Atlantic salmon ranges from [101, 259], totaling 206,400 fish. Dataset 1 was randomly divided into three groups: 1000 data pairs for training, 100 data pairs for validation, and 100 data pairs for testing. Dataset 2 contains 300 images and corresponding density map labels. The number of adult Atlantic salmon ranges from [150, 218], totaling 53,447 fish. 200 data pairs were used for training, 50 for testing, and 50 for validation. Comparative experiments were conducted on Dataset 1, which has sufficient data to validate the effectiveness and superiority of LGSDNet. The generalizability of LGSDNet was validated using Dataset 2.

[0128] To quantitatively demonstrate the effectiveness of the proposed method, mean absolute error (MAE), root mean square error (RMSE), and mean absolute percentage error (MAPE) were used to evaluate the performance of the method in this embodiment. MAE is one of the most widely used evaluation metrics in the field of target counting. It measures the average difference between the predicted and actual fish populations, representing the accuracy of the prediction. RMSE is sensitive to extreme values ​​and is used to examine the robustness of the model. MAPE considers not only the difference between the predicted and actual fish populations but also the ratio of this difference to the actual fish population in the image, making it a more comprehensive evaluation metric. The smaller the values ​​of MAE, RMSE, and MAPE, the higher the counting accuracy. The definitions of MAE, RMSE, and MAPE are as follows:

[0129]

[0130]

[0131]

[0132] Where K is the total number of test images, y i It represents the actual number of fish in the i-th image. It represents the predicted number of fish in the i-th image.

[0133] In the proposed method of this embodiment, LGCAM is used to capture local-global contextual information in images, combining LCMAM and GCAM. LCMAM is used to aggregate dense local multi-scale contextual information over a wide receptive field. GCAM is used to model global context and capture long-range dependencies in images. Furthermore, SDM is utilized to further enhance the network's representation learning, which does not require additional inference time. To demonstrate the effectiveness of LGCAM and SDM, these submodules are progressively combined and compared on the test dataset of Dataset 1. The baseline method represents a fish swarm counting network consisting of FEM and DRM. Baseline+LCMAM and Baseline+LGCAM add one LCMAM and one LGCAM to the baseline, respectively. Baseline+LGCAM+SDM represents LGSDNet of this embodiment, which adds LGCAM and SDM to the baseline. To ensure a fair comparison, all three methods use the same hyperparameters and experimental settings.

[0134] Table 1 shows the quantitative results of different methods. It can be seen that the addition of all key modules contributes to improving the performance of deep-sea aquaculture fish counting. Compared to the baseline, the introduction of dense local multi-scale contextual information from LCMAM reduces MAE and MSE by 2.4 and 2.65, respectively, and MAPE by 1.39%. Subsequently, when global contextual information is further integrated into the proposed fish counting model, the counting performance is further improved. Baseline + LGCAM has a 3.34 lower MAE, a 4.11 lower MSE, and a 1.78% lower MAPE than baseline + LCMAM. This demonstrates the effectiveness of combining dense local multi-scale contextual information and global information in enhancing the feature representation of fish of different sizes and background illumination levels. Finally, benefiting from the supervision of shallow information by deep information in SDM, LGSDNet achieves the lowest MAE (5.47), MSE (7.39), and MAPE (3.32%). Compared to the baseline, LGSDNet reduces MAE and MSE by 6.8 and 7.6, respectively, and MAPE by 3.7%. Overall, these quantitative results show that the local-global context aggregation module and the self-distillation strategy can effectively improve the accuracy of deep-sea aquaculture fish population counting.

[0135] Table 1. Performance comparison of LGSDNet and its variants

[0136]

[0137] Furthermore, to evaluate the counting performance of LGSDNet across different fish population sizes, the test dataset of Dataset 1 was divided into three subsets according to increasing fish population size: a low-density fish population image subset, a medium-density fish population image subset, and a high-density fish population image subset. Table 2 shows the counting results achieved by LGSDNet on the subsets at three different density levels. It can be seen that LGSDNet performs well at each density level. LGSDNet achieves the lowest MAE and RMSE at the low density level (5.25 and 6.16 respectively) and the lowest MAPE at the high density level (2.74%). These results demonstrate that LGSDNet exhibits good stability when counting fish populations at different density levels.

[0138] Table 2. Experimental results of LGSDNet at different density levels

[0139]

[0140] To more clearly illustrate the effect of LGSDNet, Figure 7 This paper presents representative density map estimation and fish counting results to visually compare the fish swarm performance of LGSDNet and the baseline. Overall, both the baseline and LGSDNet effectively depict the overall spatial distribution of fish swarms at different densities. However, it can be seen that by integrating LCMAM, GCM, and SDRM into a unified framework, LGSDNet obtains a density map closer to the true label and has a smaller overall counting error. Specifically, for low-density underwater scenes with sparse fish distribution, both the baseline and LGSDNet effectively estimate the total number of fish. However, for fish at different scales in underwater images, LGSDNet estimates fish swarm density more accurately and with higher quality than the baseline. For more complex medium- and high-density underwater scenes with relatively dense fish swarms, LGSDNet's advantages in density map estimation and overall counting are more pronounced, such as... Figures 9-20 As shown, for dense areas with significant variations in fish size and areas with low background illumination, the baseline method incorrectly detects some fish and underestimates the regional density of the fish population. Furthermore, the density map estimated by the baseline is not smooth enough. In contrast, LGSDNet enhances fish features in these complex regions, accurately locates their spatial distribution, and estimates regional density.

[0141] Benefiting from the combination of local-global contextual information aggregation and a self-distillation strategy, LGSDNet effectively improves the performance of fish counting models on a variety of fish of different sizes and background lighting. However, inconsistencies in fish features caused by overlap, large-scale variations, and background lighting interference make it difficult for the baseline to distinguish between fish and non-fish. Overall, these experiments demonstrate that the method in this embodiment exhibits stability and robustness not only in sparse scenes with large-scale variations in fish populations and background interference, but also in dense scenes.

[0142] To further verify the effectiveness and superiority of the proposed LGSDNet, the model presented in this embodiment is compared with two existing density estimation-based fish counting networks and the classic crowd counting network CSRNet. The two existing density estimation-based fish counting networks include Hybrid Deep Neural Networks (HDNNs) and MAN, which use deep learning-based density estimation techniques to handle dense fish counting tasks. CSRNet is a single-column neural network widely used for crowd density estimation. To ensure a fair comparison, all three methods used the same hyperparameters and experimental settings.

[0143] Table 3 lists the quantitative comparison results of these methods to evaluate their performance. The results show that LGSDNet provides the lowest counting error and outperforms the two existing methods and CSRNet on all evaluation metrics. Compared to MAN and HDNN, MAPE is reduced by 7.96% and 7.22%, respectively. A reasonable explanation for this difference is that the shallow multi-column deep neural network architecture used in MAN and HDNN extracts highly similar features, making it difficult to match the large-scale variations in fish. Furthermore, they struggle to handle non-uniform background illumination in underwater images from deep-sea aquaculture. CSRNet uses a deeper single-column structure to improve feature extraction, achieving better counting performance than existing multi-column deep neural networks. However, it neglects local multi-scale and global contextual information in the image, as well as weak supervision in the shallow layers. In contrast, LGSDNet significantly enhances the network's ability to feature and count fish in deep-sea aquaculture by aggregating local multi-scale and global contextual information and employing a self-distillation strategy. LGSDNet achieves a MAE 6.33 lower, an MSE 6.73 lower, and a MAPE 3.36% lower than CSRNet. This demonstrates LGSDNet's superiority in counting fish populations in deep-sea aquaculture.

[0144] Table 3. Performance comparison of LGSDNet with other counting methods

[0145]

[0146] also, Figures 21-38Several representative density maps with different fish distributions and the counting results estimated by three methods are presented to visually compare the performance of different methods. Overall, all methods effectively predict the spatial distribution of fish at different densities. However, the density map estimated by LGSDNet is more consistent with the ground reality and has smaller counting errors. In the density maps of different density levels estimated by MCNN and HDNN, there are obvious falsely detected density points. This is likely because the feature representations of fish extracted by simple multi-column deep neural network architectures are unstable under the interference of large-scale fish variations and non-uniform background lighting in underwater images. Furthermore, the output density maps of MAN inevitably suffer from blurring and information loss due to their relatively low resolution (the length and width of the density maps estimated by MAN and DNN are 1 / 4 and 1 / 8 of the original underwater image, respectively). In contrast, CSRNet and LGSDNet significantly filter out falsely detected density points, outputting high-resolution density maps. However, LGSDNet provides a clearer density map with smaller counting errors. For more complex medium- and high-density underwater scenes, LGSDNet demonstrates even greater advantages in density map estimation and overall counting. Even in extremely dense areas with large-scale fish population variations and in areas with excessively high or low backgrounds, the fish population density estimated by LGSDNet remains close to the actual situation on land. This proves the effectiveness and stability of LGSDNet.

[0147] To further verify LGSDNet's generalization ability to unseen aquaculture scenarios, we conducted generalization experiments on dataset 2. Due to significant scene differences in color, underwater lighting, and fish scale between the underwater images in datasets 1 and 2, it is difficult to directly generalize a supervised learning-based LGSDNet trained on one scene dataset to another, which typically does not yield ideal fish counts. Therefore, to effectively generalize LGSDNet to a new data domain with unseen underwater images, we used the training dataset from dataset 2 to fine-tune LGSDNet. This is a typical transfer learning method, aiming to transfer general knowledge from dataset 1 to the new data domain in dataset 2. Specifically, LGSDNet used the model parameters trained on dataset 1 as initial weights and then retrained 100 times on the training dataset of dataset 2 (200 "image-density map" data pairs), with the same hyperparameters as mentioned in Section 4.1.

[0148] Quantitative experimental results show that LGSDNet achieves MSE, RMSE, and MAPE of 7.53%, 9.81%, and 4.32%, respectively, on the test set of Dataset 2. This demonstrates that LGSDNet can effectively generalize to unseen aquaculture scenarios by transferring knowledge from Dataset 1 to Dataset 2. Furthermore, visualization results are shown below. Figures 39-44 As shown, the finely tuned LGSDNet achieves satisfactory fish counting results and reliable high-resolution density maps in both relatively sparse and crowded regions of the image, even under conditions of large-scale fish population variations and uneven background illumination. Nevertheless, when transferring LGSDNet to different aquaculture scenarios, sufficient annotation of underwater images for each scenario is required. In the future, semi-supervised and unsupervised domain adaptation methods should be used to further reduce the annotation burden on unseen data domains.

[0149] Therefore, this embodiment discloses the method of LGCAM, which combines the parallel operations of LMCAM and GCAM to aggregate dense local multi-scale contextual information and global contextual information in images, enabling the network to extract robust feature representations of fish with different sizes and background lighting in images. A self-distillation strategy is introduced to extract knowledge from the deep layers of the network to the shallow layers, thereby enhancing the network's representation learning without increasing inference time. Extensive experiments on datasets collected from actual deep-sea aquaculture cages demonstrate the effectiveness and superiority of LGSDNet. Furthermore, LGSDNet is extended to different aquaculture scenarios through transfer learning, further validating its generalization ability.

[0150] Example 2

[0151] In this embodiment, a fish counting system based on context aggregation and self-distillation is disclosed, including:

[0152] Fish school image acquisition module, used to acquire images of fish schools;

[0153] The fish school image feature extraction module is used to extract features from fish school images to obtain an initial spatial feature map;

[0154] The context information acquisition module is used to extract local and global context information from the initial spatial feature map;

[0155] The density map generation module is used to concatenate local and global context information and generate a density map through the density regression module. The density regression module includes multiple cascaded regression blocks and a convolutional layer. Each regression block performs convolution and transpose convolution on the input features. The convolutions of multiple regression blocks are cascaded. The features after the last regression block is convolved by the convolutional layer to generate a density map. The features after the transpose convolution of each regression block are input into the self-distillation module. In the self-distillation module, the features after the transpose convolution of deep regression blocks are used to supervise the features after the transpose convolution of shallow regression blocks.

[0156] The fish counting module is used to obtain fish count results based on the density map.

[0157] Example 3

[0158] In this embodiment, an electronic device is disclosed, including a memory and a processor, as well as computer instructions stored in the memory and running on the processor. When the processor executes the computer instructions, it performs the steps described in the fish swarm counting method based on context aggregation and self-distillation disclosed in Embodiment 1.

[0159] Example 4

[0160] In this embodiment, a computer-readable storage medium is disclosed for storing computer instructions, which, when executed by a processor, complete the steps described in the fish swarm counting method based on context aggregation and self-distillation disclosed in Embodiment 1.

[0161] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit it. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the specific implementation of the present invention. Any modifications or equivalent substitutions that do not depart from the spirit and scope of the present invention should be covered within the scope of protection of the claims of the present invention.

Claims

1. A fish swarm counting method based on context aggregation and self-distillation, characterized in that, include: Obtain images of the fish school; Feature extraction is performed on the fish school image to obtain an initial spatial feature map; Extract local and global contextual information from the initial spatial feature map; The local and global context information is concatenated and a density map is generated through the density regression module. The density regression module includes multiple cascaded regression blocks and a convolutional layer. Each regression block performs convolution and transpose convolution on the input features. The convolutions of multiple regression blocks are cascaded. The features after the last regression block is convolved by the convolutional layer to generate the density map. The features after the transpose convolution of each regression block are input into the self-distillation module. In the self-distillation module, the features after the transpose convolution of the deep regression blocks are used to supervise the features after the transpose convolution of the shallow regression blocks. The fish count results are obtained based on the density map.

2. The fish counting method based on context aggregation and self-distillation as described in claim 1, characterized in that, Feature extraction is performed on the fish images using a feature extraction module; The feature extraction module takes the fish school image as input and the initial spatial feature map as output, and is constructed using the first ten layers of VGG16.

3. The fish counting method based on context aggregation and self-distillation as described in claim 1, characterized in that, Local context information is extracted from the initial spatial feature map using a local multi-scale context aggregation module; The local multi-scale context aggregation module takes the initial spatial feature map as input and includes multiple dilated convolutional layers. The initial spatial feature map is input into each dilated convolutional layer, and the output feature map of each dilated convolutional layer is also input into all dilated convolutional layers after the dilated convolutional layer. The initial spatial feature map is concatenated with the output feature maps of all dilated convolutional layers to output local context information.

4. The fish counting method based on context aggregation and self-distillation as described in claim 3, characterized in that, A convolutional layer is set before each dilated convolutional layer. This convolutional layer performs channel compression on the features input to the dilated convolutional layer, and the convolutional features are then input into the dilated convolutional layer.

5. The fish counting method based on context aggregation and self-distillation as described in claim 1, characterized in that, Global context information is extracted from the initial spatial feature map using the global context aggregation module; The global context aggregation module takes the initial spatial feature map as input, extracts the attention weight map that is irrelevant to the query from the initial spatial feature map, and multiplies the attention weight map with the initial spatial feature map to obtain the initial global context features; The initial global context features are compressed and then restored to obtain the global context feature map. The global context feature map is added to the initial spatial feature map to output the global context information.

6. The fish counting method based on context aggregation and self-distillation as described in claim 5, characterized in that, The first convolutional layer compresses the channels of the initial global context features, normalizes the compressed features, and then the second convolutional layer restores the channels of the normalized features to obtain the global context feature map. The number of channels in the global context feature map is the same as the number of channels in the initial global context features.

7. The fish counting method based on context aggregation and self-distillation as described in claim 1, characterized in that, The self-distillation module adjusts the features of all transposed convolutions to match the channels and scales of the adjusted features. Then, the adjusted features obtained from the shallow regression blocks are used to simulate the adjusted features obtained from the adjacent deep regression blocks, thereby using the transposed convolution features of the deep regression blocks to supervise the transposed convolution features of the shallow regression blocks.

8. A fish counting system based on context aggregation and self-distillation, characterized in that, include: Fish school image acquisition module, used to acquire images of fish schools; The fish school image feature extraction module is used to extract features from fish school images to obtain an initial spatial feature map; The context information acquisition module is used to extract local and global context information from the initial spatial feature map; The density map generation module is used to concatenate local and global context information and generate a density map through the density regression module. The density regression module includes multiple cascaded regression blocks and a convolutional layer. Each regression block performs convolution and transpose convolution on the input features. The convolutions of multiple regression blocks are cascaded. The features after the last regression block is convolved by the convolutional layer to generate a density map. The features after the transpose convolution of each regression block are input into the self-distillation module. In the self-distillation module, the features after the transpose convolution of deep regression blocks are used to supervise the features after the transpose convolution of shallow regression blocks. The fish counting module is used to obtain fish count results based on the density map.

9. An electronic device, characterized in that, It includes a memory and a processor, as well as computer instructions stored in the memory and running on the processor, which, when executed by the processor, perform the steps of the fish swarm counting method based on context aggregation and self-distillation as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, Used to store computer instructions, which, when executed by a processor, complete the steps of the fish swarm counting method based on context aggregation and self-distillation as described in any one of claims 1-7.

Images