Fish school counting method based on multi-scale double-branch joint training network

Abstract

The invention discloses a fish school counting method based on a multi-scale double-branch joint training network, and belongs to the technical field of computer vision. The feature extraction capability is enhanced through density map estimation branches and a multi-scale receptive field fusion module (MRFSE), the target positioning precision is improved in combination with an improved SAM-DEBR + + detection framework, the challenges of large fish body scale change, serious shielding and the like are effectively solved, and low MAE (such as 16.97) and good cross-domain generalization performance are achieved on IOCfish5K and YoutbeFish-35 data sets. A document 2 provides an AGC double-layer power distribution strategy in which a battery energy storage cluster participates, through cluster aggregation, adjustable power calculation, state factor dynamic adjustment and an SOC and power constraint coupling mechanism, distribution of a power instruction between an energy storage cluster and a non-energy storage cluster is optimized, and AGC instruction tracking precision and service life prolonging of energy storage equipment are both considered. According to the two studies, the robustness and practicability of the system in a complex scene are remarkably improved through a multi-scale or dynamic constraint mechanism.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, specifically to a fish counting method based on a multi-scale dual-branch joint training network. Background Technology

[0002] In modern aquaculture, outdoor cage culture, with its advantages of efficient water resource utilization and favorable water exchange conditions, has been widely applied to marine and freshwater fish farming. Accurate and effective population counting of fish within marine ranching cages is crucial, providing indispensable data support for optimizing feeding strategies, monitoring fish health and welfare, estimating biomass yield, and assessing farming efficiency. Traditional manual counting methods are not only labor-intensive and time-consuming but also highly subjective and prone to significant errors, increasingly failing to meet the demands of modern large-scale aquaculture. Therefore, developing automated, non-invasive, and highly accurate underwater fish counting technology has become an urgent and valuable research focus in the field of precision marine aquaculture. This technology is essential for enabling data-driven decision-making, improving productivity, and ensuring the sustainability of aquaculture and economic development.

[0003] In recent years, with the rapid development of computer vision, non-contact fish counting methods based on computer vision have received widespread attention in aquaculture compared to traditional measurement methods that rely on manual labor or contact equipment. These methods are mainly divided into two categories: traditional image processing methods and deep learning-based methods. Traditional image processing methods were one of the earliest and most effective approaches to achieve non-contact fish counting, and they are mainly divided into region regression methods and segmentation-based methods. Summary of the Invention

[0004] To address the aforementioned shortcomings in the existing technology, this invention provides a fish counting method based on a multi-scale dual-branch joint training network.

[0005] To achieve the above-mentioned objectives, the technical solution adopted by this invention is as follows:

[0006] A fish counting method based on a multi-scale dual-branch jointly trained network includes the following steps:

[0007] S1. Construct a multi-scale bi-branch joint training network, which includes a main branch and an auxiliary branch. The main branch is used for density map estimation, and the auxiliary branch is used for target detection.

[0008] S2. Extract multi-scale features from the input image through the main branch and generate a density map of the fish population;

[0009] S3. Target detection is performed by combining the density map information of the main branch with the auxiliary branch to enhance feature extraction and localization capabilities;

[0010] S4. Train the network using a joint loss function to optimize the accuracy of density map estimation and object detection;

[0011] S5. Based on the trained network, count the fish in the input image.

[0012] Furthermore, the specific method for constructing the multi-scale dual-branch joint training network in S1 is as follows:

[0013] The main branch includes a multi-scale receptive field fusion module (MRFSE), which is used to fuse feature maps of different resolutions of the backbone network.

[0014] The auxiliary branch is based on the SAM-DETR++ framework and incorporates density map information from the main branch.

[0015] Furthermore, the specific method for extracting multi-scale features and generating density maps through the main branch in S2 is as follows:

[0016] The feature maps of the four stages of the backbone network are used as input, and the number of channels is unified through convolutional layers.

[0017] Dilated convolutional layers with different dilation rates are used to process feature maps in order to capture information about targets at different scales.

[0018] The SE module is introduced to dynamically adjust channel attention and enhance feature representation capabilities;

[0019] Finally, the feature maps are stitched together to generate a density map.

[0020] Furthermore, the specific method for target detection via auxiliary branches in S3 is as follows:

[0021] The backbone network feature map and the density map generated by the main branch are aligned using bilinear interpolation and then added together to form the encoder input;

[0022] The Transformer encoder is used to process input features, preserving image structural information and spatial prior knowledge.

[0023] Multi-scale feature fusion is achieved through a hierarchical decoder for target localization and classification.

[0024] Furthermore, the specific method for employing the joint loss function in S4 is as follows:

[0025] The density map estimation branch uses a combination of structural similarity loss and total variational loss;

[0026] The object detection branch uses focus loss for classification and L1 loss for localization;

[0027] The total loss function is the weighted sum of the loss functions mentioned above.

[0028] The present invention has the following beneficial effects:

[0029] (1) This invention proposes a JMPCNet density map estimation method to overcome the perspective effect. This method effectively solves the problem of the impact of severe occlusion and scale change on fish counting, so as to improve the accuracy of fish counting.

[0030] (2) In JMPCNet, the density map estimation branch is designed with a multi-scale receptive field fusion module to fuse features of different resolutions of the backbone network to improve the density map quality and multi-scale feature extraction capability; the detection branch further fuses the highest layer feature information on the basis of SAM-DETR++ to enhance the sensitivity to small targets in the distance and achieve more accurate multi-scale target detection.

[0031] (3) Extensive experiments were conducted on datasets from two different scenarios (underwater fish schools and human groups). The results show that the proposed JMPCNet not only performs well in the fish school counting task, but also achieves good performance in the human group counting task, demonstrating strong cross-domain generalization ability. Attached Figure Description

[0032] Figure 1 This is a schematic diagram of the fish counting method based on a multi-scale dual-branch joint training network according to the present invention.

[0033] Figure 2 This is a network diagram of JMPCNet according to an embodiment of the present invention.

[0034] Figure 3 Images of fish schools of different densities against a complex background, as shown in this embodiment of the invention.

[0035] Figure 4 The images show the visualization results of different methods from the embodiments of the present invention on two datasets.

[0036] Figure 5 The above are visualization results of other methods in embodiments of the present invention on two datasets.

[0037] Figure 6 The ablation study of the proposed MRFSE module is visualized in this embodiment of the invention.

[0038] Figure 7 This is a schematic diagram comparing the technical accuracy and sensitivity of various methods in the embodiments of the present invention.

[0039] Figure 8 The following are visualizations of the baseline, Gramformer, and JMPCNet results on two challenging population datasets, representing embodiments of the present invention. Detailed Implementation

[0040] The specific embodiments of the present invention are described below to enable those skilled in the art to understand the present invention. However, it should be understood that the present invention is not limited to the scope of the specific embodiments. For those skilled in the art, various changes are obvious as long as they are within the spirit and scope of the present invention as defined and determined by the appended claims. All inventions utilizing the concept of the present invention are protected.

[0041] A fish counting method based on a multi-scale dual-branch jointly trained network, such as... Figure 1 As shown, it includes the following steps:

[0042] S1. Construct a multi-scale bi-branch joint training network, which includes a main branch and an auxiliary branch. The main branch is used for density map estimation, and the auxiliary branch is used for target detection.

[0043] This embodiment proposes a multi-scale, dual-branch joint training network for fish counting, called JMPCNet. This network enhances the model's ability to extract and fuse features at different scales, effectively improving its robustness to scale changes and thus significantly improving its counting accuracy in real-world fish scenarios. The overall approach of this embodiment is as follows:

[0044] To address issues such as scale variations and height occlusion between fish in real-world fish school images, a JMPCNet algorithm is proposed, such as... Figure 2As shown. This network framework is a joint training framework, mainly divided into a main branch and auxiliary branches. First, in the main branch, a module called MRFSE is designed. This module receives feature maps of different resolutions from the four stages of the backbone and passes them through dilated convolutional layers with different dilation rates. This helps to capture targets of different resolutions. Based on this, an SE (Squeeze-and-Excitation) module is added at the end of each layer. This operation makes multi-scale feature extraction more targeted, especially for small target information in fish schools. Second, in the auxiliary branch, this embodiment introduces the SAM-DETR++ framework, which is an end-to-end target detection framework based on Transformer. Compared with the original... The DETR framework, through semantic alignment and hierarchical multi-scale fusion, can effectively handle targets of different resolutions. Unlike the original SAM-DETR++, this embodiment adds the last layer of features in the backbone network, making full use of the feature maps of the backbone network at different resolutions. In addition, to further improve DETR's ability to extract features of targets at different resolutions, this embodiment adds the density map information generated by the main branch to the encoding part of SAM-DETR. This helps guide the main branch to extract more general and discriminative features and improve the generalization ability of the main task (regressing target points), especially in sparse and dense images, balancing recall and precision, while optimizing computational efficiency.

[0045] S2. Extract multi-scale features from the input image through the main branch and generate a density map of the fish population;

[0046] Most existing networks rely on a single high-level feature to generate density maps. However, when these high-level features have low resolution, pixel information of small objects may be easily lost. Therefore, this embodiment fully utilizes feature maps with different resolutions from the four stages of the backbone network. As input features to the MRFSE module, these feature maps contain feature information of targets at different scales. To fuse feature information from targets at different scales, this embodiment processes the feature information from each stage through a convolutional layer, ensuring that each layer has 256 channels to facilitate subsequent feature fusion. To better generate density map information corresponding to the image, this module sets up three dilated convolutions with different dilation rates. These are used to process feature maps of different resolutions, such as Figure 2 As shown in the main branch. Therefore, the fused feature information Feature maps after dilated convolution The information is:

[0047]

[0048]

[0049] in, This indicates a convolution operation, and its channels are unified to facilitate subsequent element-wise addition. This represents a dilated convolutional layer with a different dilation rate, and ReLU represents the activation function. It's important to emphasize that this design is also intended to capture targets of different scales. Specifically, to obtain features of small, distant targets, this embodiment sets the dilation rate to a specific value. It is obtained from convolutional layers. To capture targets at general distances, the feature information of targets closer to the camera is finally obtained by a convolutional layer with r=1. This design emphasizes how branches with different hole ratios can capture information about targets at different scales in the image.

[0050] Subsequently, an SE module is introduced at the back end of each layer. This module is a lightweight attention module that enhances the model's sensitivity to important features by dynamically adjusting channel attention, and is often used in convolutional neural networks (such as ResNet and MobileNet). On the one hand, since each dilated convolutional layer only extracts corresponding local features—in other words, it is only responsible for extracting targets within a certain distance range—these features cannot utilize other information. Therefore, global average pooling is implemented for each channel of each feature. It can be defined as:

[0051]

[0052] in Indicates the first Feature map of each channel H represents the spatial dimension, where H is the height and W is the width. This further enhances the feature representation capability of each channel by fully utilizing the information after pooling and the correlation between channels. Specifically, it learns the nonlinear relationships between channels through a bottleneck structure composed of two fully connected layers:

[0053]

[0054] in Represents the ReLU activation function. For the Sigmoid function, and For learnable parameters, This represents the compression ratio.

[0055] Finally, the feature information from each layer is concatenated. Therefore, the final generated density map is:

[0056]

[0057] S3. Target detection is performed by combining the density map information of the main branch with the auxiliary branch to enhance feature extraction and localization capabilities;

[0058] While the main branch already possesses a powerful multi-scale feature extraction module, it still has certain limitations. Because images contain other targets besides fish, such as corals and reefs, and these targets are often similar in color to fish, this interferes with density map generation, leading to inaccurate density map estimation. To avoid this problem, this embodiment introduces the DETR detection framework as an auxiliary branch to better detect targets. As is well known, the Transformer-based object detection model DETR has received widespread attention due to its end-to-end design. However, DETR suffers from slow training convergence, severely limiting its application in practical tasks. Research indicates that the root cause of this problem lies in the misalignment between object queries and encoded image features in the semantic space, resulting in a complex matching process and slow convergence. It is important to note that the initial DETR detection framework used the last layer of the backbone network as the input features for the encoder. However, this last layer feature information is often lost during the backbone network feature extraction process, leading to low detection accuracy. To prevent this phenomenon, this embodiment introduces the SAM-DETR detection framework, which is an improvement on Deformable DETR. Unlike the original DETR detection framework, the input of this network encoder is a multi-scale feature input, and this input is divided into two parts: one part comes from feature maps from different stages of the backbone network, let's call it... Another part comes from the density map information generated by the main branch, denoted as Because the feature maps and density maps from different stages of the backbone network have inconsistent shapes, bilinear interpolation is used to ensure that the size of each layer's density map image matches the width and height of its corresponding feature map. This allows the corresponding feature maps to be added together, thus more accurately guiding the network to generate the density map. The expression is as follows:

[0059]

[0060] in, This represents the final input of the SAM-DETR encoder. This represents bilinear interpolation. Furthermore, to improve the model's target recognition capability in blurred scenes, this embodiment employs a traditional Transformer encoder structure, directly... By inputting this encoder, the spatial density information provided by the density branch is made more full use, enhancing the localization capability of the regression branch center for fuzzy targets. Specifically, the encoder will... Each layer is flattened to form a one-dimensional feature vector, and then positional encoding information is added to form the fused feature:

[0061]

[0062] in, Indicates to Flatten This represents positional encoding. Furthermore, the DETR network structure consists of L identical stacked Transformer layers. Each layer contains a multi-head self-attention mechanism (MHSA) and a feedforward network (FFN). The computation process of the Lth layer can be represented as:

[0063]

[0064] The final output is the feature of the Lth layer. This feature preserves the structural information of the image and incorporates spatial prior knowledge provided by the input density branch, helping the regression branch to more accurately locate blurred targets. Furthermore, in the decoding part of the network, a hierarchical decoder is designed to achieve multi-scale feature fusion. The lower-level decoder processes low-resolution features for coarse localization, while the higher-level decoder processes high-resolution features for precise localization. Notably, this hierarchical design requires no additional parameters and helps improve the detection performance of Transformer-based detectors in the field of small target detection.

[0065] S4. Train the network using a joint loss function to optimize the accuracy of density map estimation and object detection;

[0066] To improve the quality and counting accuracy of the estimated density map, this embodiment introduces a structural loss function in the density map prediction branch. This loss function, by evaluating brightness, contrast, and structural similarity, better preserves the smoothness of the fish school's spatial distribution, thereby predicting a density map that more closely approximates the actual fish school distribution. Its definition is:

[0067]

[0068]

[0069] In the above formula, and These represent the actual density map and the predicted density map, respectively. In addition, there are... , , These represent the local mean, local variance, and local covariance, respectively. and The values ​​were set to 0.01 and 0.03 respectively. Furthermore, to improve the model's stability during training, this embodiment introduces the Total Variation (TV) loss function. ), and assign a coefficient to it. , Set it to 0.01. Finally, the linear combination of the two loss functions is used as the final loss function for this branch, as shown in the following equation:

[0070]

[0071] In the auxiliary branch, the training tasks are divided into classification and localization. After Hungarian one-to-one matching, the matched target and the actual ground truth are used as the loss for backpropagation. Specifically, in the classification task, this embodiment uses focal loss as the loss function for classification. In the location task, this embodiment directly uses... The function calculates the distance between points, and its expression is:

[0072]

[0073] in This represents the location information of the target point matched by the Hungarian algorithm, while For the actual target point location, This represents the total number of matched target points, which is equal to the number of true values. Then, the Euclidean distance difference between the corresponding target points is used as part of the backpropagation loss.

[0074] In summary, the final loss function expression for both branches is:

[0075]

[0076] S5. Based on the trained network, count the fish in the input image.

[0077] To improve the model's adaptability and robustness in real-world scenarios, this embodiment uses the publicly available datasets IOCfish5k and YoutubeFish-35 as the training dataset for the network and for subsequent testing. These datasets reflect issues such as lighting variations, occlusion interference, and complex backgrounds in underwater images, helping to improve the model's detection accuracy and robustness in practical deployments.

[0078] IOCfish5k: This dataset is a large-scale dataset specifically designed for fish counting tasks in underwater scenes. It contains 5637 high-quality underwater images, covering a variety of highly camouflaged marine life such as seahorses, stonefish, lionfish, and mimicking octopuses, with a total of 659,024 target center points annotated. The data was collected from 135 underwater videos on YouTube, with an image resolution of 1080×1920, and meticulously annotated by six professional annotators over approximately 1200 hours. About 1017 images in the dataset have extremely high target density (more than 200 targets), suitable for evaluating model performance in densely occluded scenes. The original data was divided into a training set (3137 images), a validation set (500 images), and a test set (2000 images).

[0079] YoutubeFish-35: This dataset consists of 35 high-quality HD video sequences manually selected and downloaded from YouTube. The video frame rate is 25-30fps, and the resolution is 1920×1080. The images contain various underwater scenes, such as oceans, lakes, and rivers, and cover various lighting conditions (such as murky water and low-light water). 150 frames of each video were selected for detailed annotation. The center point of each visible animal in the video was marked manually. The annotation process followed the IOCfish5K annotation specification, resulting in more than 159,000 annotation points. The labelme tool was used for annotation.

[0080] Based on the density of the targets, the density of the two datasets above is divided into three levels, as shown in Table 1: low density (<50), medium density (51-200), and high density (>200). Furthermore, Figure 3 It intuitively displays some images and the corresponding visualization results after annotation.

[0081] Table 1. Number of images in different density ranges in the two datasets.

[0082]

[0083] This embodiment uses Mean Absolute Error (MAE), Mean Squared Error (MSE), and Mean Absolute Normalized Absolute Error (NAE) as evaluation metrics to assess the accuracy of fish counting. In addition, the model parameters are used to reflect the complexity of the network. The calculation methods for MAE, MSE, and NAE are shown in the following formulas:

[0084]

[0085]

[0086]

[0087] Where N is the number of test images. Indicates the first The predicted number of fish in the test image. Indicates the first The true number of fish in the image. Lower MAE and MSE values ​​indicate better model accuracy and stability. However, MAE and MSE are directly affected by the fish count results, making it difficult to distinguish whether the error stems from undercounting or overcounting. Therefore, to more effectively reflect the degree of undercounting and overcounting, this embodiment uses NAE as the evaluation metric.

[0088] When training the network model, the batch size was set to 16, the training epochs and initial learning rate were set to 1500 and 0.00001 respectively, and Adam was used as the optimizer with a weight decay of 0.1. Dropout (p=0.3) was added to prevent overfitting. The experimental hardware platform was equipped with two independent NVIDIA GeForce RTX 4090D GPUs, each with 24GB of memory, for a total memory size of 48GB. The operating system used was Ubuntu 20.04 LTS, and it was configured with CUDNN 8.7.0, Python 3.8.20, and PyTorch 2.1.2 with CUDA 11.8.

[0089] To verify the superiority of the proposed method in this embodiment, comparative experiments with typical population counting methods were conducted on two datasets, including MSENet, SLOBNet, CCTrans, and MAN. The experimental results are shown in Tables 2, 3, and 4, respectively. The proposed method in this embodiment achieved the best results in MAE, MSE, and NAE metrics.

[0090] First, detailed results on the YouTubeFish-35 dataset are shown in Table 2. It is evident that the method described in this embodiment achieves the lowest scores in MAE, MSE, and NAE compared to other methods, at 40.25, 71.56, and 2.36 respectively. This demonstrates the significant advantage of the proposed method in counting tasks. However, it should be noted that this dataset was originally designed for video counting tasks, where images exhibit high similarity (small inter-frame differences). The model may over-rely on background or fixed scene features (such as lighting and viewpoint) for counting, thus failing to adequately adapt to the diversity of real-world scenes. Furthermore, repeated predictions of the same scene may lead to inflated performance metrics, failing to fully validate its generalization ability. Therefore, this embodiment introduces a second dataset, IOCfish5K, for further validation.

[0091] The IOCfish5K dataset, with its more complex background, effectively tests the model's generalization performance. On this test set (2000 images), as shown in Table 3, the method in this embodiment shows improvements over the baseline in several metrics, with MAE, MSE, and NAE reaching 16.97%, 38.71%, and 0.36 respectively. Furthermore, its performance is significantly better than the other five methods, including the baseline. A similar trend was observed on the validation set. It is important to emphasize that although the method used in this embodiment has a larger network parameter scale compared to the baseline, the improved version of the detr method used in this embodiment only requires two Transformer layers, and the parameter increase compared to the baseline (6 layers) is not significant. Further ablation experiments will be conducted to explore the impact of the number of Transformer layers on performance. In addition, apart from the baseline methods, the other methods generally have larger overall errors in counting accuracy compared to the method in this embodiment. On the one hand, this is due to the complexity of the dataset itself, especially the complex backgrounds and lighting conditions. Other methods are mostly designed for scenes with simple backgrounds and relatively uniform lighting, making it difficult to accurately distinguish foreground targets from interfering backgrounds on this dataset. On the other hand, the multi-scale feature learning mechanism introduced in this embodiment demonstrates good adaptability, indicating that effectively handling scale changes plays a crucial role in improving the accuracy of fish counting. Therefore, the network structure proposed in this embodiment, which uses a collaborative design of main branches and auxiliary branches, outperforms existing methods in all three metrics: MAE, MSE, and NAE, demonstrating its effectiveness.

[0092] To further demonstrate the advantages of the proposed method, this embodiment also provides a partial predicted density map and compares it with the actual density map and other methods (especially IOCFormer). Figure 4 As shown in the red box, the method in this embodiment can more accurately locate the fish swarm targets to be counted. For example, in the ground truth of the first row, the red box contains 4 targets, which the baseline failed to detect, while the method in this embodiment successfully detected the corresponding targets. Figure 4 In the third line, this embodiment shows that although the counting results of the Baseline and the method of this embodiment are close to the true values, it should be noted that the density map generated by the Baseline is somewhat blurry in terms of the fish's location; at the same time, the Baseline misses small targets that are severely occluded in the distance, while the method of this embodiment can still accurately capture them. Figure 4In the example, the target in the yellow box in the second row was missed in the baseline, but was correctly identified by the method in this embodiment. This phenomenon also exists in the YoutubeFish-35 dataset (as shown in the red box in the fifth row). Finally, it should be emphasized that the reason why other methods have low accuracy in counting tasks is mainly due to the inaccurate identification of the target and background, which leads to a larger counting error. In addition, these methods are mostly suitable for simple and uniform environments, while actual underwater aquaculture scenarios are complex and varied, so their generalization ability is limited and they are difficult to cope with the diverse challenges in real-world applications. Figure 5 The visualization results of different methods on two datasets are shown. The red numbers in the upper left corner represent the counting results, the red boxes represent the detection effect of small targets at a distance, and the yellow boxes represent the detection effect of targets that are highly occluded.

[0093] Table 2 shows a comparison with state-of-the-art methods on the YouTubeFish-35 dataset.

[0094]

[0095] Table 3 compares the results with state-of-the-art methods on the IOCfish5K (test) dataset.

[0096]

[0097] Table 4 shows the comparison with state-of-the-art methods on the IOCfish5K (test set) dataset.

[0098]

[0099] To verify the effectiveness of the module proposed in this embodiment, various ablation experiments were conducted, and the impact of each part on the final counting accuracy was evaluated. Since this embodiment is an improvement upon the baseline, it focuses on comparing the performance of the improved network with the baseline. Considering the high complexity of the IOCFish5K (Test) dataset, which is more convincing, this embodiment chose to conduct experiments on the IOCFish5K (Test) dataset.

[0100] In the main branch, this embodiment designs a multi-scale receptive field module (MRFSE). This module receives output features from the four stages of the backbone network (ResNet50) to obtain a multi-scale density map, ensuring that feature information of targets of different sizes is extracted as much as possible. Based on this, this embodiment replaces the main branch with the MRFSE module of this embodiment on the baseline, and then uses the evaluation method described above to evaluate the estimation results. As shown in Table 5, MAE, MSE, and NAE all decrease, which indicates that the network design of this embodiment is effective; secondly, this embodiment can also be achieved through... Figure 6 It was observed that the method in this embodiment performs better in capturing small targets at a distance, while the baseline is only sensitive to close targets and not to distant targets. For example, in the case of a high-density school of fish, there are some small targets at a distance that are difficult to spot even with the human eye, but the method in this embodiment can still capture these small targets (such as...). Figure 6 (As shown in the red box) It should be emphasized that although some extremely distant targets are still not detected, there is still a significant improvement compared to the baseline.

[0101] In the baseline framework, the detection module uses the Conditional DETR model, which relies solely on a single-scale feature map as input. This design is prone to losing fine-grained features representing small targets during feature extraction, ultimately affecting counting accuracy. To demonstrate the contribution of the improved SAM-DETR++ detection model in this embodiment, this embodiment replaces the original Conditional DETR detection module with the improved SAM-DETR++ model based on the baseline, to explore its impact on the final counting accuracy.

[0102] As shown in Table 6, this embodiment found that the improved SAM-DETR++ model only experienced a slight decrease in the MSE metric. This is because the framework primarily utilizes its deformable attention mechanism to more efficiently leverage the multi-scale features provided by the backbone to focus on key regions where distant small-scale targets are located. Its sparse and adaptive spatial sampling capability enhances the localization accuracy for small, blurred, or highly blended targets, thereby improving counting accuracy. Simultaneously, the introduction of multi-scale features provides richer contextual information, enabling the model to better understand the distribution of targets of different sizes in complex underwater environments. Therefore, despite minimal structural changes, the model's counting robustness in complex scenarios is significantly improved thanks to a more refined multi-scale feature fusion and perception mechanism. Compared with the first ablation experiment, the main branch plays a crucial role. Furthermore, as... Figure 7As shown, although the method in this embodiment has a slightly lower counting accuracy than the baseline, the baseline is less sensitive to changes in the target scale (e.g., Figure 7 (As shown in the red box). In contrast, the SAM-DETR++ detection method can effectively identify targets at different scales, although there are instances where background is misclassified as a target (e.g., ...). Figure 7 (As shown in the yellow box), but it still reflects that the method has good scale awareness. It should be emphasized that the density map generated by the baseline has a large degree of overlap between targets, while JMPCNet can reflect the position of the fish and reduce the degree of overlap between targets in the density map.

[0103] Table 6. Impact of the enhanced SAM-DETR++ module on counting accuracy.

[0104]

[0105] Since the DETR detection framework's transformer layer count significantly impacts detection performance, this embodiment modifies the transformer layer count in the auxiliary branch to explore the optimal value for the auxiliary branch's layer count. As shown in Table 7, this embodiment clearly demonstrates that the detection performance is optimal when the transformer layer count is two. However, as the layer count increases, the performance shows a downward trend. This is because as the network deepens, the number of parameters increases, and an increase in the number of layers inevitably leads to a sharp increase in the number of model parameters. A larger model capacity means it is more likely to memorize noise and specific patterns in the training data rather than learn general features, resulting in decreased performance on the validation set (i.e., poorer generalization ability).

[0106] Table 7. Effect of different L values ​​on counting accuracy

[0107]

[0108] To further verify the generalization ability and versatility of the JMPCNet model proposed in this embodiment, we tested it on two datasets that differ significantly from the underwater fish school scene but are still dense object counting tasks: the JHU-Crowd++ dataset and the NWPU-CROWD dataset. These two datasets contain highly dense crowd scenes with severe occlusion, complex background interference, and greater scale variations. Since crowds and fish schools differ fundamentally in appearance, texture features, and movement patterns, this experiment effectively verifies the representativeness of the features learned by the model and the universality of its architecture.

[0109] This embodiment uses IOCFormer as the baseline model and selects a representative work in the field of crowd counting (Gramformer, Lin et al., 2024) for comparison. Generalization experiments show that even in crowd scenarios, the model proposed in this embodiment still exhibits good counting performance. As shown in Table 8, the visualization results of Baseline, Gramformer, and JMPCNet on two challenging crowd datasets are presented. The first to third rows belong to the JHU-Crowd dataset, and the last row belongs to the NWPU dataset. The areas highlighted in red clearly show the density map estimation results obtained by different methods. On the JHU-Crowd++ dataset, this model achieves the best performance, reducing MAE and MSE by 5.5% and 8.0% respectively compared to Gramformer. On the NWPU-CROWD dataset, although not all metrics are optimal, this model significantly reduces the MSE by 42.9%, indicating that it still has strong generalization ability in high-density crowd scenarios. Furthermore, regarding the issue of large scale variations, it can be observed through visualization of the density map that the model in this embodiment can not only detect general targets nearby, but also effectively identify small targets at a distance (such as...). Figure 8 (As shown in the red box), this demonstrates that the designed multi-scale receptive field module effectively improves counting accuracy. It is worth noting that in the visualization results in the first row, although Gramformer achieves relatively accurate count values, its density map fails to accurately locate the target position, resulting in a significant superposition effect. In summary, the JMPCNet model proposed in this embodiment not only possesses strong generalization ability and high counting accuracy but also displays the location information of the detected target more accurately, outperforming other comparative methods.

[0110] Table 8 Performance comparison on the JHU-Crowd++ dataset

[0111]

[0112] Table 9 Performance comparison on the NWPU-CROWD dataset

[0113]

[0114] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0115] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0116] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0117] Specific embodiments have been used to illustrate the principles and implementation methods of this invention. The descriptions of the embodiments above are only for the purpose of helping to understand the method and core ideas of this invention. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this invention. Therefore, the content of this specification should not be construed as a limitation of this invention.

[0118] Those skilled in the art will recognize that the embodiments described herein are intended to help the reader understand the principles of the invention, and should be understood that the scope of protection of the invention is not limited to such specific statements and embodiments. Those skilled in the art can make various other specific modifications and combinations based on the technical teachings disclosed in this invention without departing from the spirit of the invention, and these modifications and combinations are still within the scope of protection of this invention.

Claims

1. A fish counting method based on a multi-scale double-branch joint training network, characterized in that, The method comprises the following steps: S1, constructing a multi-scale double-branch joint training network, which comprises a main branch and an auxiliary branch, wherein the main branch is used for density map estimation, and the auxiliary branch is used for target detection; S2, extracting multi-scale features of the input image through the main branch, and generating a density map of the fish swarm; S3, performing target detection through the auxiliary branch combined with the density map information of the main branch to enhance the feature extraction and positioning ability; S4, training the network by using a joint loss function to optimize the accuracy of density map estimation and target detection; S5, based on the trained network, counting the fish swarm in the input image.

2. The fish counting method based on the multi-scale double-branch joint training network according to claim 1, characterized in that, The specific method of constructing a multi-scale double-branch joint training network in S1 is: The main branch comprises a multi-scale receptive field fusion module MRFSE for fusing feature maps of different resolutions of the backbone network; The auxiliary branch is based on the SAM-DETR++ framework and fuses the density map information of the main branch.

3. The fish counting method based on the multi-scale double-branch joint training network according to claim 1, characterized in that, The specific method of extracting multi-scale features through the main branch and generating a density map in S2 is: The feature maps of the four stages of the backbone network are taken as input, and the channel number is unified through a convolution layer; Different dilated convolution layers are used to process the feature maps to capture information of targets of different scales; An SE module is introduced to dynamically adjust the channel attention and enhance the feature expression ability; Finally, the processed feature maps are spliced to generate a density map.

4. The fish counting method based on the multi-scale double-branch joint training network according to claim 1, characterized in that, The specific method of target detection through the auxiliary branch in S3 is: The backbone network feature map and the density map generated by the main branch are aligned and added through bilinear interpolation to form the encoder input; The input features are processed using a Transformer encoder to retain image structure information and spatial prior knowledge; Multi-scale feature fusion is realized through a hierarchical decoder to perform target positioning and classification.

5. The fish counting method based on the multi-scale double-branch joint training network according to claim 1, characterized in that, The specific method of using a joint loss function in S4 is: The density map estimation branch uses a combination of structural similarity loss and total variation loss; The target detection branch uses focal loss for classification and L1 loss for positioning; The total loss function is the weighted sum of the above loss functions.

Images