High-precision fry counting method for generating density map by using limited training samples

The LDNet network with IMBDA data augmentation and a generalized loss function enables high-precision fish fry counting using limited training samples, addressing the challenges of data quantity and disturbance in existing methods, achieving superior accuracy and robustness.

CN119068510APending Publication Date: 2024-12-03ZHUHAI KEYIPU TESTING TECH CO LTD
View PDF 3 Cites 0 Cited by

Patent Information

Application Number
CN202411079858.1
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-07
Publication Date
2024-12-03

AI Technical Summary

Technical Problem

Existing deep learning-based fish fry counting methods require large amounts of training data and are prone to disturbances and high annotation costs, especially in high-density scenes, and are not suitable for high-precision counting.

Method used

A high-precision fry counting method using limited training samples, employing the LDNet network with a feature extraction module and density map generation module, combined with the IMBDA data augmentation strategy and a generalized loss function based on unbalanced optimal transportation cost to generate density maps.

Benefits of technology

Achieves high-precision counting with reduced data annotation costs and minimal disturbance to fish, while effectively handling complex conditions, demonstrating superior accuracy and robustness even with limited training samples.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119068510A_ABST
    Figure CN119068510A_ABST
Patent Text Reader

Abstract

The invention discloses a high-precision fry counting method for generating a density map by using limited training samples, which has the following beneficial effects: (1) data enhancement is performed on a training set and a verification set by adopting an I MBDA strategy to expand the training set and the verification set, so that the diversity of training data is increased; various complex conditions possibly encountered in the actual data acquisition process are simulated, high counting precision is achieved under limited training samples, the data marking cost is reduced, and invasion to fishes is reduced; (2) the LDNet network can well extract the feature information of the fry in various complex conditions by introducing channel attention; and in the LDNet network training process, different from pixel-level regression of an existing GT density map, loss constraint is carried out by using a loss function based on unbalanced OT to effectively supervise density map generation, so that the quality of the generated density map is better ensured, and a high-precision fry counting effect can be realized through the mode.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The invention relates to the technical field of fish fry counting, and in particular to a high-precision fish fry counting method using limited training samples to generate a density map. Background Art

[0002] In recent years, with the continuous growth of the global population and the rapid development of the economy, fish farming has developed rapidly around the world. Accurate fry counting not only helps fishery managers monitor stocking density, choose the best fishing time and improve economic efficiency, but also helps control bait feeding, reduce bait pollution and develop scientific breeding plans, and reduce the labor burden of the entire aquaculture process. Although fry counting plays an important role in aquaculture, manual counting is still the most commonly used method in the field of aquaculture counting, mainly including weighing method and cup measurement method. This invasive counting method has an impact on fish health and growth, and it is also impossible to perform large-scale counting. Therefore, non-invasive fry counting methods have received widespread attention and research. Non-invasive fry counting methods are mainly divided into acoustic counting methods and counting methods based on computer vision counting.

[0003] As a quantitative and non-destructive detection method, underwater acoustic technology has been widely used. Taniguchi Y et al. obtained the coordinate information of fish through multiple sonars and used a clustering-based method to estimate the number of fish. The visibility of visible light images underwater is very poor, especially in dim environments. Acoustic imaging technology provides another way to obtain underwater environmental information for underwater target recognition. The most commonly used acoustic imaging method is sonar-based imaging, and then image processing is used to identify and detect targets. Nowadays, dual-frequency identification sonar (DIDSON) and adaptive resolution imaging sonar (ARIS) have been widely used by researchers in the field of fish counting. Jing D et al. proposed a method based on dual-frequency identification sonar imaging to estimate the number of fish. This method uses the Sobel operator to extract the edges of fish from complex images, and uses a depth-first search algorithm to extract fish targets. Finally, the nearest neighbor algorithm combined with the extended Kalman filter is used to track multiple fish targets. Tarling P et al. used deep learning and density-based regression methods to count fish in low-resolution sonar images using a large wild mullet (Mugilliza) sonar dataset.

[0004] The acoustic fish counting method plays an important role in the field of underwater ecological monitoring. However, it can only provide the coordinate information of fish, and the sonar equipment is easily affected by environmental factors when propagating in the water, resulting in blurring or changes in the echo signal. In addition, the acoustic method has high requirements for equipment and expertise. Computer vision methods provide a new research approach. The fish counting method based on computer vision uses image processing and deep learning technology to achieve high-precision recognition and classification of fish. Compared with the acoustic method, the computer vision method has stronger real-time performance, can quickly collect and process data, and realize real-time monitoring and counting. In addition, the equipment used in the computer vision method is relatively portable and easy to deploy and operate in different water environments. The fish counting method based on image segmentation has attracted widespread attention from scholars. Hernández-Ontiveros, JM et al. segmented the image by manually selecting a suitable threshold and integrated the algorithm into an embedded ornamental fish counting device for counting small fish (0.5 to 2.3 cm) in aquariums under different lighting conditions. Zhang L et al. proposed a fish counting method based on image density grading and local regression. First, the connected fish target area was segmented, and then the image features of the connected area were extracted and redundant features were removed. Then the density level of the fish connected area image was graded. Finally, a fish quantity prediction model based on BP neural network was constructed for the connected area data set at each density level for counting.

[0005] In recent years, deep learning-based methods have been widely used in fish counting research. Deep learning can discover high-dimensional features and deep information, overcoming the shortcomings of manual feature selection in traditional methods. Fish counting methods based on deep learning are mainly divided into two categories: based on target detection and based on density map regression. The purpose of target detection is to determine the location and category of semantic objects. CaiK et al. proposed a fish detection and counting algorithm based on YOLOv3 and MobileNet. This method replaces the selection of fixed feature maps in the original YOLOv3 framework by optimizing the determination strategy of feature maps in the MobileNet backbone network; at the same time, in order to obtain better fish features, this method selects a smaller dataset containing 16 types of fish images as a sample for pre-training of the backbone network instead of ImageNet. In practical applications, even the detection of the same type of fish will encounter the problem of large scale changes. To solve this problem, Li Kai et al. improved Faster RCNN and applied it to the detection and counting of eels in recirculating aquaculture. They used ResNet50+FPN network as the backbone and integrated multi-scale features to solve the problem of small-scale target detection. They used k-means clustering to generate 15 anchor frames of different scales to adapt to the scale of eels, and used Soft-NMS instead of NMS to screen candidate frames to reduce missed detections in overlapping areas.

[0006] In sparse scenes, although the method based on target detection can achieve better counting accuracy, for high-density scenes, the method based on density map generation is more suitable. The method based on density map generation mainly uses deep learning technology to linearly map local pixel features. The output density map represents the density distribution of the input image, which can obtain more fine-grained counting results. This method first extracts features from the image, then generates a single-channel density map, and finally sums the generated density map to obtain the final statistical number. This method is widely used for counting dense objects. In recent years, researchers have explored the feasibility of applying methods based on density map generation to fish counting tasks.

[0007] For high-density scenes, although the method based on density map regression is more suitable, the existing deep learning-based fish fry counting method uses a large amount of training data. The cost of labeling this data is high, and there may be overlaps and occlusions in high-density scenes, making the labeling process more complicated and difficult. Moreover, when photographing fish for a long time to obtain a large amount of training data, it is often difficult to avoid causing a certain degree of disturbance to the fish. Summary of the invention

[0008] 1. Technical issues to be resolved

[0009] In view of the deficiencies of the prior art, the present invention provides a high-precision fry counting method for generating a density map using limited training samples, which can solve the above technical problems.

[0010] (II) Technical solution

[0011] To solve the above technical problems, the present invention provides the following technical solution: a high-precision fry counting method for generating a density map using limited training samples, comprising the following steps:

[0012] S1: Obtain a data set of fish fry images, and further divide the data set into a training set, a validation set, and a test set;

[0013] S2: Label the key points of the fish fry images in the training set and the validation set;

[0014] S3: Perform data augmentation on the training set and validation set to expand the training set and validation set;

[0015] S4: Input the data-enhanced training set and validation set into the LDNet network;

[0016] S5: Generate density maps corresponding to the fry images of the training set and the validation set through the LDNet network, wherein a loss function is used to impose loss constraints during the generation of the density map;

[0017] S6: Input the test set into the LDNet network to generate a density map corresponding to the fry images of the test set;

[0018] S7: performing fish fry point summation on each density map obtained in step S6 to obtain a corresponding predicted number of fish fry.

[0019] Preferably, the LDNet network includes a feature extraction module and a density map generation module; in step S5, the fry feature information of the fry image is extracted by the feature extraction module, and the density map generation module further generates a density map using the fry feature information.

[0020] Preferably, the density map generation module includes a convolutional layer and a channel attention module.

[0021] Preferably, the loss function is a generalized loss function based on unbalanced optimal transportation cost.

[0022] Preferably, in step S3, the IMBDA strategy is used to perform data enhancement on the training set and the validation set.

[0023] Preferably, the IMBDA strategy includes performing horizontal flipping, vertical flipping and / or horizontal and vertical flipping on the fry image.

[0024] Preferably, the IMBDA strategy includes performing the following on the fry image: Gaussian blur processing, Gaussian noise processing, salt and pepper noise processing, Gaussian motion blur processing, contrast increase, contrast decrease, brightness and contrast increase and / or brightness and contrast decrease.

[0025] Preferably, the high-precision fry counting method for generating a density map using limited training samples further comprises step S8: calculating the prediction accuracy according to the predicted number of fry and the true value of the number of fry.

[0026] Preferably, the data set includes pictures of fry with different lighting backgrounds and different shadow effects.

[0027] Preferably, the data set comprises pictures of fry at different fry density levels.

[0028] (III) Beneficial effects

[0029] Compared with the prior art, the present invention provides a high-precision fry counting method for generating a density map using limited training samples, which has the following beneficial effects: (1) The present invention increases the diversity of training data by performing data enhancement on the training set and the validation set to expand the training set and the validation set, simulates various complex conditions that may be encountered in the actual data collection process, and achieves high counting accuracy under limited training samples, reduces data annotation costs, and reduces disturbance to fish; (2) The LDNet network of the present invention can well extract the characteristic information of fry under various complex conditions by introducing channel attention; and in the training process of the LDNet network, different from the existing GT density map for pixel-level regression, the present invention uses a loss function for loss constraint to effectively supervise the generation of the density map, so as to better ensure the quality of the generated density map. In this way, a high-precision fry counting effect can be achieved. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 A flow chart of the steps of the high-precision fry counting method of the present invention using limited training samples to generate a density map;

[0031] Figure 2 These are pictures of fry at different fry density levels in the dataset;

[0032] Figure 3 Label the image for labelme;

[0033] Figure 4 This is a comparison chart of the effect of data enhancement on fish fry images;

[0034] Figure 5 This is the network structure diagram of LDNet;

[0035] Figure 6 This is the training loss change graph;

[0036] Figure 7 MAE and RMSE comparison charts for comparative experiments with different models;

[0037] Figure 8 Accuracy comparison chart for comparative experiments on different models;

[0038] Fig. 9 Output comparison chart of density maps for different models;

[0039] Fig.10 This is the MAE box plot of the comparative experiment;

[0040] Fig.11 This is the Accuracy box plot of the comparative experiment;

[0041] Fig.12This is a comparison chart of MAE and RMSE changes of IMBDA strategy ablation;

[0042] Fig.13 This is the Accuracy change diagram of the IMBDA strategy ablation;

[0043] Fig.14 This is a comparison chart of MAE and RMSE changes with the amount of training samples ablated;

[0044] Fig.15 This is the Accuracy change diagram of the training sample size ablation;

[0045] Fig.16 This is a graph of the prediction results of the present invention on the CARPK dataset and the PUCPR+ dataset;

[0046] Fig.17 This is the prediction result of the present invention on the Penaeus_1k dataset. DETAILED DESCRIPTION

[0047] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0048] The present invention provides a high-precision fry counting method for generating a density map using limited training samples, comprising the following steps:

[0049] S1: Obtain a dataset of fish fry images, and further divide the dataset into a training set, a validation set, and a test set.

[0050] Specifically, the dataset Fish Count-824 in this embodiment consists of 824 fish fry images, which are taken using a mobile phone under natural light conditions and have different lighting backgrounds and shadow effects (some fish fry images are as follows: Figure 2 The above dataset Fish Count-824 contains eight kinds of gradient images, where the gradient changes from low to high represent the changes in the density level of fish fry from low to high.

[0051] It can be understood that the training set and validation set are used for training and validation of the network model respectively, corresponding to the training phase of the model; while the test set is used for testing the network model, corresponding to the testing phase of the model.

[0052] S2: Label the key points of the fry images in the training set and the validation set.

[0053] In step S2, the open source graphic annotation software labelme can be used to annotate the key points of the fry head in the fry image to obtain various key points, or called: ground truth points, GT (ground truth) points, such as Figure 3 At the same time, after key point marking, the coordinates of each key point on the fry image, namely the GT point coordinates, can be obtained accordingly.

[0054] In this implementation, the training set contains 16 labeled images, of which there are two images under each gradient; the validation set contains 8 labeled images, of which there is one image under each gradient; the test set contains 800 unlabeled images (the true value of the number of fry in the test set is known), and in the test set, there are 100 images under each gradient. The overall situation of the dataset FishCount-824 is shown in Table 1 below.

[0055] Table 1 Dataset Fish Count-824

[0056]

[0057] S3: Perform data augmentation on the training set and validation set to expand the number of fish fry images in the training set and validation set, that is, to expand the number of training samples.

[0058] Since this embodiment only uses 24 labeled images for model training, it is a very small training data set compared to the aquatic counting research based on deep learning; in the case of such a small amount of training data, the overfitting problem will be more serious. Therefore, the present invention proposes a data enhancement strategy called IMBDA (Image Manipulation Based Data Augmentation) (abbreviated as IMBDA strategy) to perform data enhancement and expansion on the training set and the validation set. The IMBDA strategy uses strategies such as adding noise and image flipping to enhance the data, simulate the adverse conditions that may occur under a fixed viewing angle, increase the diversity of the training set and the validation set, reduce the overfitting problem, and improve the robustness and generalization ability of the model.

[0059] Specifically, the IMBDA strategy of this embodiment consists of 11 parts, including the following: 1. horizontal flipping, 2. vertical flipping, 3. horizontal and vertical flipping of the original picture (the picture of the fry with key points marked in the training set and the verification set), and the following: 4. Gaussian blur processing, 5. Gaussian noise processing, 6. Salt and pepper noise processing, 7. Gaussian motion blur processing, 8. contrast increase, 9. contrast decrease, 10. brightness contrast increase at the same time, and 11. brightness contrast decrease at the same time.

[0060] Through the application of the IMBDA strategy, the training set was expanded from the original 16 fry pictures to 576, and the validation set was expanded from the original 8 to 288 fry pictures. Table 2 shows the changes in the number of training set pictures and validation set pictures after the IMBDA strategy. Figure 4 The figure shows the changes of the fry pictures after the data enhancement strategy, where origin is the original picture and the number represents the above number corresponding to the IMBDA strategy. In addition, it should be understood that the object of data enhancement is the fry pictures that have been annotated with key points (that is, the GT point coordinates are known), so the pictures obtained after data enhancement do not need to be annotated with key points, and the new GT point coordinates can be directly obtained from the original GT point coordinates (for example, according to the flip angle of the picture), or the new GT point coordinates are the same as the original GT point coordinates (for example, Gaussian blur processing, contrast enhancement and other data enhancement strategies), that is, only 16 of the 576 fry pictures in the expanded training set above need to be annotated with key points, and the remaining 560 fry pictures obtained by data enhancement do not need to be annotated with key points, and the same applies to the verification set.

[0061] Table 2 Changes in the number of images after the IMBDA strategy

[0062]

[0063] The application of the IMBDA strategy of the present invention not only expands the data volume of the training set and the validation set, but also increases the diversity of the data; through operations such as flipping, noise processing and blur processing, various complex conditions that may be encountered in the actual acquisition process, such as illumination changes, background noise, etc., are simulated; in this way, the training model can better adapt and have stronger generalization ability when facing various uncertain factors in the real world.

[0064] S4: Input the data-enhanced training set and validation set into the LDNet network, that is, input the expanded training set, validation set fish fry pictures, and corresponding GT point coordinates into the LDNet network.

[0065] S5: Generate density maps corresponding to the fry images of the training set and the validation set through the LDNet network, wherein the loss function is used to perform loss constraints in the process of generating the density map.

[0066] The LDNet network is a neural network model for fry counting, which adopts an architecture based on density map generation. The LDNet network combines a feature extraction module and a density map generation module, and uses a Generalized Loss Function (or GL function) based on unbalanced OT (Optimal Transport) cost to supervise the density map generation process, so as to achieve accurate estimation of the number of fry. In step S5, the fry feature information of the fry image is extracted by the feature extraction module, and the density map is further generated by the density map generation module using the fry feature information.

[0067] The network structure of LDNet consists of two key parts: Feature Extraction module and Density Map Generation module. The overall structure of LDNet network is as follows: Figure 5 As shown in the figure, the details of each module are shown in Table 3. The feature extraction module uses the VGG19 network without the pooling layer and the fully connected layer as the front-end part to extract rich feature information from the input image. It consists of multiple convolutional layers and pooling layers, and uses multiple 3x3 convolutional kernels to stack the neural network to deepen the depth of the entire neural network and effectively capture the semantic information and local features of the image.

[0068] As the backend part of LDNet, the density map generation module is composed of the SEblock (channel attention module) proposed by Hu et al. and the convolutional layer in series to enhance the modeling ability of the network. Specifically, the density map generation module consists of three convolutional layers and three SEblock channel attention modules. The structure of SEblock is shown in Figure 5, see Table 4 for details. SEblock first performs adaptive global average pooling on the input feature map to compress the information of each feature channel; then the pooled global feature vector is processed through two fully connected layers: the first fully connected layer is used for dimensionality reduction to reduce the dimension of the feature vector; the second fully connected layer restores the feature vector to the original number of channels; the output feature vector is the weight corresponding to each feature channel, which is used to measure the contribution of the channel to the final feature representation. Finally, the features of each channel are multiplied by the weight of the corresponding channel to obtain an adaptively weighted enhanced feature map. Such operations are performed in series to further improve the accuracy and robustness of the density map generation module. The convolution layer receives the feature map weighted by SEblock for convolution and finally generates a density map.

[0069] Table 3 LDNet network structure details

[0070]

[0071] Table 4 SEblock details

[0072]

[0073] About the loss function:

[0074] This implementation uses a generalized loss function based on entropy regularized unbalanced optimal transport cost, i.e., a generalized loss function based on unbalanced OT (Optimal Transport) cost (or GL function) to supervise the density map generation process. First, define is the predicted density map, where a i For x i ∈R 2 The predicted density is n, where n is the number of pixels; the GT point map (i.e. the fish fry picture marked with GT points) is defined as Among them, y j is the position of the jth key point (GT point), m is the number of key point annotations, b j The number of fry corresponding to the key point mark. In the present invention, a key point mark corresponds to only one fry, i.e., b j ==1. Let a=[a i ] i Represented as a predicted density map, b = [b j ] j This is the GT point map.

[0075] The GL function is shown in the following formula (1):

[0076]

[0077] On the whole, is the transportation (transmission) cost matrix, where means x i The predicted density is moved to the GT point annotation y j The transportation cost of i ,y j ) is an adaptive perspective factor, mapped to a P is a transport matrix with a unique solution and a unique form, which transforms each position x in A i Assign to y in B j , to measure the transportation cost; the optimal transportation cost is obtained by minimizing the loss on P.

[0078] GL functions can be divided into the following four parts:

[0079] The first part is Among them, C is the transportation cost matrix and P is the transportation matrix. The first part aims to obtain the lowest transportation cost, encourages predicting density values ​​near the annotations (labels), and pushes the predicted density towards the GT point annotations during training.

[0080] The second part εH(P)=-Σ ij P ij logP ij It is the entropy regularization term, which is used on the transport matrix P to measure the degree of overlap of the transport matrix. It can reduce the invalid transmission inside the transport matrix and make the predicted density value closer to the true value mark point (GT point). A larger ε value will produce a non-compact predicted density map, and vice versa.

[0081] Part III D1(P1 m |a) is the representation of the predicted density map a and the constructed intermediate density map The pixel-by-pixel loss between , i.e., the GT density map, uses the L2 loss (see Equation 2 below), and D1 ensures that all predicted density values ​​have corresponding annotations, i.e., any predicted density that is not associated with an annotation will be penalized.

[0082]

[0083] Part IV D2 (P T 1 n |b) is the reconstruction annotation The point-wise loss between the ground-truth point annotations is calculated using the L1 loss (see Equation 3 below). The third and fourth parts are complementary - the pixel-wise term D1 ensures that all predicted density values ​​have corresponding annotations, while the point-wise term D2 ensures that all ground-truth point annotations are considered (for transportation planning). In other words, any predicted density that is not associated with an annotation is penalized, and any annotation that is not used is also penalized.

[0084] D2(P T 1 n |b)=||P T 1 n -b1 (3)

[0085] S6: Input the test set into the LDNet network to generate a density map corresponding to the fry pictures of the test set. It can be understood that after the above step S5, the training of the LDNet network is completed, and the test set is further used to train the LDNet network.

[0086] S7: summing up the fry points of each density map obtained in step S6 to obtain the corresponding predicted number of fry, that is, the predicted number of fry can be obtained by counting the number of fry points on the density map.

[0087] S8: Calculate the prediction accuracy based on the predicted number of fry (obtained in the above step S7) and the actual value of the number of fry (known).

[0088] The fry counting performance of the present invention is experimentally evaluated below.

[0089] In the experiment, MAE, MSE and Accuracy are used to evaluate the counting performance of the present invention (see the following formulas 4, 5 and 6). MAE (Mean Absolute Error) is used to measure the mean absolute error between the predicted value and the true value, MSE (Mean Squared Error) is used to measure the mean square error between the predicted value and the true value, and Accuracy is used to evaluate the counting accuracy, where n is the number of images, y is the number of images, and i is the true value of the i-th image, is the predicted value of the i-th image.

[0090]

[0091] The definition of is as shown in the following formula (7), where D w,l is the pixel at (w,l) in the generated density map.

[0092]

[0093] During the training process, the initial learning rate was set to 1e-5 to control the update speed of the model parameters. In order to avoid overfitting of the model, the weight decay coefficient was set to 1e-5. A total of 500 rounds of training were conducted, and validation began at the 50th round, and validation was performed every 5 rounds to monitor the training progress and performance of the model.

[0094] To train the model, the resolution of the input image was adjusted to 1024x1365. In addition, all experiments were conducted on a workstation equipped with Ubuntu 20.04.3 LTS operating system. The main specifications of the workstation include GPU: NVIDIA 3090, CPU: 11th Generation Intel Core i7-11700@2.50GHz, RAM: ADATA 32G, and hard disk: Western Digital 2T. The counting model was built using the Python programming language of the PyTorch deep learning framework and the experiments were conducted on this workstation.

[0095] Experimental results and discussion: In order to prove the superiority of the LDNet+IMBDA algorithm combination of the present invention, a series of comparative experiments and ablation analysis were carried out on the fry data set, and experiments were carried out on the CARPK, PUCPR+ and Penaeus_1k data sets to verify the generalization ability of the algorithm of the present invention. First, a comparative test was carried out with several popular counting algorithms, and the results showed that LDNet+IMBDA has obvious advantages in counting accuracy. Secondly, an ablation analysis was carried out to verify the importance of IMBDA strategy, training sample size, loss function, model using IMBDA and training strategy in the LDNet+IMBDA algorithm, and the optimality of the algorithm combination of the present invention was proved. Finally, through experiments on the CARPK, PUCPR+ and Penaeus_1k data sets, the generalization performance of the LDNet+IMBDA algorithm combination on different data sets was verified. The results show that it can obtain good counting accuracy and has good generalization ability when using only a small number of training samples.

[0096] Performance of LDNet and IMBDA: In order to verify the effectiveness of the IMBDA strategy and the counting performance of LDNet, the present invention uses the IMBDA strategy to perform data enhancement on the training set of 16 fry pictures and the verification set of 8 fry pictures, and inputs them into LDNet for training. The training is fully trained until it converges. The loss change diagram during training is shown in Figure 6 After the training, the test sets of 8 gradients were tested respectively. The results show that each gradient can achieve good counting accuracy, as shown in Table 5 below. The overall MAE reached 8.2719, RMSE reached 9.9698, and Accuracy reached 0.9901.

[0097] Table 5 Gradient test results

[0098]

[0099] The present invention uses the LDNet+IMBDA combination to conduct comparative experiments with other mainstream crowd counting models based on density map regression, including CSRNet, BL model, DM-Count model, MAN model and GL model (the results are shown in Tables 6 and Figure 7-8 ). The results show that the MAE of the method of the present invention (ours) reached 8.2719, the RMSE reached 9.9698, and the Accuracy reached 0.9901, which is significantly better than other models. Although the performance of the GL model is not very good, compared with other models, GL can be greatly improved with the support of the IMBDA strategy, which will be discussed in detail in the subsequent ablation experiments. Fig. 9 The density map output results of different models are shown. The first row is the original image. The number in the upper left corner represents the number of GTs. The second, third, fourth and fifth rows are the density maps of the original image output of the first row predicted by different models. The number in the upper left corner represents the predicted number after summing the density maps. Rounding is done here.

[0100] In order to better and more objectively demonstrate the counting accuracy of the present invention, a box plot of the prediction results is drawn (see Figure 10-11 ), the vertical axis is MAE, the horizontal axis is the model, the box represents the 50% range of all data, the horizontal line in the box represents the median, the upper and lower horizontal lines of the box represent the quartiles in the box, that is, the upper and lower limits, and the black dots are extreme values ​​that exceed 1.5 times the interquartile range. From the box plot, we can see that the box height of LDNet+IMBDA is the smallest, indicating that its data fluctuation is the smallest, and its upper and lower limits are the smallest, reflecting its high prediction accuracy. In summary, LDNet+IMBDA is significantly better than other models in counting performance, accuracy, and stability.

[0101] Table 6 Comparative experimental results

[0102]

[0103] Ablation analysis: In order to prove that the enhancement strategy and network structure used in the present invention are appropriate, the present invention conducted an ablation analysis on the IMBDA strategy, training sample size, network model and training strategy.

[0104] IMBDA strategy ablation: The present invention gradually adds 11 parts of the IMBDA strategy to LDNet, IMBDA1 represents the first part, IMBDA5 represents parts 1-5, and so on. Figure 12-13It can be seen that with the gradual addition of the IMBDA strategy, the accuracy of LDNet's fry counting is also gradually improved, which shows that each step of the IMBDA strategy is effective, successfully simulating the adverse conditions that may occur under a fixed perspective, increasing the diversity of the training set and the validation set, alleviating the overfitting problem, and improving the robustness and generalization ability of the model.

[0105] Table 7 Ablation analysis of IMBDA strategy

[0106]

[0107] Training sample size ablation: The present invention performs ablation analysis on the training sample size, uses different training sample sizes combined with the IMBDA strategy for enhancement, and uses 200 training sample sizes without the IMBDA strategy to train LDNet (see Table 8 and Figure 14-15 ), the results show that the optimal fry counting accuracy is achieved when the training sample size is 24 and combined with the IMBDA strategy, which is even higher than the counting accuracy when the training sample size is 200.

[0108] Table 8 Sample size ablation changes

[0109]

[0110] Network ablation analysis: The IMBDA strategy was used on other networks to verify the correctness of the network design. ResNet50 is a network used for classification. Its classification head was removed and a three-layer convolution module was used as its density map generation head. The network ablation results are shown in Table 9 below. The results show that LDNet combined with the IMBDA strategy can achieve the best results. Although the BL, DM-Count and GL networks can achieve an accuracy of 94% without the IMBDA strategy, the accuracy improvement after using the IMBDA strategy is not obvious, which shows that LDNet is better at extracting the common features of fry under various complex conditions and has strong adaptability.

[0111] Table 9 Network ablation results

[0112]

[0113]

[0114] Attention module ablation: LDNet connects three-layer convolution modules and three-layer channel attention modules in series as density map generation modules to enhance the generation quality of density maps. The three-layer channel attention module is ablated (see Table 10 below). The results show that the best results are obtained when using the three-layer attention module.

[0115] Table 10 Ablation results of attention layers

[0116]

[0117] Loss function ablation: The LDNet of the present invention uses the Generalized Loss based on unbalanced OT as the loss function, which is replaced here with the loss functions Bayesian Loss and DM-Count Loss commonly used for crowd counting for ablation analysis (see Table 11). The results show that the use of LDNet combined with the Generalized Loss function is the optimal combination.

[0118] Table 11 Loss function ablation results

[0119]

[0120] Cropping strategy ablation: In order to adapt to the training of LDNet, the images are randomly cropped to 512*512 for training network parameters. The images are randomly cropped to 1024*1024 and 256*256 sizes for training network parameters for ablation analysis (see Table 12). The results show that randomly cropping to 512*512 size is the most appropriate choice.

[0121] Table 12 Ablation results of cropping strategy

[0122]

[0123]

[0124] Versatility test: In order to evaluate the counting ability of the proposed method in unfamiliar scenes, experiments were conducted on the CARPK dataset and the PUCPR+ dataset. The CARPK dataset and the PUCPR+ dataset were proposed by Hsieh et al. in 2017. The CARPK dataset contains 989 training images and 459 test images taken by drones, containing information on 89,777 cars. The PUCPR+ dataset is a revised version of the original PUCPR dataset, captured by an fxe camera, containing 100 training images and 25 test images, with a total of 17,000 car information. The above datasets involve significant changes in vehicle lighting, darkness, and background.

[0125] Here, 50 training images from the CARPK training dataset and 10 images from the PUCPR+ training dataset are randomly selected and enhanced using the IMBDA strategy, and then used to train LDNet. As shown in Tables 13 and 14, using only about 5% of the training data of the CARPK training dataset can achieve a MAE of 3.85 and a RMSE of 5.80, and using only about 10% of the training data of the PUCPR+ training dataset can achieve a MAE of 2.72 and a RMSE of 3.73, which are not inferior to other methods. The prediction density maps of the CARPK dataset and the PUCPR+ dataset are shown in Fig.16 , where the first column is the original image, the second column is the predicted density map, and the number in the upper left corner represents the predicted number after the integral summation of the density map, which is rounded off here.

[0126] Table 13 Results of the proposed method (ours) and existing methods on the CARPK dataset

[0127]

[0128]

[0129] Table 14 Results of the proposed method (ours) and existing methods on the PUCPR+ dataset

[0130]

[0131] In addition, experiments were conducted on the high-resolution image dataset Penaeus_1k proposed by Li et al. The Penaeus_1k dataset contains 1420 images of black tiger shrimp larvae and provides common annotations of three key points, which are suitable for density map counting, key point regression and other methods. 47 images were randomly selected from 700 training samples and enhanced with the IMBDA strategy for training LDNet. The results in Table 15 show that only about 7% of the training data was used to achieve a MAE of 13.14, a RMSE of 15.80 and an Accuracy of 0.9632. The predicted density map is shown in Fig.17 , where the first line is the original image, the second line is the predicted density map, and the number in the upper left corner represents the predicted number after the integral summation of the density map, which is rounded off here.

[0132] Table 15 Results of the proposed method and the existing method on the Penaeus_1k dataset

[0133]

[0134] Compared with the prior art, the present invention provides a high-precision fry counting method for generating a density map using limited training samples, which has the following beneficial effects: (1) The present invention uses the IMBDA strategy to perform data enhancement on the training set and the validation set to expand the training set and the validation set, thereby increasing the diversity of the training data, simulating various complex conditions that may be encountered in the actual data collection process, and achieving high counting accuracy under limited training samples, reducing data annotation costs, and reducing the intrusion on fish; (2) The LDNet network of the present invention can well extract the characteristic information of fry under various complex conditions by introducing channel attention; and in the training process of the LDNet network, different from the existing GT density map for pixel-level regression, the present invention uses the loss function based on unbalanced OT for loss constraint to effectively supervise the generation of the density map, so as to better ensure the quality of the generated density map. In this way, a high-precision fry counting effect can be achieved. Experimental results show that the counting accuracy of LDNet on the FishCount-824 dataset containing various light and shadow scenes reaches 99.01%. In addition, the LDNet+IMBDA strategy of the present invention is also migrated to the vehicle counting datasets CARPK and PUCPR+, and the giant tiger prawn counting dataset Penaeus_1k. The method of the present invention basically achieves the best counting performance using only 5%-10% of the original training samples. In general, the LDNet combined with the IMBDA strategy of the present invention provides a feasible idea and foundation for the future development of the field of fish population estimation, and provides strong support for the actual deployment of efficient fish counting systems.

[0135] It should be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or device including the element.

[0136] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A high-precision fry counting method using a limited training sample to generate a density map, characterized in that: The following steps are involved: S1: Obtain a data set of fish fry images, and further divide the data set into a training set, a validation set, and a test set; S2: marking key points of the fry images in the training set and the validation set; S3: performing data enhancement on the training set and the validation set to expand the training set and the validation set; S4: Input the data-enhanced training set and the validation set into the LDNet network; S5: generating density maps corresponding to the fry images of the training set and the validation set through the LDNet network, wherein a loss function is used to perform loss constraints during the generation of the density map; S6: inputting the test set into the LDNet network to generate a density map corresponding to the fry pictures of the test set; S7: performing fish fry point summation on each density map obtained in step S6 to obtain a corresponding predicted number of fish fry.

2. The high-precision fry counting method for generating a density map using limited training samples according to claim 1, characterized in that: The LDNet network includes a feature extraction module and a density map generation module; in the step S5, the fry feature information of the fry picture is extracted by the feature extraction module, and the density map generation module further generates a density map using the fry feature information.

3. The high-precision fry counting method for generating a density map using limited training samples according to claim 2, characterized in that: The density map generation module includes a convolutional layer and a channel attention module.

4. The high-precision fry counting method for generating a density map using limited training samples according to claim 1, characterized in that: The loss function is a generalized loss function based on unbalanced optimal transportation cost.

5. The high-precision fry counting method for generating a density map using limited training samples according to claim 1, characterized in that: In the step S3, the IMBDA strategy is used to perform data enhancement on the training set and the validation set.

6. The high-precision fry counting method for generating a density map using limited training samples according to claim 5, characterized in that: The IMBDA strategy includes performing horizontal flipping, vertical flipping and / or horizontal and vertical flipping on the fry image.

7. The high-precision fry counting method for generating a density map using limited training samples according to claim 5, characterized in that: The IMBDA strategy includes performing Gaussian blur processing, Gaussian noise processing, salt and pepper noise processing, Gaussian motion blur processing, contrast increase, contrast decrease, brightness and contrast increase and / or brightness and contrast decrease on the fry picture.

8. The high-precision fry counting method for generating a density map using limited training samples according to claim 1, characterized in that: The method further comprises step S8: calculating the prediction accuracy according to the predicted number of fry and the actual value of the number of fry.

9. The high-precision fry counting method using limited training samples to generate density maps according to claim 1, characterized in that: The data set includes pictures of fry with different lighting backgrounds and different shadow effects.

10. The high-precision fry counting method for generating a density map using limited training samples according to claim 1, characterized in that: The data set includes pictures of fry at different fry density levels.

Citation Information

Patent Citations

  • Method for detecting densely shielded fishes based on YOLOv5 network

    CN114782759A

  • Underwater fish school small target robust counting method based on density map

    CN117953361A

  • Method for detecting densely occluded fish based on YOLOv5 network

    US11790640B1