Fish counting method in dense scene

Abstract

The invention relates to the field of target counting, in particular to a fish counting method in a dense scene, which comprises the following steps: firstly, acquiring fish images in a culture pond, forming a fish image set and marking, extracting features of the fish images and examples through ViT and a convolutional neural network, displaying the similarity between the captured images and the given example fishes by using an attention mechanism, and counting the fishes in the culture pond according to the similarity. A density map adaptive optimization module is added in training, so that accurate counting and density distribution of fish targets are achieved, the influence of shielding and overlapping due to dense scenes is overcome, and high accuracy of a counting model is achieved.

Description

Technical Field

[0001] The invention relates to the field of target counting, and in particular to a fish counting method by generating a density map in a dense scene. Background Art

[0002] In recent years, with the growth of global population and the increase of consumption demand, the scale and output of aquaculture have continued to expand. Accurately counting individual fish is crucial to the aquaculture industry. It can not only help farmers monitor the health of fish, control the stocking density, and prevent the spread of diseases, but also effectively manage resources, improve the efficiency of farming, and achieve the goal of sustainable farming. Therefore, the number of fish is of great significance in promoting the development of the aquaculture industry, enhancing competitiveness, and achieving sustainable development.

[0003] However, the traditional manual fish counting method consumes a lot of time and manpower, and the accuracy of manual counting will be greatly challenged in complex conditions such as turbid water and dense fish schools. In addition, manual counting is inefficient and has large errors, which does not meet the needs of modern aquaculture for efficient and precise management.

[0004] In order to solve the above problems, the method of using hardware photoelectric conversion processing to count fish has been widely studied and applied in recent years. For example, the method of using sensors or cameras to count fish has higher efficiency than the traditional manual counting method, but its hardware cost is high and the reliability and performance requirements of the equipment are high, which limits its popularization and application in large-scale farms.

[0005] With the rapid development of computer vision and deep learning technology, fish counting methods are gradually developing towards intelligence and automation. At present, fish counting is mainly divided into two methods: target detection-based method and density map-based method.

[0006] The method based on target detection identifies fish by generating bounding boxes of fish and gives the location and quantity. For example, the invention patent application number: 202311305616.5, title: A fish counting method based on multi-mode fusion of dynamic visual images, applies the YOLO series algorithm. However, this type of method is easily affected by occlusion and overlap in dense scenes, resulting in a serious decrease in counting accuracy, which limits its effectiveness in practical applications.

[0007] In contrast, the fish counting method based on density map has become the mainstream method of fish counting at present. Density map can not only give the target number, but also reflect the density distribution of fish. For example, invention patent application number: 202410353582.5, name: A robust counting method for small underwater fish targets based on density map, proposes to use underwater sonar equipment for imaging, and perform data enhancement on the generated image and then label it, generate a density map through a Gaussian kernel function, and use the annotated fish sonar image as the input of the counting model to train the counting model. This method improves the accuracy and robustness of fish counting to a certain extent.

[0008] Although the density map-based method has certain advantages in fish counting, it is sensitive to environmental and background conditions and still faces challenges in complex scenes. In particular, in dense scenes, the overlap and occlusion of fish still seriously affect the counting accuracy.

[0009] In summary, current fish counting technologies have the following shortcomings in dense scenes: (1) methods based on target detection are easily affected by occlusion and overlap in dense fish schools, resulting in reduced accuracy; (2) methods based on density maps are sensitive to environmental and background conditions, and the counting accuracy is not high in complex scenes. Therefore, a method that can effectively count fish in dense scenes is urgently needed to improve the accuracy and efficiency of aquaculture management. Summary of the invention

[0010] In order to overcome the deficiencies in the prior art, the present invention provides a new fish counting method, which can accurately count fish schools in dense scenes.

[0011] The technical solution of the present invention is as follows:

[0012] The present invention first obtains fish images in the breeding pond, constructs a fish image set and annotates it, extracts the features of the fish images and examples through ViT and convolutional neural networks, uses the attention mechanism to explicitly capture the similarity between the image and the given example fish, and adds a density map adaptive optimization module to the training, so as to achieve accurate counting and density distribution of fish targets.

[0013] A method for counting fish in a dense scene includes the following steps:

[0014] Step 1: Obtain fish images in the breeding pond to form a fish image set F = {f i |i=1,2,…,n},f i represents the i-th fish image, and n represents the number of fish images;

[0015] Step 2: Label the fish image set F to obtain the labeled fish image set FB = {(f i ,s i ,di )|i=1,2,…,n},s i = {b ij =(p ij ,q ij ,v ij ,w ij )|j=1,2,…,m i}, where s i Indicates that from f i The envelope set of some fish selected in b ij Indicates i The envelope of the jth fish in (p ij ,q ij ) and (v ij ,w ij ) represent the envelope box b ij The coordinates of the upper left and lower right corners, m i represents the i-th fish image f i The number of envelopes selected in d i Represents the image corresponding to f i The binary space density map, image f i The density value den for each coordinate on i (x, y) is calculated according to formula (1), d i is a two-dimensional matrix as shown in formula (2);

[0016]

[0017] Among them, (x ij ,y ij ) represents the image corresponding to f i The coordinates of the jth fish in n i Represents image f i The number of fish caught, σ represents the standard deviation of the Gaussian kernel, H i and W i Respectively represent the image f i The height and width of

[0018] Step 3: Construct a fish counting neural network. The specific steps are as follows:

[0019] Step 3.1: Extract f according to formula (3) i Specifically, the image is divided into 16×16 pixel blocks, which are projected into tokens through a shared multi-layer perceptron, and position encoding is added to each token. Through a series of transformer encoder layers (referring to multiple Transformer encoder layers of the same structure, these layers are stacked to achieve deeper feature extraction and representation learning), the image features are output according to formula (3);

[0020] F ViT =Φ ViT-ENC (f i ) (3)

[0021] Among them, Ф ViT-ENC represents the ViT-based image encoder, F ViT Represents image features;

[0022] Step 3.2: Extract s according to formula (4) i Specifically, the example passes through four convolutional layers and global average pooling to output the example features;

[0023] F CNN =Φ CNN-ENC (s i ) (4)

[0024] Among them, Ф CNN-ENC represents an example encoder based on a lightweight convolutional neural network, F CNN Indicates example features; examples refer to standard or reference fish images or their features used for image comparison in fish counting neural networks. Examples are extracted through convolutional layers and used together with the features of the input image for subsequent feature fusion and matching;

[0025] Step 3.3: The features obtained in step 3.1 and step 3.2 are fused, and the fused features are output through the decoder, which is characterized in that the features of step 3.1 and step 3.2 are fused using a feature interaction module, and the image feature F ViT As Query, the example feature F CNN The two different linear projections of are used as Value and Key, and through a series of Transformer decoder layers, the features with the same dimension as the image features are output, as shown in formula (5);

[0026] F INT =Φ INT (F ViT ,W k ·F CNN ,W v ·F CNN ) (5)

[0027] Among them, Ф INT represents the feature fusion module, F INT represents the interaction feature, W k Represents the weight matrix of Key, W v Represents the weight matrix of Value;

[0028] Step 3.4: According to feature F INTGenerate a predicted density map, characterized in that according to formula (6) through the decoder Φ DECODER F INT Reshape back to a two-dimensional feature map and restore to the same size as the image f i Same original resolution; uses a progressive upsampling design, where the vector sequence is first reshaped into a dense feature map and then processed by a convolutional neural network-based decoder, using 4 upsampling blocks, each block contains a convolutional layer and a 2x bilinear interpolation, and after the last upsampling block, a linear layer is used as a density regressor to output a single-channel predicted density map

[0029]

[0030] Step 4: Train the fish counting neural network model. The specific steps are as follows:

[0031] Step 4.1: Take the image and examples as input and get the encoded features and predicted density map

[0032] Step 4.2: Reconstruct the feature encoding to obtain the reconstructed image

[0033] Step 4.3: Output the predicted density map According to formula (7), adaptive optimization is performed to obtain the optimized density map

[0034]

[0035] in, Represents the predicted density map The value at position (x,y), Represents the optimized density map The value at position (x, y), N(x, y) represents the neighborhood of position (x, y), σ represents the scale parameter that controls the similarity measure, and α is a hyperparameter that controls the adjustment amplitude;

[0036] Step 4.4: Calculate the total loss of the reconstructed image and the optimized density map, characterized in that the total loss is calculated according to equations (8), (9) and (10);

[0037] L=λ 1 L 1 +λ 2 L 2 (8)

[0038]

[0039] Among them, L 1 Denotes the input image fi and reconstruct the image The pixel space mean square error, L 2 Represents the original density map d i And the optimized density map The mean square error, N represents the total number of training samples;

[0040] Step 5: Sum the fish images to be counted, specifically: input the fish images to be counted into the fish counting neural network model, and sum all pixel values ​​of the predicted density map.

[0041] By adopting the above technology, compared with the prior art, the beneficial effects of the present invention are as follows: the present invention adopts the Transformer architecture, extracts the features of fish images and examples through ViT and convolutional neural networks, uses the attention mechanism to explicitly capture the similarity between the image and the given example fish, and adds a density map adaptive optimization module to the training, so as to achieve accurate counting and density distribution of fish targets, thereby overcoming the influence of occlusion and overlap due to dense scenes and achieving a high accuracy of the counting model. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 For fish images;

[0043] Figure 2 for Figure 1 Point annotation diagram of

[0044] Figure 3 for Figure 1 Binary space density map of ;

[0045] Figure 4 is the fish image to be counted;

[0046] Figure 5 for Figure 4 Prediction density plot of . DETAILED DESCRIPTION

[0047] The specific implementation method of the fish counting in a dense scene of the present invention is described in detail below in conjunction with embodiments. It should be understood that the specific embodiments described here are only used to explain the present invention and are not used to limit the present invention.

[0048] The method for counting fish in dense scenes of the present invention first obtains fish images in a breeding pond, forms a fish image set and annotates it, extracts features of fish images and examples through ViT and convolutional neural networks, uses an attention mechanism to explicitly capture the similarity between the image and a given example fish, and adds a module for adaptive optimization of density maps during training, so as to achieve accurate counting and density distribution of fish targets.

[0049] The specific steps include:

[0050] Step 1: Obtain images of fish in the breeding pond, such as Figure 1 As shown, the fish image set F = {f i |i=1,2,…,n},f i represents the i-th fish image, and n represents the number of fish images; Step 2: annotate the fish image set F to obtain the annotated fish image set FB = {(f i ,s i ,d i )|i=1,2,…,n},s i = {b ij =(p ij ,q ij ,v ij ,w ij )|j=1,2,…,m i}, click the annotation picture as shown Figure 2 As shown, where s i Indicates that from f i The envelope set of some fish selected in b ij Indicates i The envelope of the jth fish in (p ij ,q ij ) and (v ij ,w ij ) represent the envelope box b ij The coordinates of the upper left and lower right corners, m i represents the i-th fish image f i The number of envelopes selected in d i Represents the image corresponding to f i The binary space density map of Figure 3 As shown, image f i The density value den for each coordinate on i (x, y) is calculated according to formula (1), d i is a two-dimensional matrix as shown in formula (2);

[0051]

[0052] Among them, (x ij ,y ij ) represents the image corresponding to f i The coordinates of the jth fish in n i Represents image f i The number of fish caught, σ represents the standard deviation of the Gaussian kernel, H i and W i Respectively represent the image f i The height and width of

[0053] Step 3: Construct a fish counting neural network. The specific steps are as follows:

[0054] Step 3.1: Extract f according to formula (3) i Specifically, the image is divided into 16×16 pixel blocks, which are projected into tokens through a shared multi-layer perceptron, and a position code is added to each token. After a series of transformer encoder layers, the image features are output according to formula (3);

[0055] F ViT =Φ ViT-ENC (f i ) (3)

[0056] Among them, Ф ViT-ENC represents the ViT-based image encoder, F ViT Represents image features;

[0057] Step 3.2: Extract s according to formula (4) i Specifically, the example passes through four convolutional layers and global average pooling to output the example features;

[0058] F CNN =Φ CNN-ENC (s i ) (4)

[0059] Among them, Ф CNN-ENC represents an example encoder based on a lightweight convolutional neural network, F CNN Represents example features;

[0060] Step 3.3: The features obtained in step 3.1 and step 3.2 are fused, and the fused features are output through the decoder, which is characterized in that the features of step 3.1 and step 3.2 are fused using a feature interaction module, and the image feature F ViT As Query, the example feature F CNN The two different linear projections of are used as Value and Key, and through a series of Transformer decoder layers, the features with the same dimension as the image features are output, as shown in formula (5);

[0061] F INT =Φ INT (F ViT ,W k ·F CNN ,W v ·F CNN ) (5)

[0062] Among them, Ф INT represents the feature fusion module, F INT represents the interaction feature, W kRepresents the weight matrix of Key, W v Represents the weight matrix of Value;

[0063] Step 3.4: According to feature F INT Generate a predicted density map, characterized in that according to formula (6) through the decoder Φ DECODER F INT Reshape back to a two-dimensional feature map and restore to the same size as the image f i Same original resolution; uses a progressive upsampling design, where the vector sequence is first reshaped into a dense feature map and then processed by a convolutional neural network-based decoder, using 4 upsampling blocks, each block contains a convolutional layer and a 2x bilinear interpolation, and after the last upsampling block, a linear layer is used as a density regressor to output a single-channel predicted density map

[0064]

[0065] Step 4: Train the fish counting neural network model. The specific steps are as follows:

[0066] Step 4.1: Take the image and examples as input and get the encoded features and predicted density map

[0067] Step 4.2: Reconstruct the feature encoding to obtain the reconstructed image

[0068] Step 4.3: Output the predicted density map According to formula (7), adaptive optimization is performed to obtain the optimized density map

[0069]

[0070] in, Represents the predicted density map The value at position (x,y), Represents the optimized density map The value at position (x, y), N(x, y) represents the neighborhood of position (x, y), σ represents the scale parameter that controls the similarity measure, and α is a hyperparameter that controls the adjustment amplitude;

[0071] Step 4.4: Calculate the total loss of the reconstructed image and the optimized density map, characterized in that the total loss is calculated according to equations (8), (9) and (10);

[0072] L=λ 1 L 1 +λ 2 L 2 (8)

[0073]

[0074] Among them, L 1 Denotes the input image f i and reconstruct the image The pixel space mean square error, L 2 Represents the original density map d i And the optimized density map The mean square error, N represents the total number of training samples; Step 5: sum the fish images to be counted, specifically: Figure 4 As shown, the fish image to be counted is input into the fish counting neural network model, and all pixel values ​​of the predicted density map are summed up. The predicted density map is as follows Figure 5 shown.

[0075] The accuracy of a single image of the present invention can reach 95%, and the MAE reaches 8.1.

[0076] The contents described in the embodiments of this specification are merely an enumeration of implementation forms of the inventive concept. The protection scope of the present invention should not be regarded as limited to the specific forms described in the embodiments. The protection scope of the present invention also extends to equivalent technical means that can be conceived by those skilled in the art based on the inventive concept.

Claims

1. A method for counting fish in a dense scene, characterized in that: The steps include: Step 1: Obtain fish images in the breeding pond to form a fish image set F = {f i |i=1,2,…,n},f i represents the i-th fish image, and n represents the number of fish images; Step 2: Label the fish image set F to obtain the labeled fish image set FB = {(f i ,s i ,d i )|i=1,2,…,n},s i = {b ij =(p ij ,q ij ,v ij ,w ij )|j=1,2,…,m i }, where s i Indicates that from f i The envelope set of some fish selected in b ij Indicates i The envelope of the jth fish in (p ij ,q ij ) and (v ij ,w ij ) represent the envelope box b ij The coordinates of the upper left and lower right corners, m i represents the i-th fish image f i The number of envelopes selected in d i Represents the image corresponding to f i The binary space density map, image f i The density value den for each coordinate on i (x, y) is calculated according to formula (1), d i is a two-dimensional matrix as shown in formula (2); Among them, (x ij ,y ij ) represents the image corresponding to f i The coordinates of the jth fish in n i Represents image f i The number of fish caught, σ represents the standard deviation of the Gaussian kernel, H i and W i Respectively represent the image f i The height and width of Step 3: Build a fish counting neural network; Step 4: Train the fish counting neural network model; Step 5: Input the fish image to be counted into the fish counting neural network model and sum all pixel values ​​of the predicted density map.

2. The method for counting fish in a dense scene according to claim 1, characterized in that: The specific steps of step 3 are as follows: Step 3.1: Extract f according to formula (3) i Features: The image is divided into 16×16 pixel blocks, projected into tokens through a shared multi-layer perceptron, and position encoding is added to each token. After a series of transformer encoder layers, the image features are output according to formula (3); F ViT =Φ ViT-ENC (f i ) (3) Among them, Ф ViT-ENC represents the ViT-based image encoder, F ViT Represents image features; Step 3.2: Extract s according to formula (4) i Example features: The example passes through four convolutional layers and global average pooling to output example features, where the example refers to the standard used for comparison with the image in the fish counting neural network; F CNN =Φ CNN-ENC (s i ) (4) Among them, Ф CNN-ENC represents an example encoder based on a lightweight convolutional neural network, F CNN Represents example features; Step 3.3: Fuse the features obtained in step 3.1 and step 3.2, output the fused features through the decoder, use the feature interaction module to fuse the features of step 3.1 and step 3.2, and convert the image feature F ViT As Query, the example feature F CNN The two different linear projections of are used as Value and Key, and through a series of Transformer decoder layers, the features with the same dimension as the image features are output, as shown in formula (5); F INT =Φ INT (F ViT ,W k ·F CNN ,W v ·F CNN ) (5) Among them, Ф INT represents the feature fusion module, F INT represents the interaction feature, W k Represents the weight matrix of Key, W v Represents the weight matrix of Value; Step 3.4: According to feature F INT Generate predicted density map: According to formula (6), decoder Φ DECODER F INT Reshape back to a two-dimensional feature map and restore to the same size as the image f i Same original resolution; uses a progressive upsampling design, where the vector sequence is first reshaped into a dense feature map and then processed by a convolutional neural network-based decoder, using 4 upsampling blocks, each block contains a convolutional layer and a 2x bilinear interpolation, and after the last upsampling block, a linear layer is used as a density regressor to output a single-channel predicted density map 3. The method for counting fish in a dense scene according to claim 2, characterized in that: The specific steps of step 4 are as follows: Step 4.1: Take the image and examples as input and get the encoded features and predicted density map Step 4.2: Reconstruct the feature encoding to obtain the reconstructed image Step 4.3: Output the predicted density map According to formula (7), adaptive optimization is performed to obtain the optimized density map in, Represents the predicted density map The value at position (x,y), Represents the optimized density map The value at position (x, y), N(x, y) represents the neighborhood of position (x, y), σ represents the scale parameter that controls the similarity measure, and α is a hyperparameter that controls the adjustment amplitude; Step 4.4: Calculate the total loss of the reconstructed image and the optimized density map: Calculate the total loss according to equations (8), (9) and (10); L=λ1L1+λ2L2 (8) Where L1 represents the input image f i and reconstruct the image The pixel space mean square error, L2 represents the original density map d i And the optimized density map The mean square error of , N represents the total number of training samples.

Images