Fish feeding intensity evaluation method based on improved VGG16 and spectrogram

Abstract

The invention provides a fish ingestion intensity evaluation method based on improved VGG16 and a spectrogram, and relates to the field of aquaculture. The method comprises the following steps: acquiring fish ingestion sound signals by adopting a hydrophone, dividing the ingestion sound signals into four types of intensities of'strong ', 'medium', 'weak 'and'no', manually marking the ingestion sound signals, converting the sound signals into spectrograms, and constructing a fish ingestion sound signal data set; an attention mechanism fusion module is designed, the attention mechanism fusion module is integrated on a convolutional neural network, a fish ingestion sound signal data set is used as input, and automatic grading of ingestion intensity is achieved. The method provided by the invention can efficiently evaluate the fish ingestion intensity so as to judge the fish appetite. The method can be applied to fish feeding strategy optimization and behavior research.

Description

Technical Field

[0001] The present invention relates to the field of aquaculture, and in particular to a method for evaluating fish feeding intensity based on improved VGG16 and spectrogram. Background Art

[0002] In aquaculture, feed cost is one of the main expenses. In the actual breeding process, it is easy to underfeed or overfeed. Underfeeding affects the growth rate of fish, and overfeeding increases the breeding cost and reduces the breeding efficiency. In addition, the uningested feed decomposes to produce ammonia nitrogen and nitrate compounds, which pollute the water environment and affect the healthy growth of fish. The feeding intensity of fish reflects the intensity and variation of the movements and behaviors of fish during feeding, which can directly reflect the appetite of fish. At present, the method of using machine vision to evaluate the feeding intensity of fish is easily affected by the fluctuation of water quality and light pattern. In order to remove this influence and improve the classification accuracy, sound signals are used to evaluate the feeding intensity of fish.

[0003] Fish feeding intensity assessment based on acoustic signals was first proposed by Cui Meng et al. A new method for fish feeding intensity assessment using audio was introduced. A deep learning framework for FFIA was proposed, which first converted the audio signal into acoustic features, namely mel spectrograms, and then used a convolutional neural network (CNN)-based model to classify the feeding intensity of fish. Summary of the invention

[0004] To solve the above problems, the present invention proposes a fish feeding intensity assessment method based on improved VGG16 and spectrogram, which improves the accuracy of feeding intensity classification and better guides feeding strategies.

[0005] The present invention is implemented by the following scheme: a method for evaluating fish feeding intensity based on improved VGG16 and spectrogram, the method comprising the following steps:

[0006] Step 1: Use a hydrophone to obtain fish feeding sound signals, manually annotate the intensity of the feeding sound signals to obtain intensity classification results, and convert the sound signals into spectrograms to construct a spectrogram dataset of fish feeding sound signals;

[0007] Step 2: Embed the SimAM parameter-free attention module into the SENet network to obtain an attention mechanism fusion module that can simultaneously focus on the feature weights in the feature map space and the channel;

[0008] Step 3: The designed attention mechanism fusion module is integrated into the maximum pooling layer of the convolutional neural network to extract global features and perform classification;

[0009] Step 4: Use the constructed spectrogram dataset of feeding sound signals as input to the improved convolutional neural network, output the distribution probability of intensity classification, and realize automatic classification of feeding intensity.

[0010] Furthermore, in step 1, the feeding sound signal is divided into four intensities: "strong", "medium", "weak" and "none".

[0011] Furthermore, the step 1 converts the feeding sound signal X(k, t) into a spectrogram using the following formula:

[0012]

[0013] Where x(n) is the framed signal, ω(n) represents the Hamming window, n represents the sample index in the discrete time series, and d is the number of samples.

[0014] Furthermore, a new attention mechanism fusion module is designed in step 2 to embed the SimAM parameter-free attention module into SENet. This module combines the advantages of SENet and SimAM modules. This module can not only adaptively adjust feature weights in the channel dimension, but also generate attention weights in the spatial dimension using local self-similarity.

[0015] Furthermore, the mathematical expression of SENet is:

[0016]

[0017] In the formula, Z represents the global information after compression, U represents the input image feature, and F sq Represents channel weight; w represents the width of the input feature map. For the input image feature U, its dimensions are H, W, C, H and W represent the height and width of the image, and C represents the number of channels. The global average pool is used to squeeze the corresponding spatial information on each channel into the corresponding channel. The compressed channel information will be applied to the excitation step 2. The purpose is to improve the feature representation ability of the network by learning how to accurately adjust the contribution of each channel.

[0018] Furthermore, the mathematical expression of SimAM is as follows:

[0019]

[0020] In the formula, e t represents the attention weight or energy value at time step t; w t ,b t The two parameters represent the calculation of e t The weights and bias terms used when y represents the feature value at a certain position in the input feature map; x represents the feature value at a certain position in the input feature map; iRepresents an element in the input feature map, where i is the index used to traverse the elements in the feature map; y t represents the predicted value or feature value at time step t; M represents the number of elements considered when calculating the average error term; y o Represents the predicted value or characteristic value of a target value or center position; represents the estimated variance value, where is the variance; λ represents the regularization parameter; minimizing the above formula is equivalent to training neurons in the same channel Linear separability between neurons and other neurons; important neurons are identified by measuring the linear separability between neurons. The linear separability is measured by constructing an energy function To achieve this, based on the activity value t of the target neuron and the average activity value of other neurons The difference can be used to judge the distinctiveness and importance of the target neuron.

[0021] Furthermore, the VGG16 model is selected as the benchmark network, and the step 4 of realizing automatic classification of feeding intensity includes the following steps:

[0022] Step 4.1: The spectrogram dataset is cropped to 224x224x3 size as input, and after two layers of 3x3 convolution kernels, the output feature map size is 112x112x64;

[0023] Step 4.2: The image is further subjected to 3x3 convolution kernel and linear layer to extract features, and the feature map size becomes 56x56x128;

[0024] Step 4.3: After three layers of 3x3 convolution kernels and one linear layer, the feature map size is transformed from 56x56x128 to 7x7x512;

[0025] Step 4.4: Flatten the 7x7x512 feature map into a 4096-dimensional vector and input it into the fully connected layer;

[0026] Step 4.5: The 4096-dimensional vector is fused into an attention mechanism module to further extract global features.

[0027] Step 4.6: Input the 4096-dimensional vector after extracting the global features into the maximum pooling layer. After classification by the maximum pooling layer, the distribution probability of the intensity classification is output.

[0028] Compared with the prior art, the present invention has the following beneficial effects:

[0029] 1. Based on the fact that the sound signal of fish feeding does not need to rely on the light conditions of the visual environment, the present invention uses a Hamming window to convert the sound signal into a spectrogram. The two-dimensional image can be conveniently used as the input of the convolutional neural network to quickly and efficiently realize the evaluation of fish feeding intensity.

[0030] 2. The attention mechanism fusion module designed by the present invention has the characteristics of being able to pay attention to the feature weights on the feature map space and the channel at the same time. After embedding the module into the maximum pooling layer of the VGG16 model, the new algorithm can not only adaptively adjust the feature weights in the channel dimension, but also generate attention weights in the spatial dimension using local self-similarity, thereby achieving more comprehensive feature learning and utilization. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 A flowchart for implementing the present invention;

[0032] Figure 2 Four types of food intake intensity spectrograms provided by the embodiments of the present invention;

[0033] Figure 3 Graphs showing the training accuracy and training loss of the new algorithm in the embodiment of the present invention. DETAILED DESCRIPTION

[0034] The present invention is further described in detail below with reference to the accompanying drawings and specific embodiments:

[0035] like Figure 1 As shown, the embodiment of the present invention provides a method for evaluating fish feeding intensity based on improved VGG16 and spectrogram, obtaining fish feeding sound signals collected by a hydrophone, and the distribution probability of the sound signal strength classification, specifically including the following steps:

[0036] S1. Use a hydrophone to obtain fish feeding sound signals, manually mark the feeding sound signals based on their intensity, and classify them. In the embodiment of the present invention, the feeding sound signals are divided into four intensity levels: "strong", "medium", "weak", and "none", and the sound signals are converted into a spectrogram, such as Figure 2 As shown, a fish feeding sound signal dataset is constructed;

[0037] In this embodiment, the specific implementation process of step 1 is as follows:

[0038] S1.1, perform frame and window processing on the original sample feeding sound signal x(n), where n represents the sample index in the discrete time series and N is the length of the window function. For each frame signal after framing, perform fast Fourier transform to achieve conversion from time domain to frequency domain:

[0039]

[0040] Wherein, ω(n) represents a Hamming window, which is used for preprocessing. The Hamming window is a widely used windowing technique that can reduce spectrum leakage and improve the accuracy of spectrum estimation.

[0041] S1.2, calculate the power spectrum of each frame of speech based on the obtained spectral features, take its logarithm to obtain the logarithmic power spectrum, and finally splice the logarithmic power spectrum of each frame according to the time dimension to obtain the original grayscale spectrogram, map them to colors one by one, and finally obtain the spectrogram.

[0042] In this embodiment, the following formula is used to convert the feeding sound signal X(k, t) into a spectrogram:

[0043]

[0044] S2. The SimAM parameter-free attention module is embedded into SENet to obtain an attention mechanism fusion module, which can simultaneously focus on the feature weights of the feature map space and the channel. The specific implementation steps are as follows:

[0045] S2.1: The spectrogram image is fed into the basic convolutional layer of SENet for preliminary feature extraction;

[0046] S2.2: After basic feature extraction, SENet compresses the feature map into a single vector through a global average pooling layer. This vector contains the global information of the entire feature map; the mathematical expression of SENet is:

[0047]

[0048] In the formula, Z represents the global information after compression, U represents the input image feature, and F sq Represents channel weight; w represents the width of the input feature map. For the input image feature U, its dimensions are H, W, C, H and W represent the height and width of the image, and C represents the number of channels. The global average pool is used to squeeze the corresponding spatial information on each channel into the corresponding channel. The final output dimension becomes 3x3xC. The compressed channel information will be applied to the excitation step S2. The purpose is to improve the feature representation ability of the network by learning how to accurately adjust the contribution of each channel.

[0049] S2.3: After the Squeeze operation, the obtained global feature vector is fed into the SimAM module. The SimAM module reweights the elements in the feature vector through a simple self-attention mechanism. The mathematical expression of SimAM is as follows:

[0050]

[0051] In the formula, e t represents the attention weight or energy value at time step t. t ,b t , these two parameters represent the calculation of e t The weights and biases used when y represents the feature value at a certain position in the input feature map. i Represents an element in the input feature map, where i is the index used to traverse the elements in the feature map. t y represents the predicted value or feature value at time step t. M represents the number of elements considered when calculating the mean error term. o Represents the predicted value or feature value of a target value or center location. This symbol represents the estimated variance value, where is the variance. λ represents the regularization parameter. Minimizing the above formula is equivalent to training neurons in the same channel Linear separability from other neurons; in neuroscience research, active neurons tend to inhibit the activity of neighboring neurons through spatial inhibition effects. This phenomenon indicates that in visual processing, neurons with significant spatial inhibition properties are more important for information screening and processing. In order to achieve a similar selection mechanism, important neurons are identified by measuring the linear separability between neurons. The measurement of linear separability is achieved by constructing an energy function. Energy function The purpose of quantifying the neuron discrimination is to compare the activity value of a neuron with the average activity value of other neurons. When the difference is large, it leads to This indicates that the target neuron has higher discrimination and importance. On the contrary, when t is When approaching, The value will increase, indicating that the neuron has low discrimination.

[0052] S2.4: Send the weighted feature vector output by the SimAM module back to the Excitation operation of SENet. The Excitation operation maps the weighted feature vector into a new feature vector through a fully connected layer and a ReLU activation function, and then generates a channel weight through another fully connected layer sigmoid activation function.

[0053] S2.5: In this embodiment, the channel weights generated by the Excitation operation are multiplied by the original feature map to complete the feature reweighting. This step enhances the network's perception of important features while suppressing unimportant features.

[0054] S3. The attention mechanism fusion module is embedded into the maximum pooling layer of VGG16 to extract global features and classify them, thereby achieving more comprehensive feature learning and utilization.

[0055] S4. The constructed feeding sound signal is used as the input of the new feeding intensity assessment algorithm, and the distribution probability of four categories is output to realize the automatic classification of feeding intensity. The training accuracy graph and training loss graph of the improved VGG16 network are shown in Figure 2. Figure 3 shown.

[0056] In this embodiment, the step 4 to realize automatic classification of food intake intensity includes the following steps:

[0057] S4.1: The spectrogram is used as the input of the improved VGG16 network in step 3, cropped to 224x224x3 size, and passed through two layers of 3x3 convolution kernels with a stride of 1 and padding of 1. The output feature map size is 112x112x64;

[0058] S4.2: The image continues to be feature extracted by 3x3 convolution kernel and linear layer, and the feature map size becomes 56x56x128;

[0059] S4.3: After three layers of 3x3 convolution kernels (with a step size of 2) and one linear layer, the feature map size is transformed from 56x56x128 to 7x7x512;

[0060] S4.4: Flatten the 7x7x512 feature map into a 4096-dimensional vector and input it into the fully connected layer;

[0061] S4.5: The 4096-dimensional vector is fused through the attention mechanism module to further extract global features;

[0062] S4.6: The 4096-bit vector extracted from the global features is input into the maximum pooling layer, and the maximum pooling layer classifies and outputs the probability of four types of feeding intensity distribution.

[0063] In order to prove the feasibility of the method of the present invention, relevant simulation experiments were carried out and compared with four other classic convolutional neural networks. The results are shown in Table 1.

[0064] Table 1 Comparison of indicators of convolutional neural network of each method and feeding intensity assessment algorithm of the present invention

[0065]

[0066]

[0067] From the analysis of the experimental data in Table 1, it can be seen that the average test accuracy of Shufflenet, MobilenetV2, Alexnet, and Resnet18 are 0.8922, 0.9175, 0.7512, and 0.9428, respectively. The results show that compared with the above models, the test accuracy of the model proposed in the present invention is improved by 0.0515, 0.0262, 0.1925, and 0.0009, respectively. The results show that the algorithm proposed in the present invention has better performance. The model combines the focus on channel and spatial features of SENet and SimAM, thereby more comprehensively extracting features on feature maps and solving the problems caused by random prediction.

[0068] Although the disclosure is disclosed as above, the protection scope of the disclosure is not limited thereto. Those skilled in the art can make various changes and modifications without departing from the spirit and scope of the disclosure, and these changes and modifications will fall within the protection scope of the present invention.

Claims

1. A method for evaluating fish feeding intensity based on improved VGG16 and spectrogram, characterized in that: The method comprises the following steps: Step 1: Use a hydrophone to obtain fish feeding sound signals, manually annotate the intensity of the feeding sound signals to obtain intensity classification results, and convert the sound signals into spectrograms to construct a spectrogram dataset of fish feeding sound signals; Step 2: Embed the SimAM parameter-free attention module into SENet to obtain an attention mechanism fusion module that can simultaneously focus on the feature weights in the feature map space and the channel; Step 3: The designed attention mechanism fusion module is integrated into the maximum pooling layer of the convolutional neural network, and a new fish feeding intensity assessment algorithm is proposed to extract global features and perform classification; Step 4: Use the constructed spectrogram dataset of feeding sound signals as input to the improved convolutional neural network, output the distribution probability of intensity classification, and realize automatic classification of feeding intensity.

2. The method for evaluating fish feeding intensity based on improved VGG16 and spectrogram according to claim 1, characterized in that: In step 1, the feeding sound signals are classified into four categories, namely, "strong", "medium", "weak", and "none".

3. The method for evaluating fish feeding intensity based on improved VGG16 and spectrogram according to claim 1, characterized in that: The step 1 converts the feeding sound signal X(k, t) into a spectrogram using the following formula: Where x(n) is the framed signal, ω(n) represents the Hamming window, n represents the sample index in the discrete time series, and d is the number of samples.

4. The method for evaluating fish feeding intensity based on improved VGG16 and spectrogram according to claim 1, characterized in that: In step 2, a new attention mechanism fusion module is designed to embed the SimAM parameter-free attention module into SENet. This module combines the advantages of SENet and SimAM modules. This module can not only adaptively adjust feature weights in the channel dimension, but also generate attention weights in the spatial dimension using local self-similarity.

5. The method for evaluating fish feeding intensity based on improved VGG16 and spectrogram according to claim 1, characterized in that: The mathematical expression of SENet is: In the formula, Z represents the global information after compression, U represents the input image feature, and F sq Represents channel weight, w represents the width of the input feature map. For the input image feature U, its dimensions are H, W, C, H and W represent the height and width of the image, and C represents the number of channels. The global average pool is used to squeeze the corresponding spatial information on each channel into the corresponding channel. The compressed channel information will be applied to the excitation step 2. The purpose is to improve the feature representation ability of the network by learning how to accurately adjust the contribution of each channel.

6. The method for evaluating fish feeding intensity based on improved VGG16 and spectrogram according to claim 1, characterized in that: The mathematical expression of SimAM is as follows: In the formula, e t represents the attention weight or energy value at time step t; w t ,b t The two parameters represent the calculation of e t The weights and bias terms used when y represents the feature value at a certain position in the input feature map; x represents the feature value at a certain position in the input feature map; i Represents an element in the input feature map, where i is the index used to traverse the elements in the feature map; y t represents the predicted value or feature value at time step t; M represents the number of elements considered when calculating the average error term; y o Represents the predicted value or characteristic value of a target value or center position; represents the estimated variance value, where is the variance; λ represents the regularization parameter; minimizing the above formula is equivalent to training neurons in the same channel Linear separability between neurons and other neurons; important neurons are identified by measuring the linear separability between neurons. The linear separability is measured by constructing an energy function To achieve this, based on the activity value t of the target neuron and the average activity value of other neurons The difference can be used to judge the distinctiveness and importance of the target neuron.

7. The method for evaluating fish feeding intensity based on improved VGG16 and spectrogram according to claim 1, characterized in that: The VGG16 model is selected as the benchmark network, and the step 4 to realize automatic classification of feeding intensity includes the following steps: Step 4.1: The spectrogram dataset is cropped to 224x224x3 size as input, and after two layers of 3x3 convolution kernels, the output feature map size is 112x112x64; Step 4.2: The image is further subjected to 3x3 convolution kernel and linear layer to extract features, and the feature map size becomes 56x56x128; Step 4.3: After three layers of 3x3 convolution kernels and one linear layer, the feature map size is transformed from 56x56x128 to 7x7x512; Step 4.4: Flatten the 7x7x512 feature map into a 4096-dimensional vector and input it into the fully connected layer; Step 4.5: The 4096-dimensional vector is fused into an attention mechanism module to further extract global features. Step 4.6: Input the 4096-dimensional vector after extracting the global features into the maximum pooling layer. After classification by the maximum pooling layer, the distribution probability of the intensity classification is output.

Images