A method for constructing a fish activity recognition model based on deep learning

Abstract

The present invention discloses a method for constructing a fish activity level recognition model based on deep learning. The method decomposes 3D convolution into 2D spatial convolution and 1D temporal convolution. The 2D spatial convolution kernel is increased to 17×17 and reparameterized using a 5×5 convolution kernel. This provides the model with a sufficiently large receptive field and better ability to aggregate spatial information, making the model more accurate in feature extraction, thereby improving recognition accuracy from 41.75% to 66.99%. At the same time, the number of spatiotemporal modules is reduced, thereby significantly reducing the number of parameters and achieving model lightweighting. In addition, a convolutional feedforward network is added after the spatiotemporal module to increase the network depth, thereby improving the model's representation ability, thereby achieving the purpose of adding more nonlinearity and cross-channel information exchange to the model, and being able to better fit the data distribution to make the model more flexible.

Description

Technical Field

[0001] The present invention belongs to the technical field of digital image classification and recognition, and in particular relates to a method for constructing a fish activity level recognition model based on deep learning. Background Art

[0002] Fish activity can reflect their hunger level and health. For example, fish in different appetite states will exhibit different behavioral responses to feed dropped into the pond, such as aggregation, dispersion, and swimming speed. Therefore, monitoring and identifying fish activity levels in farmed fish can help control feed dosage, frequency, and timing. This can prevent underfeeding, which reduces muscle conversion and leads to food-grabbing and aggression among fish, and overfeeding, which leads to feed waste and water pollution. It can also promptly detect abnormal behavior during fish farming, enabling early warning and timely remedial measures. In short, monitoring and identifying fish activity levels in farmed fish can reduce losses during the farming process and increase production and economic income.

[0003] In the past, fish farming relied on direct manual observation of fish behavior to identify feeding intensity. This was not only time-consuming and labor-intensive, but also difficult to parameterize, making it impossible for automatic feeding machines to accurately feed. With the development of computer technology, most methods, such as spatiotemporal keypoints and the iDT trajectory method, were used to classify captured fish video images. However, these methods faced challenges inherent in underwater image recognition, such as poor image quality, uncontrollable targets and environments, and difficulty obtaining representative samples. Furthermore, there were still problems such as over-reliance on manual judgment and long debugging errors. There are also a growing number of deep learning-based video classification methods. For example, CNN-based methods do not require manual feature extraction and have significantly surpassed traditional methods in performance. These methods are exemplified by models such as RNN / LSTM time series models, two-stream methods, and 3D convolution. Among them, RNN / LSTM video and voice signals are both time-series signals. RNN is used to extract features for each video frame, and LSTM is used to model the time-series relationship. However, it has disadvantages such as general parallel processing effect and time-consuming calculation. The two-stream method contains two channels, one is the RGB image channel for modeling spatial information, and the other is the optical flow channel for modeling time-series information. The two are jointly trained and information is fused. Since this method needs to add the step of extracting optical flow, it has high requirements on the data set. 3D convolution is more suitable for video classification problems. Compared with images, videos have an additional time dimension, and 3D convolution can be used to process this dimension. In this structure, each feature map in the convolution layer will be connected to multiple adjacent continuous frames in the previous layer, which is conducive to capturing motion information. However, ordinary 3D spatiotemporal convolution mixes spatial information and dynamic information, which is not easy to optimize. Currently, 3D convolution is also divided into 2D spatial convolution and 1D temporal convolution for terrestrial video classification. The 2D spatial convolution is composed of multiple 3×3 convolution kernels. Although it increases the network depth and reduces the amount of computation, the receptive field is small and it is prone to overfitting. In particular, it is not effective when identifying fish behavior in underwater videos, with a low recognition accuracy of only 41.75%. Summary of the Invention

[0004] The present invention aims to solve the above-mentioned technical problems existing in the prior art and provides a method for constructing a fish activity level recognition model based on deep learning.

[0005] The technical solution of the present invention is: a method for constructing a fish activity level recognition model based on deep learning, which is carried out in the following steps:

[0006] Step 1: Collect fish video data, manually annotate the activity levels of the fish as labels, and then preprocess the manually annotated video dataset by dividing the video dataset into image datasets, and then dividing the image dataset into training set, validation set, and test set;

[0007] Step 2. Perform data augmentation on the image dataset, i.e., enhance sample diversity;

[0008] Step 3. Let epoch = 1, T is the number of iterations;

[0009] Step 4. Input the training set samples into the backbone network and perform convolution operations with a step size of 1 pixel. The first layer of the backbone network is a spatiotemporal module, the second layer is a convolutional feedforward neural network, the third layer is a spatiotemporal module, the fourth layer is a convolutional feedforward neural network, the fifth and sixth layers are both convolutional layers (conv) with a convolution kernel of 3×3, and the seventh layer is a fully connected layer (FC).

[0010] The first layer of the spatiotemporal module is a 2D spatial convolution layer with convolution kernels of 17×17 and 5×5 respectively, the second layer is a BN layer, the third layer is a 1D temporal convolution layer with a convolution kernel of 3×3, the fourth layer is a BN layer, and the fifth layer is an activation function ReLU layer;

[0011] The first layer of the convolutional feedforward neural network is a BN layer, the second and third layers are both 3D convolution layers with a convolution kernel of 1×1×1, the fourth layer is an activation function ReLU layer, and the fifth layer is a residual structure;

[0012] Step 5. Calculate the cross entropy loss function for the fully connected layer FC output according to formula (1):

[0013]

[0014] Among them, p(x i ) is the true sample distribution, q(x i ) is the sample distribution predicted by the model, i = 1, 2, 3...n, i is the sample ordinal number, n is the total number of samples;

[0015] Step 6. epoch←epoch+1 If epoch>T, the trained model is obtained. Otherwise, continue to optimize the network parameters and go to step 4.

[0016] The present invention decomposes the 3D convolution into 2D spatial convolution and 1D temporal convolution, increases the original 3×3 convolution kernel to 17×17, and uses a 5×5 convolution kernel for reparameterization, providing the model with a sufficiently large receptive field and better ability to aggregate spatial information, making the model more accurate in feature extraction, thereby improving recognition accuracy; at the same time, the number of spatiotemporal modules is reduced, that is, the number of 2D spatial convolution and 1D temporal convolution layers is reduced, thereby significantly reducing the number of parameters, achieving model lightweighting, achieving the effect of data processing immediacy, and improving the convenience and practicality of model method application. In addition, a convolutional feedforward network is added after the spatiotemporal module, increasing the network depth, thereby improving the model's representation ability, so as to achieve the purpose of adding more nonlinearity and cross-channel information exchange to the model, and being able to better fit the data distribution to make the model more flexible. Compared with the existing technology, the recognition accuracy is improved from 41.75% to 66.99%, providing a theoretical basis for the start and stop of the bait feeder, the selection of feed amount and feeding frequency during the intelligent feeding process; at the same time, because the model has been lightweight, only 0.97M parameters are involved, which greatly reduces the data processing time and is more suitable for deployment in equipment with limited resources in fish identification sites. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 It is a schematic diagram of the backbone network structure of an embodiment of the present invention.

[0018] Figure 2 This is a diagram showing the effect of using an embodiment of the present invention to recognize a video image. DETAILED DESCRIPTION

[0019] The present invention provides a method for constructing a fish activity recognition model based on deep learning, which is carried out in the following steps:

[0020] Step 1. Install a camera in the aquaculture pond to capture real-time video of fish feeding after feeding as a fish video dataset. Manually annotate the activity levels of the fish (calm feeding, active feeding, no feeding) as labels. Then, perform preprocessing such as frame slicing and normalization on the manually annotated video dataset. That is, split the video dataset into an image dataset, and then divide the image dataset into a training set, a validation set, and a test set.

[0021] Step 2. Perform data augmentation on the image dataset, i.e., random aspect ratio cropping, random rotation, size adjustment, color adjustment, etc., to enhance sample diversity;

[0022] Step 3. Let epoch = 1, T is the number of iterations;

[0023] Step 4. Input the training set samples (32-channel RGB images with a size of 256×256 pixels) into the backbone network. The structure of the backbone network is as follows: Figure 1As shown: the first layer is the spatiotemporal module (ST-Block), the second layer is the convolutional feedforward neural network (ConvFFN), the third layer is the spatiotemporal module (ST-Block), the fourth layer is the convolutional feedforward neural network (ConvFFN), the fifth and sixth layers are both convolutional layers with a convolution kernel of 3×3 (conv3×3), and the seventh layer is the fully connected layer FC, with 512 units fully connected;

[0024] The first spatial conv layer of the spatiotemporal module is a 2D spatial convolution layer with convolution kernels of 17×17 and 5×5, respectively, and performs convolution operations with a step size of 1 pixel. The second layer is a batch normalization layer, the third temporal conv layer is a 1D temporal convolution layer with a convolution kernel of 3×3, the fourth layer is a batch normalization layer, and the fifth layer is a ReLU layer with an activation function.

[0025] The first layer of the convolutional feedforward neural network structure is a BN layer, the second layer 1×1×1conv3D and the third layer 1×1×1conv3D are both 3D convolution layers with a convolution kernel of 1×1×1, a channel of 32, the fourth layer is an activation function ReLU layer, and the fifth layer is a residual structure;

[0026] Step 5. Calculate the cross entropy loss function for the fully connected layer FC output according to formula (1):

[0027]

[0028] Among them, p(x i ) is the true sample distribution, q(x i ) is the sample distribution predicted by the model, i = 1, 2, 3...n, i is the sample ordinal number, n is the total number of samples;

[0029] Step 6. If epoch←epoch+1>T, select the optimal model output, that is, obtain the trained model. Otherwise, continue to optimize the network parameters and go to step 4.

[0030] After verification with the validation set data, a fish activity recognition model based on deep learning is obtained. The pictures of the test set are passed into the fish activity recognition model based on deep learning constructed in the embodiment of the present invention to classify the videos of farmed fish in the feeding state. The results are as follows: Figure 2 As shown, the recognition algorithm of the present invention can accurately classify the input video image into the corresponding three states of quiet feeding, active feeding and weak feeding.

Claims

1. A method for constructing a fish activity recognition model based on deep learning, characterized in that Follow the steps below: Step 1: Collect fish video data, manually annotate the activity levels of the fish as labels, and then preprocess the manually annotated video dataset by dividing the video dataset into image datasets, and then dividing the image dataset into training set, validation set, and test set; Step 2. Perform data augmentation on the image dataset, i.e., enhance sample diversity; Step 3. Let epoch = 1, T is the number of iterations; Step 4. Input the training set samples into the backbone network; The first layer of the backbone network is a spatiotemporal module, the second layer is a convolutional feedforward neural network, the third layer is a spatiotemporal module, the fourth layer is a convolutional feedforward neural network, the fifth and sixth layers are both convolutional layers (conv) with a convolution kernel of 3×3, and the seventh layer is a fully connected layer (FC); The first layer of the spatiotemporal module is a 2D spatial convolution layer with convolution kernels of 17×17 and 5×5 respectively, the second layer is a BN layer, the third layer is a 1D temporal convolution layer with a convolution kernel of 3×3, the fourth layer is a BN layer, and the fifth layer is an activation function ReLU layer; The first layer of the convolutional feedforward neural network is a BN layer, the second and third layers are both 3D convolution layers with a convolution kernel of 1×1×1, the fourth layer is an activation function ReLU layer, and the fifth layer is a residual structure; Step 5. Calculate the cross entropy loss function for the fully connected layer FC output according to formula (1): Among them, p(x i ) is the true sample distribution, q(x i ) is the sample distribution predicted by the model, i = 1, 2, 3...n, i is the sample ordinal number, n is the total number of samples; Step 6. epoch←epoch+1. If epoch>1, the trained model is obtained. Otherwise, continue to optimize the network parameters and go to step 4.

Images