A method for detecting fish feeding status

Abstract

The present invention discloses a method for detecting the feeding status of fish, which relates to the field of fish farming technology and solves the technical problem that existing methods for classifying the feeding status of fish schools cannot be applied to complex factory farming environments. The method includes: using a depth camera to collect depth images of a fish school's complete feeding process; converting the depth images into depth pseudo-color images based on depth changes; labeling the depth pseudo-color images as high feeding, medium feeding, low feeding, and no feeding; constructing a simple convolutional neural network model, using the labeled depth pseudo-color images to train the simple convolutional neural network model to obtain a first detection model; and using the first detection model to actually detect the feeding status of fish. The present invention uses a depth camera in combination with an image processing program to obtain a depth image of the fish school's feeding status and feeding status sequence data in real time, which can be classified using a relatively simple convolutional network or a simple recurrent neural network.

Description

Technical Field

[0001] The present invention relates to the technical field of fish breeding, and more particularly to a method for detecting the feeding status of fish. Background Art

[0002] Currently, the growing demand for aquaculture products has driven the rapid development of fish farming. my country's aquaculture area has reached 7.036 million hectares, and this area continues to increase. However, the industry's extensive, unrefined farming practices are placing increasing pressure on the environment. Precision feeding technology, tailored to fish feeding needs, is key to ensuring healthy fish growth, minimizing environmental pollution, and lowering production costs, maximizing the benefits and profitability of aquaculture.

[0003] Currently, research on fish feeding behavior, combining feature vectors reflecting the feeding state of a school with neural network classification models, has become a hot topic. For example, a LeNet5 convolutional neural network (CNN) was used to classify near-infrared images of feeding fish, achieving an average accuracy of 90%. A dual-stream recurrent network (DSRN) with a VGGNet and ResNet backbone achieved binary classification of 20 frames of fish feeding images with an accuracy of 81.4%. However, due to the DSRN's complex network structure, numerous computational parameters, and long computation time, it requires a computer equipped with a high-performance graphics card, which is costly and unsuitable for industrial aquaculture. Fish feeding videos were screened using a voting strategy, and then optical flow was used to extract inter-frame motion features (phase angle characteristics and amplitude distribution). Combined with a CNN network, the fish feeding state was classified, achieving an average accuracy of 97.6%. A kinetic energy model based on optical flow was used to quantify the spatial characteristics of fish feeding behavior. A long-short term memory (LSTM) neural network was then used to classify the vector sequences based on these spatial characteristics, achieving an average accuracy of 98.31%. Extracting inter-frame motion features using optical flow requires post-processing of the video, making the feature extraction process cumbersome, lacking real-time and continuity, and impractical for long-term monitoring of fish feeding status. Furthermore, video acquisition is easily affected by environmental lighting, making it unsuitable for engineering applications.

[0004] In addition, most fish feeding status classification algorithms are based on ideal conditions, such as recirculating aquaculture and good light source illumination. As a result, such algorithms are often limited by conditions such as generation cost, lighting environment, and water quality clarity when applied. There are also problems with deployment, and they cannot be applied to complex factory farming environments. Summary of the Invention

[0005] The technical problem to be solved by the present invention is to address the above-mentioned shortcomings of the prior art. The purpose of the present invention is to provide a method for detecting the feeding status of fish, which can reduce the demand for computer resources while ensuring high recognition accuracy, and is conducive to promotion in actual production environments.

[0006] The technical solution of the present invention is: a method for detecting the feeding status of fish, comprising:

[0007] Use a depth camera to collect depth images of a school of fish during a complete feeding process;

[0008] Converting the depth image into a depth pseudo-color image according to the depth change;

[0009] The deep pseudo-color image is labeled according to strong feeding, moderate feeding, weak feeding, and no feeding;

[0010] Constructing a simple convolutional neural network model, and using the labeled depth pseudo-color image to train the simple convolutional neural network model to obtain a first detection model;

[0011] The first detection model is used to actually detect the feeding status of fish.

[0012] As a further improvement, different colors are used to represent different depths. The distance information of the fish school in the depth image is linearly converted to the Hue value of the HSV color model. The color values of the HSV model are then converted into corresponding RGB color values to obtain a depth pseudo-color image. The formula for calculating the Hue value is as follows:

[0013] Hue=γ×[1-(Z(x,y)-Z0) / (Z1-Z0)](1)

[0014] In formula (1), Z(x, y) represents the depth value at the coordinates x and y in the depth map, that is, the vertical distance from the target to the depth camera. Its value range is the effective measurement range set by the depth camera, [Z0, Z1] is the lower and upper limits of the effective measurement range set by the depth camera, and γ is an attenuation factor with a value of 0.67.

[0015] Furthermore, the simple convolutional neural network model includes, in sequence, an input layer with a dimension of 3, 3 hidden layers, a global pooling layer, and a 4-classification output layer; the depth pseudo-color image is input into the input layer, and the 3 hidden layers all use a two-dimensional convolution kernel with a size of 5×5 and a step size of 1. The activation functions of the 3 hidden layers all use the Relu activation function, and the number of convolution kernels in the 3 hidden layers is 32, 64, and 128, respectively. Each convolution layer is followed by an average pooling layer with a step size of 2×2 for downsampling, and a Dropout layer with a dropout rate of 0.2 is used to prevent network overfitting; finally, a global pooling layer is used to reduce the calculation parameters, and a 4-classification output layer is connected, wherein the output layer uses the Softmax classification function to predict the category probability.

[0016] Furthermore, the output layer is also trained using a cross-entropy loss function, as follows:

[0017]

[0018] P in formula (2) i is the final predicted probability of category i, z i is the original budget output of the classification model for category i; z j is the original budget output of the classification model for category j, C is the total number of categories, L in formula (3) is the cross entropy loss function, y i is the label corresponding to category i.

[0019] Furthermore, the model of the depth camera is Azure Kinect DK.

[0020] Furthermore, it also includes:

[0021] The difference between the sum of the pixels of two adjacent depth images is calculated and the difference is used as the quantitative index E(k) of the feeding intensity of the fish school.

[0022]

[0023] Among them, f(k) represents the sum of the target pixel points in the kth depth image, Z(x, y) represents the depth value at the coordinates x, y in the depth map, that is, the vertical distance value from the target to the depth camera, and the pixels of the depth image are M*N;

[0024] Taking J consecutive E(k) data as a group, all quantitative indicators of feeding intensity E(k) are divided into several groups of time series;

[0025] The time series is labeled as strong feeding, moderate feeding, weak feeding, and no feeding;

[0026] Constructing a gated recurrent unit network model, and using the labeled time series to train the gated recurrent unit network model to obtain a second detection model;

[0027] The first detection model and the second detection model are evaluated using accuracy, precision, recall, specificity, and F1 score; if the F1 score of the first detection model is higher than the F1 score of the second detection model, the first detection model is used to actually detect the feeding status of fish; otherwise, the second detection model is used to actually detect the feeding status of fish;

[0028] Accuracy=(TP+TN) / (TP+TN+FP+FN) (7)

[0029] Precision=TP / (TP+FP) (8)

[0030] Specificity=TN / (TN+FP) (9)

[0031] Recall=TP / (TP+FN) (10)

[0032] F1=2*Precision*Recall / (Precision+Recall) (11)

[0033] Where TP is the number of samples whose true value is positive and whose prediction is also positive; FP is the number of samples whose true value is negative but whose prediction is positive; FN is the number of samples whose true value is positive but whose prediction is negative; TN is the number of samples whose true value is negative and whose prediction is also negative.

[0034] Furthermore, the gated recurrent unit network model includes an embedding layer with a dimension of 32, two hidden layers, a fully connected layer with 10 convolution kernels, and a 4-classification output layer. The two hidden layers use 32 and 16 gated recurrent units as feature extraction units, respectively. A Dropout layer with a dropout rate of 0.2 is used to prevent network overfitting. The output layer uses a Softmax classification function to predict category probability, and optimizes training based on a cross-entropy loss function.

[0035] Beneficial effects

[0036] Compared with the prior art, the present invention has the following advantages:

[0037] This invention combines a depth camera with an image processing program to obtain real-time depth maps and feeding status sequence data of fish schools. Because the depth maps of fish feeding status have clear outlines and are clearly distinguishable from the background, they can be classified using a relatively simple convolutional network. Furthermore, the feeding status sequence data quantifies the feeding status of fish schools, using simple and effective quantification parameters, which also allows for classification using a simple recurrent neural network. This invention maintains high recognition accuracy while reducing the demand for computer resources, making it suitable for practical application in production environments. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 Schematic diagram of the detection structure of the present invention;

[0039] Figure 2 Schematic diagram of detection parameters of the depth camera in the present invention;

[0040] Figure 3 is the depth pseudo-color map in the present invention;

[0041] Figure 4 This is a simple convolutional neural network model diagram in the present invention;

[0042] Figure 5 This is a diagram of the gated recurrent unit network model in the present invention. DETAILED DESCRIPTION

[0043] The present invention will be further described below with reference to the specific embodiments in the accompanying drawings.

[0044] Example 1

[0045] See Figures 1 to 4 , a method for detecting the feeding status of fish, comprising:

[0046] The depth camera is used to collect depth images of a school of fish during a complete feeding process, such as Figure 1 As shown;

[0047] The depth image is converted into a depth pseudo-color image according to the depth change, such as Figure 3 As shown;

[0048] The deep pseudo-color images were labeled as strong feeding, moderate feeding, weak feeding, and no feeding;

[0049] Build a simple convolutional neural network model, such as Figure 4 As shown, the first detection model is obtained by training a simple convolutional neural network model using the labeled depth pseudo-color image;

[0050] The first detection model is used to actually detect the feeding status of fish.

[0051] In this embodiment, the model of the depth camera is Azure Kinect DK. Figure 1 As shown, depth camera 1 continuously emits modulated infrared light pulses into the breeding pond via infrared light emitter 2. The light strikes the surface of the feeding fish and, after reflection, enters depth camera 1. Because the distance from the infrared sensor to the fish varies from location to location when feeding, the reflected light travels different distances in space. Therefore, the phase difference between the transmitted and received pulses is detected at the receiving end. The depth camera's internal digital signal processor processes the phase difference matrix and reconstructs a depth map with a resolution of 640*576. The pixels in the depth map represent the depth of the object, reflecting the outline characteristics of the fish within the viewing angle.

[0052] In this example, the carp is detected. Its feeding habit is to float to the surface to eat bait. The Azure KinectDK depth camera is placed in the center of the breeding pond and shoots downward. Figure 2 The effective measurement range of the Azure Kinect DK depth camera is 0.5 to 3.58 meters. Exceeding this range will cause significant errors. Therefore, the distance between the depth camera and the water surface of the breeding pond was adjusted to 0.5 meters, so that the depth camera's field of view covers the entire surface of the breeding pond. The carp feeding area is 0.08 meters below the water surface, where the fish are most likely to feed. Therefore, the effective measurement range of the depth camera was set to 0.48 to 0.58 meters. This allows the depth camera to focus on the feeding area for imaging, eliminating interference from other fish that have not entered the feeding area and further improving the accuracy of the fish feeding intensity classification. The feeding intensity of the fish is shown in Table 1.

[0053] Feeding behavior level Fish schooling behavior Dataset category powerful Fish schools float to the surface to feed actively and have a large range of movement and a large distribution area. Feeding status middle Some fish in the school float to the surface and start feeding, with a small range of movement Feeding status weak A small number of fish in a school rise to the surface and only react to nearby bait. Feeding status none Fish do not respond to bait and hide underwater Non-feeding state

[0054] Table 1

[0055] Different colors are used to represent different depths. The distance information of the fish school in the depth image is linearly converted to the Hue value of the HSV color model. The color value of the HSV model is then converted into the corresponding RGB color value to obtain a depth pseudo-color image. The formula for calculating the Hue value is as follows:

[0056] Hue=γ×[1-(Z(x,y)-Z0) / (Z1-Z0)] (1)

[0057] In formula (1), Z(x, y) represents the depth value at the coordinates x and y in the depth map, that is, the vertical distance value from the target to the depth camera. Its value range is the effective measurement range set by the depth camera. [Z0, Z1] is the lower and upper limits of the effective measurement range set by the depth camera, that is, the value range is 500-580, Z0 and Z1 are 500 and 580 respectively. γ is an attenuation factor with a value of 0.67. When the fish changes from 500 to 580 mm in the water surface, the corresponding hue value range is 0-0.67. Then, the ColorConvertHSVtoRGB(H, S, V, R, G, B) method of the C# development platform is used to convert the HSV color model to the RGB model, where H is equal to the Hue value of formula (1), S and V are 1, and the converted R, G, B values are multiplied by 255 to obtain the actual RGB color value, thereby obtaining a pseudo-color image based on distance information. The pseudo-color image can improve the accuracy of judging the feeding status of fish.

[0058] In this embodiment, the simple convolutional neural network model includes, in sequence, an input layer with a dimension of 3, 3 hidden layers, a global pooling layer, and a 4-classification output layer; the depth pseudo-color image is input into the input layer, and the 3 hidden layers all use a 2D convolution kernel with a size of 5×5 and a step size of 1. The activation functions of the 3 hidden layers all use the Relu activation function, and the number of convolution kernels in the 3 hidden layers is 32, 64, and 128, respectively. Each convolution layer is followed by an average pooling layer with a step size of 2×2 for downsampling, and a Dropout layer with a dropout rate of 0.2 is used to prevent network overfitting; finally, a global pooling layer (Global Average Pooling, GAP) is used to reduce the calculation parameters, and a 4-classification output layer is connected, where the output layer uses the Softmax classification function to predict the category probability.

[0059] The output layer is also trained using a cross-entropy loss function, as follows:

[0060]

[0061] P in formula (2) i is the final predicted probability of category i, z i is the original budget output of the classification model for category i; z j is the original budget output of the classification model for category j, C is the total number of categories, L in formula (3) is the cross entropy loss function, y i is the label corresponding to category i.

[0062] Example 2

[0063] Furthermore, based on the embodiment, embodiment 2 also includes:

[0064] The difference between the sum of the pixels of two adjacent depth images is calculated and the difference is used as the quantitative index E(k) of the feeding intensity of the fish school.

[0065]

[0066] Among them, f(k) represents the sum of the target pixel values in the kth depth image, Z(x, y) represents the depth value (in mm) at the coordinates x and y in the depth image, that is, the vertical distance value from the target to the depth camera, and the pixels of the depth image are M*N. Since the depth image pixels are 640*576, the values of x and y are 0~640 and 0~576 respectively, that is, M=640, N=576, Z0 and Z1 are 500 and 580 respectively; Formula (6) indicates that when the depth value of the pixel point is in the range of 500~580mm (this area is the feeding area of the fish school), the value of this pixel point is set to 1 and represented by a grayscale value of 255 (white); pixels outside this range are represented by a grayscale value of 0 (black).

[0067] From Equation (5), we can see that the more intensely a school of fish swims around the water surface or up and down at different depths, the greater the difference in the total number of target pixels of the school of fish, E(k), indicating a higher feeding intensity of the school of fish, and vice versa. Therefore, E(k) can be used as a quantitative data indicator to quantify the feeding intensity of fish.

[0068] All feeding intensity quantitative indicators E(k) are divided into several groups of time series, with J continuous E(k) data as a group; in the feeding state, the video acquisition time is 6 to 8 minutes, and 1800 to 2400 E(k) data can be obtained. According to the length of the fish feeding state, 35 E(k) data are taken as a time series, that is, J = 35. Through each feeding activity, 10 to 20 groups of sequence data reflecting the feeding state of the fish are obtained.

[0069] The time series were labeled as strong feeding, moderate feeding, weak feeding, and no feeding;

[0070] Construct a gated recurrent unit network model, such as Figure 5 As shown, the second detection model is obtained by using the labeled time series to train the gated recurrent unit network model;

[0071] The first detection model and the second detection model are evaluated using accuracy, precision, recall, specificity, and F1 score. If the F1 score of the first detection model is higher than the F1 score of the second detection model, the first detection model is used to actually detect the feeding status of fish; otherwise, the second detection model is used to actually detect the feeding status of fish.

[0072] Accuracy=(TP+TN) / (TP+TN+FP+FN) (7)

[0073] Precision=TP / (TP+FP) (8)

[0074] Specificity=TN / (TN+FP) (9)

[0075] Recall=TP / (TP+FN) (10)

[0076] F1=2*Precision*Recall / (Precision+Recall) (11)

[0077] Where TP is the number of samples whose true value is positive and whose prediction is also positive; FP is the number of samples whose true value is negative but whose prediction is positive; FN is the number of samples whose true value is positive but whose prediction is negative; TN is the number of samples whose true value is negative and whose prediction is also negative.

[0078] The gated recurrent unit network model consists of an embedding layer with a dimension of 32, two hidden layers, a fully connected layer with 10 convolution kernels, and a 4-class output layer. The two hidden layers use 32 and 16 gated recurrent units as feature extraction units respectively. A dropout layer with a dropout rate of 0.2 is used to prevent network overfitting. The output layer uses the Softmax classification function to predict category probabilities, and the training is optimized based on the cross-entropy loss function.

[0079] This invention combines a depth camera with an image processing program to obtain real-time depth maps and feeding status sequence data of fish schools. Because the depth maps of fish feeding status have clear outlines and are clearly distinguishable from the background, they can be classified using a relatively simple convolutional network. Furthermore, the feeding status sequence data quantifies the feeding status of fish schools, using simple and effective quantification parameters, which also allows for classification using a simple recurrent neural network. This invention maintains high recognition accuracy while reducing the demand for computer resources, making it suitable for practical application in production environments.

[0080] The above is only a preferred embodiment of the present invention. It should be pointed out that for those skilled in the art, several modifications and improvements can be made without departing from the structure of the present invention. These will not affect the effect of the implementation of the present invention and the practicality of the patent.

Claims

1. A method for detecting the feeding status of fish, characterized in that: include: Use a depth camera to collect depth images of a school of fish during a complete feeding process; Converting the depth image into a depth pseudo-color image according to the depth change; The deep pseudo-color image is labeled according to strong feeding, moderate feeding, weak feeding, and no feeding; Constructing a simple convolutional neural network model, and using the labeled depth pseudo-color image to train the simple convolutional neural network model to obtain a first detection model; Using the first detection model to actually detect the feeding status of fish; Different colors are used to represent different depths. The distance information of the fish school in the depth image is linearly converted to the Hue value of the HSV color model. The color values of the HSV model are then converted into corresponding RGB color values to obtain a depth pseudo-color image. The formula for calculating the Hue value is as follows: Hue=γ×[1-(Z(x,y)-Z0) / (Z1-Z0)](1) In formula (1), Z(x, y) represents the depth value at the coordinates x, y in the depth map, that is, the vertical distance from the target to the depth camera. Its value range is the effective measurement range set by the depth camera, [Z0, Z1] is the lower and upper limits of the effective measurement range set by the depth camera, and γ is the attenuation factor with a value of 0.

67. The simple convolutional neural network model includes, in sequence, an input layer with a dimension of 3, 3 hidden layers, a global pooling layer, and a 4-classification output layer; the depth pseudo-color image is input into the input layer, and the 3 hidden layers all use a 2D convolution kernel with a size of 5×5 and a step size of 1. The activation functions of the 3 hidden layers all use the Relu activation function, and the number of convolution kernels in the 3 hidden layers are 32, 64, and 128, respectively. Each convolution layer is followed by an average pooling layer with a step size of 2×2 for downsampling, and a Dropout layer with a dropout rate of 0.2 is used to prevent network overfitting; finally, a global pooling layer is used to reduce the calculation parameters, and a 4-classification output layer is connected, where the output layer uses the Softmax classification function to predict the category probability.

2. A method for detecting fish feeding status according to claim 1, characterized in that: The output layer is also trained using a cross-entropy loss function, as follows: P in formula (2) i is the final predicted probability of category i, z i is the original budget output of the classification model for category i; z j is the original budget output of the classification model for category j, C is the total number of categories, L in formula (3) is the cross entropy loss function, y i is the label corresponding to category i.

3. The method for detecting fish feeding status according to claim 1, wherein: The model of the depth camera is Azure Kinect DK.

4. A method for detecting fish feeding status according to any one of claims 1 to 3, characterized in that: Also includes: The difference between the sum of the pixels of two adjacent depth images is calculated and the difference is used as the quantitative index E(k) of the feeding intensity of the fish school. Among them, f(k) represents the sum of the target pixel points in the kth depth image, Z(x, y) represents the depth value at the coordinates x, y in the depth map, that is, the vertical distance value from the target to the depth camera, and the pixels of the depth image are M*N; Taking J consecutive E(k) data as a group, all quantitative indicators of feeding intensity E(k) are divided into several groups of time series; The time series is labeled as strong feeding, moderate feeding, weak feeding, and no feeding; Constructing a gated recurrent unit network model, and using the labeled time series to train the gated recurrent unit network model to obtain a second detection model; The first detection model and the second detection model are evaluated using accuracy, precision, recall, specificity, and F1 score; if the F1 score of the first detection model is higher than the F1 score of the second detection model, the first detection model is used to actually detect the feeding status of fish; otherwise, the second detection model is used to actually detect the feeding status of fish; Accuracy=(TP+TN) / (TP+TN+FP+FN) (7) Precision=TP / (TP+FP) (8) Specificity=TN / (TN+PF) (9) Recall=TP / (TP+FN) (10) F1=2*Precision*Recall / (Precision+Recall) (11) Where TP is the number of samples whose true value is positive and whose prediction is also positive; FP is the number of samples whose true value is negative but whose prediction is positive; FN is the number of samples whose true value is positive but whose prediction is negative; TN is the number of samples whose true value is negative and whose prediction is also negative.

5. A method for detecting fish feeding status according to claim 4, characterized in that: The gated recurrent unit network model includes an embedding layer with a dimension of 32, two hidden layers, a fully connected layer with 10 convolution kernels, and an output layer with four categories. The two hidden layers use 32 and 16 gated recurrent units as feature extraction units, respectively. A Dropout layer with a dropout rate of 0.2 is used to prevent network overfitting. The output layer uses a Softmax classification function to predict category probabilities, and optimizes training based on a cross-entropy loss function.

Images