Fish Feeding Intensity Evaluation Method Based on Dual-Stream Residual Convolutional Neural Network
The double-flow residual convolutional neural network evaluates the intensity of fish feeding, which solves the problem of inaccurate assessment of feeding status in fish farming, and achieves more efficient fish feeding intensity assessment and more detailed grading, which improves the breeding benefits.
Patent Information
- Application Number
- CN202211613436.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-12-15
- Publication Date
- 2025-08-05
- Estimated Expiration
- 2042-12-15
AI Technical Summary
The prior art is difficult to accurately evaluate the intensity of fish feeding in fish farming, resulting in improper feeding, affecting the benefits of farming and healthy growth of fish.
Using a method based on a dual-stream residual convolutional neural network, the spatial position map and optical flow map of fish feeding are extracted by collecting videos, and the spatial and temporal flow ResNet50 network is built, and the characteristic layer is integrated to classify and evaluate the intensity of fish feeding.
It improves the accuracy of fish feeding intensity assessment, can more carefully classify the fish feeding status, and is suitable for large-scale breeding, reducing labor costs, and improving breeding benefits.
Smart Images
Figure CN115775369B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the fields of aquaculture and deep learning, and in particular to a fish feeding intensity assessment method based on a dual-stream residual convolutional neural network. Background Art
[0002] Currently, there are two main feeding methods in fish farming in my country. One is artificial feeding, which relies mainly on the farmers' own farming experience to control the feeding amount. This requires a high level of farming experience, consumes a lot of manpower costs in large-scale farming bases, and greatly reduces farming efficiency. The other is mechanical feeding, which has the disadvantage of ignoring the changes in the fish's feeding state and failing to automatically adjust the feeding amount according to the fish's appetite. This can easily lead to improper feeding, thus affecting the healthy growth of farmed fish and reducing farming efficiency. Therefore, in the fish farming process, accurately understanding the actual feeding status of fish will help to scientifically feed the feed, which is of great significance to achieving precise, efficient and green fish farming.
[0003] As a non-invasive, economical, and efficient method, machine vision technology has become an important research tool for detecting and assessing the feeding status of fish. This method can quickly detect the feeding status of farmed fish without affecting their normal growth and development. However, the features extracted by this method are artificially selected and cannot extract high-dimensional semantic information from the image. This leads to low detection accuracy and poor robustness, making it difficult to apply to actual aquaculture environments.
[0004] In recent years, deep learning technology has also been applied to detecting and identifying fish feeding patterns. Deep learning models have deeper network structures, and the features they extract are acquired through extensive neural network learning. They can tap into high-level semantic information hidden in images, making them less susceptible to factors such as the size and species of farmed fish, and uneven lighting in the aquaculture environment. Compared to traditional machine vision methods, they offer greater robustness. Therefore, it is necessary to apply deep learning modules to assess fish feeding intensity. Summary of the Invention
[0005] The purpose of this invention is to propose a fish feeding intensity assessment method based on a two-stream residual convolutional neural network, which can extract the correlation characteristics between the temporal and spatial information of fish feeding behavior, so that the extracted fish feeding behavior characteristics are more complete and the accuracy of feeding intensity assessment can be improved.
[0006] The present invention provides a fish feeding intensity assessment method based on a dual-stream residual convolutional neural network, comprising the following steps:
[0007] Step 1: Use a video acquisition device to capture fish feeding videos;
[0008] Step 2: Process the collected video to obtain the spatial position map of fish feeding and its corresponding optical flow map, classify the fish feeding intensity into several levels, construct a fish feeding intensity dataset according to the classification of the collected images by the fish feeding intensity, where the feeding intensity data of each level includes the spatial position map and its corresponding optical flow map, and divide this dataset into a training set and a test set;
[0009] Step 3: Build a two-stream residual convolutional neural network model, including a spatial stream and a temporal stream ResNet50 network. Each ResNet50 network contains 49 convolutional layers. After the spatial stream and the temporal stream ResNet50 networks are fused at two feature layers, they are connected to an average pooling layer, and the output of this average pooling layer is sent to a fully connected layer for fish feeding intensity classification. The number of neurons contained in this fully connected layer is the same as the number of categories to be classified;
[0010] Step 4: Use the training set to train the two-stream residual convolutional neural network model built in Step 3, and then use the test set to test the trained model. After verifying the effectiveness of the two-stream residual convolutional neural network, obtain a two-stream residual convolutional neural network model for classifying and evaluating fish feeding intensity;
[0011] Step 5: Input the spatial position map of fish feeding and its corresponding optical flow map into the spatial stream and the temporal stream ResNet50 networks respectively to extract the spatial and temporal features of fish feeding behavior. Fuse the features output by the spatial stream and the temporal stream ResNet50 networks at the feature layer, and finally perform class prediction through the average pooling layer and the fully connected layer.
[0012] Use the Farneback dense optical flow algorithm to process the collected video to obtain the spatial position map of fish feeding and its corresponding optical flow map, and use an extended polynomial to approximately represent the neighborhood values of each pixel point:
[0013] f(x) = x T Ax + b T x + c
[0014] In the formula: A is a symmetric matrix, b is a vector, and c is a scalar;
[0015] Represent the previous frame of fish feeding image as:
[0016]
[0017] Then the next frame of fish feeding image becomes after the global displacement d:
[0018] f2(x) = f1(x - d) = (x - d) T A1(x - d) + b1T (x - d)+c1
[0019] = x T A1x+(b1 - 2A1d) T x + d T A1d - b1 T d + c1
[0020] = x T A2x + b2 T x + c2
[0021] Using the principle that the luminance values between adjacent frames remain constant, make the coefficients of the corresponding terms of f1(x) and f2(x) equal, and obtain the following equations:
[0022] A2 = A1,
[0023] b2 = b1 - 2A1d,
[0024] c2 = d T A1d - b1 T d + c1
[0025] If A1 in the formula is a non - singular matrix, solve for the global displacement d:
[0026]
[0027] After calculating the dense optical flow field, color it using the Munsell color system, visualize the dense optical flow field as an optical flow image, and thus obtain the spatial position map of the fish feeding and its corresponding optical flow map.
[0028] The classification of the fish feeding intensity is divided into 5 levels: strong, relatively strong, normal, relatively weak, and weak.
[0029] The convolutional layer of the ResNet50 network described above includes 5 convolutional modules. The first convolutional module contains a 7×7 convolutional layer, followed by a max - pooling layer; the latter four convolutional modules are stacked by multiple residual structures, and each residual structure consists of three convolutional layers with convolutional kernel sizes of 1×1, 3×3, and 1×1 respectively. The number of residual structures included in the second to fifth convolutional modules is 3, 4, 6, and 3 respectively, and the dimensions of the feature maps output by the second to fifth convolutional modules are 256 - dimensional, 512 - dimensional, 1024 - dimensional, and 2048 - dimensional respectively; an average - pooling layer is connected after the fifth convolutional module.
[0030] Fusion is performed at the feature layer after the last convolutional module of the spatial - flow and temporal - flow ResNet50 networks. Two feature layers with 2048 channels are concatenated at the same spatial position, and the fused feature layer with 4096 channels is obtained.
[0031] The dual-stream residual convolutional neural network adopts the following loss function:
[0032]
[0033] In the formula: m represents the number of samples, n represents the number of classification categories, y ic represents the true probability of the i-th sample, and p ic represents the predicted probability of the i-th sample.
[0034] A computer device includes a processor and a memory. A computer program is stored in the memory. It is characterized in that when the computer program is executed by the processor, it realizes any one of the above-mentioned fish feeding intensity evaluation methods based on the dual-stream residual convolutional neural network.
[0035] A computer-readable storage medium stores a computer program on it. It is characterized in that when the computer program is executed by the processor, it realizes any one of the above-mentioned fish feeding intensity evaluation methods based on the dual-stream residual convolutional neural network.
[0036] The beneficial effects of the present invention are as follows:
[0037] (1) The present invention divides the fish feeding intensity level more detailedly into 5 levels: strong, relatively strong, normal, relatively weak, and weak, which is applicable to the evaluation of fish feeding intensity in large-scale industrial aquaculture ponds and earth ponds.
[0038] (2) The present invention makes corresponding improvements to the problems existing in the traditional Two-stream. It uses the ResNet50 network to replace the VGG16 network with a relatively shallow network structure to extract sufficient fish feeding behavior features, improving the evaluation accuracy of the network for fish feeding intensity; and through the feature layer fusion method, it fuses the features extracted by the spatial stream and temporal stream networks, enabling the network to be trained in parallel to extract the correlation features between the spatio-temporal information of fish feeding behavior, thereby improving the evaluation accuracy of fish feeding intensity. The dual-stream residual convolutional neural network adopted by the present invention utilizes richer fish feeding behavior features, including spatial position features, movement trends, speeds, etc. of fish feeding behavior, and can perform more accurate evaluations. Moreover, the classification algorithm of the dual-stream residual convolutional neural network is ResNet50, and the network structure has 50 layers, which is much higher than the 7-layer network structure of LeNet-5, and can extract more sufficient fish feeding behavior features, improving the evaluation accuracy of feeding intensity. BRIEF DESCRIPTION OF THE DRAWINGS
[0039] Figure 1 It is a schematic diagram of the structure of the dual-stream residual convolutional neural network in the present invention;
[0040] Figure 2Schematic diagram of the change of training loss of the dual-stream residual convolutional neural network in the present invention;
[0041] Figure 3 Schematic diagram of the change of the accuracy rate of the dual-stream residual convolutional neural network in the present invention on the test set;
[0042] Figure 4 Feeding intensity discrimination rule table of fish in the present invention;
[0043] Figure 5 Comparison table of evaluation accuracy rates of the dual-stream residual convolutional neural network, Two-stream, and dual-stream network in the present invention;
[0044] Figure 6 Comparison table of the evaluation results of the dual-stream residual convolutional neural network in the present invention and the evaluation results of aquaculture technicians in the seven-day fish feeding intensity evaluation application test.
[0045] The following further details the present invention in conjunction with the accompanying drawings and specific embodiments. Specific embodiments
[0046] In the first embodiment, the present invention provides a method for evaluating the feeding intensity of fish based on a dual-stream residual convolutional neural network. The specific steps include:
[0047] Step 1: Use video acquisition devices such as near-infrared industrial cameras and Hikvision surveillance cameras to collect multiple batches of fish feeding videos;
[0048] Step 2: Use the Farneback dense optical flow algorithm to process the collected videos, obtain the spatial position map of fish feeding and its corresponding optical flow map, and divide the fish feeding intensity into 5 levels: strong, relatively strong, normal, relatively weak, and weak. As Figure 4 shown, construct a fish feeding intensity dataset by classifying the collected images according to the fish feeding intensity, where the feeding intensity data of each level includes the spatial position map and its corresponding optical flow map, and divide this dataset into a training set and a test set;
[0049] The Farneback dense optical flow algorithm does not require the scene space in the video to be stationary, and is applicable to extracting the optical flow information generated by large-scale movements of objects. By estimating the global displacement amount between two consecutive frames of images, the problem of rapid change of gray values in the images is solved. Each pixel neighborhood value is approximately represented by an extended polynomial:
[0050] f(x) = x T Ax + b T x + c
[0051] In the formula: A is a symmetric matrix, b is a vector, and c is a scalar;
[0052] The previous frame of fish feeding image is represented as:
[0053] f1(x)=x T A1x+b1 T x+c1
[0054] Then the next frame of fish feeding image becomes:
[0055]
[0056] By using the principle of constant brightness between adjacent frames, the corresponding coefficients of f1(x) and f2(x) are made equal, and the following equation is obtained:
[0057] A2=A1,
[0058] b2=b1-2A1d,
[0059] c2=d T A1d-b1 T d+c1
[0060] If A1 is a non-singular matrix, the global displacement d can be solved:
[0061]
[0062] After the dense optical flow field is calculated, it is colored using the Munsell color system and visualized as an optical flow image, thereby obtaining the spatial position map of the fish's feeding and its corresponding optical flow map;
[0063] Step 3: Build a two-stream residual convolutional neural network model, such as Figure 1 As shown, it includes spatial stream and temporal stream ResNet50 networks, each ResNet50 network contains 49 convolutional layers. After the two feature layers are fused, the spatial stream and temporal stream ResNet50 networks are connected to an average pooling layer to enhance the consistency of feature mapping and classification categories. The average pooling layer outputs to a fully connected layer for fish feeding intensity classification. The number of neurons contained in the fully connected layer is consistent with the number of categories required for classification, which is 5 in the present invention.
[0064] The convolutional layer of the ResNet50 network described above includes 5 convolutional modules. The first convolutional module contains a 7×7 convolutional layer, followed by a max pooling layer. The subsequent four convolutional modules are stacked by multiple residual structures, and each residual structure consists of three convolutional layers with kernel sizes of 1×1, 3×3, and 1×1 respectively. The number of residual structures included in the second to fifth convolutional modules are 3, 4, 6, and 3 respectively, and the dimensions of the feature maps output by the second to fifth convolutional modules are 256 dimensions, 512 dimensions, 1024 dimensions, and 2048 dimensions respectively. An average pooling layer is connected after the fifth convolutional module to enhance the consistency between feature mapping and classification categories;
[0065] Compared with the traditional Two-stream network, the above-mentioned two-stream residual convolutional neural network model has two improvements. The first is to use the ResNet50 network to replace the VGG16 network with a shallower network structure to extract sufficient fish feeding behavior features. The second is to use the feature layer fusion method to fuse the spatio-temporal features extracted by the spatial stream and temporal stream ResNet50 networks, enabling the spatial stream and temporal stream ResNet50 networks to be trained in parallel to extract the correlation features between the spatio-temporal information of fish feeding behavior;
[0066] For the two-stream residual convolutional neural network of the present invention, feature layer fusion is set after the last convolutional module of the spatial stream and temporal stream ResNet50 networks. The feature layer fusion is to splice two feature layers with 2048 channels at the same spatial position, and the fused feature layer has 4096 channels;
[0067] Step 4: Use the training set to train the two-stream residual convolutional neural network model built in Step 3, save the trained model, and then use the test set to test the trained model. After verifying the effectiveness of the two-stream residual convolutional neural network, obtain the two-stream residual convolutional neural network model for classifying and evaluating the fish feeding intensity;
[0068] Step 5: Input the spatial position map of fish feeding and its corresponding optical flow map into the spatial stream and temporal stream ResNet50 networks respectively to extract the spatial and temporal features of fish feeding behavior. Fuse the features output by the spatial stream and temporal stream ResNet50 networks at the feature layer, and finally perform class prediction through the average pooling layer and fully connected layer.
[0069] The two-stream residual convolutional neural network uses the following loss function:
[0070]
[0071] In the formula: m represents the number of samples, n represents the number of classification categories, y icrepresents the true probability of the i-th sample, p ic represents the predicted probability of the i-th sample.
[0072] During the training process of this two-stream residual convolutional neural network, the value of the network loss function is recorded every time an epoch is completed. Its changing trend is as Figure 2 shown. It can be seen from the figure that during the training process, the network loss drops rapidly before approximately the 18th epoch, and gradually stabilizes after the 50th epoch, proving that the network model has achieved convergence.
[0073] After the two-stream residual convolutional neural network finishes training each epoch, it is evaluated through the test set, and the accuracy of the network at this time is recorded. The change in the accuracy of the network during the training stage is as Figure 3 shown. It can be known from the figure that the accuracy of the network during training rises rapidly, and the accuracy reaches more than 95% at the 10th epoch. As the training progresses, the accuracy rises slowly, and gradually stabilizes after the 20th epoch. Finally, the accuracy of the network reaches 98.6%.
[0074] Testing the trained model is to verify the effectiveness of the two-stream residual convolutional neural network. It is trained on the same training set, and at the same time, the evaluation accuracy of the network is compared on its test set. The experimental environment and the set parameters during the experiment are kept consistent. The comparison table of the evaluation accuracies of the two-stream residual convolutional neural network, Two-stream, and the two-stream network is as Figure 5 shown. Among them, the evaluation accuracy of the Two-stream network is 95.4%. After replacing the classification algorithm VGG-16 of Two-stream with ResNet50, the two-stream network is obtained, and its evaluation accuracy is 96.2%. Compared with the Two-stream network, the accuracy has a slight improvement of 0.8%, indicating that ResNet with a deeper network structure can extract more effective features than VGG-16. The accuracy of the two-stream residual convolutional neural network of the present invention is 98.6%, which is 3.2% and 2.4% higher than that of the Two-stream network and the two-stream network respectively, indicating that in the feature layer fusion, the network can fully learn the spatio-temporal correlation features of fish feeding behavior, so that the two-stream residual convolutional neural network of the present invention has a higher evaluation accuracy for fish feeding intensity.
[0075] In order to further verify the effectiveness of the two-stream residual convolutional neural network of the present invention in fish feeding intensity evaluation, an application experiment was carried out in a fish earthen pond for seven days. The comparison of the network evaluation results and the evaluation results of aquaculture technicians is as Figure 6 shown. The evaluation results of the aquaculture technicians are consistent with the evaluation results of the two-stream residual convolutional neural network of the present invention.
[0076] In Embodiment 2, the present invention provides a computer device, which includes a processor, a memory, a network interface, a display screen, and an input device connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. When the computer program is executed by the processor, it realizes the method for evaluating the feeding intensity of fish based on the dual-stream residual convolutional neural network in Embodiment 1. Embodiment 1 and Embodiment 2 have the same functions and beneficial effects.
[0077] In Embodiment 3, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by the processor, it realizes the method for evaluating the feeding intensity of fish based on the dual-stream residual convolutional neural network in Embodiment 1. Embodiment 1 and Embodiment 3 have the same functions and beneficial effects.
[0078] Those of ordinary skill in the art can understand that to implement all or part of the processes in the above embodiments, it can be completed by instructing relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above method. Among them, the non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. The volatile memory can include random access memory (RAM) or an external cache memory. By way of illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.
[0079] The technical features of the above embodiments can be combined arbitrarily. For the sake of concise description, not all possible combinations of the technical features in the above embodiments are described. However, as long as these combinations of technical features do not conflict, they should be considered as the scope recorded in this specification.
[0080] The above-described embodiments merely represent several implementation manners of the present application. The description thereof is relatively specific and detailed, but it should not be construed as a limitation on the scope of the invention patent. It should be noted that for those of ordinary skill in the art, without departing from the concept of the present application, several modifications and improvements can be made, and these all fall within the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be subject to the appended claims.
Claims
1. A fish feeding intensity assessment method based on a two-stream residual convolutional neural network is characterized by The steps include: Step 1: Use a video acquisition device to capture fish feeding videos; Step 2: Process the collected video to obtain the spatial position map of fish feeding and its corresponding optical flow map, classify the fish feeding intensity into several levels, and construct a fish feeding intensity dataset based on the collected images according to the fish feeding intensity classification, where the feeding intensity data of each level includes the spatial position map and its corresponding optical flow map, and the dataset is divided into a training set and a test set; Step 3: Build a two-stream residual convolutional neural network model, including spatial and temporal stream ResNet50 networks. Each ResNet50 network contains 49 convolutional layers. After the two feature layers are fused, the spatial and temporal stream ResNet50 networks are connected to an average pooling layer. The average pooling layer outputs to a fully connected layer for fish feeding intensity classification. The number of neurons in the fully connected layer is consistent with the number of categories required for classification. Step 4: Use the training set to train the two-stream residual convolutional neural network model built in step 3, and then use the test set to test the trained model. After verifying that the two-stream residual convolutional neural network is effective, a two-stream residual convolutional neural network model for classifying and evaluating fish feeding intensity is obtained; Step 5: Input the spatial position map of fish feeding and its corresponding optical flow map into the spatial stream and temporal stream ResNet50 networks respectively to extract the spatial and temporal features of fish feeding behavior. The features output by the spatial stream and temporal stream ResNet50 networks are fused at the feature layer, and finally the category prediction is performed through the average pooling layer and the fully connected layer.
2. The method for estimating fish feeding intensity based on a dual-stream residual convolutional neural network according to claim 1, wherein: The Farneback dense optical flow algorithm is used to process the collected video to obtain the spatial position map of fish feeding and its corresponding optical flow map. An extended polynomial is used to approximate the neighborhood value of each pixel: Where: A is a symmetric matrix, b is a vector, and c is a scalar; The previous frame of fish feeding image is represented as: Then the next frame of fish feeding image becomes: Using the principle of constant brightness between adjacent frames, f1 ( x )and f2 ( x ) The coefficients of the corresponding terms are equal, and the following equation is obtained: If A1 is a non-singular matrix, the global displacement d is solved as: After the dense optical flow field is calculated, it is colored using the Munsell color system and visualized as an optical flow image, thereby obtaining the spatial position map of the fish's feeding and its corresponding optical flow map.
3. The method for estimating fish feeding intensity based on a dual-stream residual convolutional neural network according to claim 1, wherein: The fish feeding intensity is classified into five levels: strong, relatively strong, normal, relatively weak and weak.
4. The method for estimating fish feeding intensity based on a dual-stream residual convolutional neural network according to claim 1, wherein: The convolutional layer of the ResNet50 network includes 5 convolutional modules, of which the first convolutional module contains a 7×7 convolutional layer, followed by a maximum pooling layer; and the next four convolutional modules are composed of multiple residual structures stacked, each residual structure consists of three convolutional layers, and the convolution kernel sizes are 1×1, 3×3 and 1×1 respectively. The number of residual structures contained in the second to fifth convolutional modules are 3, 4, 6 and 3 respectively, and the dimensions of the feature maps output by the second to fifth convolutional modules are 256 dimensions, 512 dimensions, 1024 dimensions and 2048 dimensions respectively; the average pooling layer is connected after the fifth convolutional module.
5. The method for estimating fish feeding intensity based on a dual-stream residual convolutional neural network according to claim 1 or 4, characterized in that: After the last convolutional module of the spatial stream and temporal stream ResNet50 network, the feature layer is fused. The two feature layers with 2048 channels are spliced at the same spatial position, and the fused feature layer has 4096 channels.
6. The method for estimating fish feeding intensity based on a dual-stream residual convolutional neural network according to claim 1 or 4, characterized in that: The two-stream residual convolutional neural network adopts the following loss function: Where: m represents the number of samples, n represents the number of classification categories, yic represents the true probability of the i-th sample, and pic represents the predicted probability of the i-th sample.
7. A computer device comprising a processor and a memory, wherein the memory stores a computer program, wherein: When the computer program is executed by a processor, it implements the fish feeding intensity assessment method based on a dual-stream residual convolutional neural network as described in any one of claims 1 to 6.
8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, it implements the fish feeding intensity assessment method based on a dual-stream residual convolutional neural network as described in any one of claims 1 to 6.