A fish feeding behavior recognition method based on Deeplabv3+ and FAIvar

Abstract

The present invention discloses a method for identifying fish feeding behavior based on Deeplabv3+ and FAIvar, and belongs to the field of image recognition technology. The present invention uses the Deeplabv3+ network to perform high-precision image segmentation on fish in waters to form an accurate fish segmentation image. The fish feeding behavior recognition module FAIvar is used to divide the results of fish segmentation into four regions; then the fish area percentage and the maximum fish area percentage in each sub-image are calculated to more comprehensively capture the spatial distribution characteristics of fish feeding behavior; the fish feeding index of each region is subtracted from the fish feeding index before the set number of frames and the variance is calculated, and the real-time monitoring and judgment of fish feeding behavior is achieved through the size of the variance. The real-time judgment method based on image analysis in the present invention can be effectively applied to the real-time monitoring and control of fish feeding behavior, and provide a reliable feeding strategy for the intelligent feeding machine.

Description

Technical Field

[0001] The present invention belongs to the technical field of image recognition, and in particular relates to a fish feeding behavior recognition method of Deeplabv3+ and FAIvar (fish feeding behavior recognition module). Background Art

[0002] With the explosion of science and technology, the fishery production model of developed countries abroad has gradually shifted from traditional farming to intelligent and information-based farming. However, my country is still in an important stage of transitioning from traditional farming to intelligent farming, and farming methods still rely mostly on manual labor. At present, computer vision has developed rapidly and has played a significant role in aquaculture, such as fish detection, fish identification, body length measurement, fish counting, feeding behavior identification, etc. Among them, the research on the recognition of fish feeding behavior is of great significance to the development of intelligent farming.

[0003] Traditional methods for identifying fish feeding behavior mainly rely on machine learning methods that manually extract features to achieve fish feeding behavior recognition. For example, Zhou Chao et al. used image enhancement and other methods to obtain binary images of fish in Near-infrared imaging to quantify the feeding behavior of fish in aquaculture, and calculated the grouping index of fish feeding behavior through Delaunary triangles, but this method is difficult to apply when the fish is occluded. At present, these methods are often greatly affected by factors such as poor real-time performance, background interference, and water splashes, making it difficult to effectively achieve real-time fish feeding behavior recognition, and therefore cannot directly provide accurate feeding strategies for intelligent feeding machines. Improvements in fish feeding behavior recognition methods can reduce the waste and environmental pollution caused by insufficient or excessive feeding during the breeding process. Therefore, improving methods for identifying fish feeding behavior is crucial to realizing intelligent fish farming. Summary of the invention

[0004] The purpose of the present invention is to design a fish feeding behavior recognition model, which segments fish targets through a deep learning model, and then combines the time series analysis of the fish status to further realize the accurate recognition of fish feeding behavior, thereby solving the problems of low model accuracy and poor real-time performance caused by overlapping fish targets and complex background in the fish feeding behavior recognition process in the prior art.

[0005] To achieve the above object, the present invention provides the following technical solutions:

[0006] A fish feeding behavior recognition method based on Deeplabv3+ and FAIvar includes the following steps:

[0007] S1: Collect fish feeding behavior videos, extract frames to form a fish semantic segmentation dataset and perform preprocessing.

[0008] S2: Input the preprocessed fish semantic segmentation dataset into the semantic segmentation module for training to train a fish semantic segmentation model.

[0009] S3: The fish semantic segmentation results output by the fish semantic segmentation model are input into the fish feeding behavior recognition module FAIvar. The fish feeding behavior recognition module FAIvar first segments each fish semantic segmentation result to obtain four sub-graphs.

[0010] S4: Calculate the percentage of all fish pixels in each of the four sub-images obtained in step S3 to all pixels in the sub-image FishAreaRatio i The percentage of the total number of pixels with the largest connected area in all segmented regions to all pixels in the sub-image AreaMaxRatio i , then set FishAreaRatio i and AreaMaxRatio i Add up to get the feeding behavior index FAI of each subgraph i ; where i is a subgraph, i ranges from 1 to 4.

[0011] S5: The FAI obtained in the four subgraphs are i The FAI of the corresponding area before the preset number of frames i Subtract the values ​​to get the FAIdiff of each subgraph i ; The pre-set number of frames is determined by the intensity of the fish's eating.

[0012] S6: FAIdiff for four subgraphs i If the variance is greater than the set threshold, it is judged to be in a feeding state.

[0013] Preferably, in step S1, the preprocessing of the fish semantic segmentation dataset is: manually annotating the collected fish images using the labelme tool, the annotation method is Create Polygons, the label is fish, the annotation file format is json, and finally the annotated dataset is converted into the VOC dataset format.

[0014] Preferably, in step S2, the DeepLabv3+ network structure is used as the fish semantic segmentation model. This model combines dilated convolution, multi-scale feature fusion and decoder, etc., and can efficiently and accurately segment the image into regions with semantic information; the specific process is:

[0015] S2.1: Preprocess the input fish images, including denoising and size normalization, to ensure that the input meets the requirements of the model;

[0016] S2.2: Input the processed images into the Deeplabv3+ semantic segmentation model for training to obtain the fish semantic segmentation model weights;

[0017] S2.3: Use the trained fish semantic segmentation model weights to segment the fish images and obtain the fish image segmentation results.

[0018] Preferably, in step S3, the method for segmenting the fish semantic segmentation results is: split the output fish semantic segmentation results according to a distance of half the width and half the height, that is, each fish semantic segmentation result obtains four sub-graphs.

[0019] The beneficial effects of the present invention are as follows: (1) the DeepLabV3+ semantic segmentation algorithm is used to accurately segment the fish targets in the image, providing accurate basic data for the subsequent feeding behavior analysis; (2) the image is divided into four regions horizontally and vertically, and then the fish area percentage and the maximum fish area percentage in each sub-image are calculated. This method of subdividing the image area can more comprehensively capture the spatial distribution characteristics of fish feeding behavior and make the analysis more refined; (3) by subtracting the feeding behavior index before the set number of frames and calculating the variance, the real-time monitoring and judgment of the fish feeding behavior is achieved through the size of the variance. The real-time judgment method based on image analysis in the present invention can be effectively applied to the real-time monitoring and control of fish feeding behavior, and has a high application value. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 The figure is a flow chart of the method for identifying fish feeding behavior of the present invention.

[0021] Figure 2 It is a structural diagram of the fish feeding behavior identification method of the present invention.

[0022] Figure 3 This is a structural diagram of the semantic segmentation module of the present invention.

[0023] Figure 4 This is an example of fish segmentation results output by the semantic segmentation module of the present invention.

[0024] Figure 5 It is a flow chart of the fish feeding behavior recognition module of the present invention.

[0025] Figure 6 This is an example diagram of the fish area percentage of the present invention.

[0026] Figure 7This is an example diagram of the maximum area percentage of the present invention.

[0027] Figure 8 This is an example of a non-feeding behavior result output by the feeding behavior recognition module of the present invention.

[0028] Fig. 9 This is an example of the feeding behavior result output by the feeding behavior recognition module of the present invention. DETAILED DESCRIPTION

[0029] In order to make the objectives, technical solutions and advantages of the present invention more clearly understood, the specific implementation modes of the present invention will be described in detail below with reference to the accompanying drawings.

[0030] like Figure 1-2 The fish feeding behavior recognition method shown in the figure includes two modules: semantic segmentation module and feeding behavior recognition module; the semantic segmentation module segments all fish targets in the fish image based on the Deeplabv3+ network, and the feeding behavior recognition module accurately recognizes the fish feeding behavior based on the FAIvar fish feeding behavior recognition model. The specific steps of the method are as follows:

[0031] S1: Collect fish feeding behavior videos, extract frames to form a fish semantic segmentation dataset and perform preprocessing.

[0032] The preprocessing of the fish semantic segmentation dataset is: manually annotate the collected fish images, and convert the dataset format after annotation. The annotation tool used is labelme, the annotation method is Create Polygons, the label is fish, the annotation file format is json, and finally the annotated dataset is converted to the VOC dataset format.

[0033] S2: The processed fish semantic segmentation dataset is input into the semantic segmentation module. The semantic segmentation module performs training and testing of fish semantic segmentation based on the Deeplabv3+ network.

[0034] The algorithm used in the semantic segmentation module of fish is the DeeplabV3+ algorithm, whose core principle is to use a deep convolutional neural network (CNN) to classify images at the pixel level. The DeepLabv3+ algorithm can effectively capture the semantic information in the image and perform well in preserving boundary details by introducing technologies such as dilated convolution and spatial pyramid pooling. Figure 3As shown in the figure, the semantic segmentation module based on the DeeplabV3+ algorithm extracts features through the Resnet101 backbone network, then passes through the hole convolution layer to increase the receptive field, and then performs multi-scale feature fusion to obtain richer semantic information. It is then upsampled by the decoder and fused with shallow features to improve segmentation accuracy, and finally sent to the classification layer to classify each pixel to obtain a semantic segmentation result with the same size as the input image.

[0035] First, the input fish images are preprocessed, including denoising and size normalization, to ensure that the input meets the requirements of the model; then the pre-trained DeepLabv3+ model is used to perform semantic segmentation on the preprocessed images. In this step, the semantic segmentation model assigns each pixel in the fish image to a different category, including fish and background. This fish segmentation method based on the DeepLabv3+ algorithm has the advantages of high accuracy, versatility, and automation. Compared with traditional methods, it can better cope with complex scenes and diverse fish images.

[0036] S3: The fish semantic segmentation result output by the semantic segmentation module is input into the fish feeding behavior recognition module FAIvar and preprocessed.

[0037] The fish semantic segmentation results are preprocessed by dividing them into two halves in the horizontal and vertical directions to obtain four equal-sized sub-images to observe the conditions in different areas of the water in more detail. To facilitate subsequent analysis, the upper left portion of the image is labeled top left, the upper right portion is labeled top right, the lower left portion is labeled bottom left, and the lower right portion is labeled bottom right.

[0038] S4: Calculate FishAreaRatio of four areas i and AreaMaxRatio i And add up to get the feeding behavior index FAI of each area i , where i is a subgraph.

[0039] When fish are feeding, they will quickly gather in one area. FishAreaRatio i The calculation is for all fish pixels in the sub-image Occupies all pixels in this sub-image This percentage can reflect the approximate number of fish in the sub-image area in the current frame and can be used to analyze the distribution characteristics of fish schools in the water area; the formula is:

[0040]

[0041] AreaMaxRatioi The pixel with the largest connected area in all segmented areas of the sub-image is calculated. The total number of pixels in this sub-image This percentage can clearly show the aggregation of fish; the formula is:

[0042]

[0043] FAI i The calculation is the percentage of the maximum area of ​​fish in each area AreaMaxRatio i FishAreaRatio i This value can not only reflect the distribution of fish schools in each area, but also reflect the aggregation characteristics of fish schools in each area; the formula is:

[0044] FAIi=FishAreaRatio i +AreaMaxRatio i

[0045] S5: The FAI obtained in the four subgraphs are i The value is the same as the FAI of the corresponding area 20 frames ago i Subtract the values ​​to get the FAIdiff of each subgraph i .

[0046] Calculated FAI i The value can only reflect the distribution and aggregation of each area of ​​the fish school in the current frame, but the feeding behavior of fish is a fast and continuous process. The dynamic changes of fish can be analyzed in combination with time to more accurately identify the feeding behavior of fish. Therefore, the FAI in the current frame can be used. i The value is the same as the FAI of the corresponding area 20 frames ago i The values ​​are compared, so that the changes in the feeding index of fish in each area within 20 frames can be clearly seen. The formula is:

[0047] FAIdiff i =FAI t-20(i) -FAI t(i)

[0048] S6: FAIdiff for four subgraphs i If the variance is greater than the set threshold, it is judged to be in the feeding state.

[0049] If fish are not feeding for a period of time, their movements are usually relatively slow, which means that the FAIdiff of the four areas iWhen fish are feeding, they usually swim quickly to one of the areas, causing the FAI in that area to i The value increased significantly, while the FAI in other areas i The value will decrease. In this case, the FAIdiff of the four regions i The values ​​will show a larger dispersion, resulting in a larger variance. It represents the FAIdiff of four regions within 20 frames. i The average value of the value is:

[0050]

[0051] In short, when fish are feeding, the FAIdiff in the four areas i Values ​​tend to show greater variation, which is reflected in the results having a larger variance, as given by:

[0052]

[0053] The size of the variance Var(FAI) reflects the change in the feeding behavior index of the fish school within a period of time. The threshold is set to 100. When the variance is greater than 100, the fish school is judged to be feeding; when the variance is less than 100, the state of the fish school is stable and it is judged to be non-feeding.

[0054] In order to verify the reliability of the method, 100 videos were randomly selected from the data set for experiment. The results showed that the fish feeding behavior recognition method of the present invention achieved an accuracy rate of 90%, which fully proved the reliability and effectiveness of the method.

[0055] The above are specific embodiments of the present invention, but the protection scope of the present invention should not be limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed by the present invention should be included in the protection scope of the present invention, so the protection scope of the present invention should be based on the protection scope defined in the claims.

Claims

1. A fish feeding behavior recognition method based on Deeplabv3+ and FAIvar, characterized in that: The method comprises the following steps: S1: Collect fish feeding behavior videos, extract frames to form a fish semantic segmentation dataset and perform preprocessing; S2: Input the preprocessed fish semantic segmentation dataset into the semantic segmentation module for training to train a fish semantic segmentation model; S3: inputting the fish semantic segmentation results output by the fish semantic segmentation model into the fish feeding behavior recognition module FAIvar. The fish feeding behavior recognition module FAIvar first segments each fish semantic segmentation result to obtain four sub-graphs. S4: Calculate the fish area percentage FishAreaRatio in each of the four sub-graphs obtained in step S3 i and the maximum area percentage of fish AreaMaxRatio i , then FishAreaRatio i and AreaMaxRatio i Add up to get the feeding behavior index FAI of each sub-graph i ; Where i is a subgraph, i ranges from 1 to 4; S5: The FAI obtained in the four subgraphs are i The FAI of the corresponding area before the preset number of frames i Subtract the values ​​to get the FAIdiff of each subgraph i ; S6: FAIdiff for four subgraphs i If the variance is greater than the set threshold, it is judged to be in a feeding state.

2. The method for identifying fish feeding behavior based on Deeplabv3+ and FAIvar according to claim 1, characterized in that: In step S4, FishAreaRatio i Represents all fish pixels in the sub-image Occupies all pixels in this sub-image The percentage is as follows: AreaMaxRatio i Represents the total number of pixels with the largest connected area in all segmented regions of the sub-image Occupies all pixels of this sub-image The percentage is as follows:

3. A method for identifying fish feeding behavior based on Deeplabv3+ and FAIvar according to claim 1 or 2, characterized in that: In step S2, the DeepLabv3+ network structure is used as the fish semantic segmentation model, and the specific process is as follows: S2.1: Preprocess the input fish images, including denoising and size normalization; S2.2: Input the processed images into the Deeplabv3+ semantic segmentation model for training to obtain the fish semantic segmentation model weights; S2.3: Use the trained fish semantic segmentation model weights to segment the fish images and obtain the fish image segmentation results.

4. A method for identifying fish feeding behavior based on Deeplabv3+ and FAIvar according to claim 1 or 2, characterized in that: In step S3, the method for segmenting the fish semantic segmentation results is: split the output fish semantic segmentation results according to a distance of half the width and half the height, that is, each fish semantic segmentation result obtains four sub-graphs.

5. The method for identifying fish feeding behavior based on Deeplabv3+ and FAIvar according to claim 3, characterized in that: In step S3, the method for segmenting the fish semantic segmentation results is: split the output fish semantic segmentation results according to a distance of half the width and half the height, that is, each fish semantic segmentation result obtains four sub-graphs.

6. A method for identifying fish feeding behavior based on Deeplabv3+ and FAIvar according to claim 1 or 2, characterized in that: In step S1, the preprocessing of the fish semantic segmentation dataset is: manually annotating the collected fish images using the labelme tool, the annotation method is Create Polygons, the label is fish, the annotation file format is json, and finally the annotated dataset is converted into the VOC dataset format.

7. The method for identifying fish feeding behavior based on Deeplabv3+ and FAIvar according to claim 6, characterized in that: In step S6, the threshold is set to 100.

Images