Method for identifying abnormal fish belly state based on machine vision

Abstract

The present application relates to a kind of fish belly state anomaly identification method based on machine vision, belong to the field of aquaculture, comprising the following steps: S1, underwater waterproof camera is used to collect video information, and fish body key point detection model is constructed based on the video information;S2, fish body each part instance segmentation model is constructed, and each part of fish body is segmented;S3, join tracking model, and fish body in video information is tracked and photographed, and fish body image is obtained;S4, the validity of fish is judged, and effective fish body image is obtained;S5, fish belly state identification is carried out based on the fish body image.The present application has the advantages that: through the identification of fish belly state, whether fish is hungry or not can be found in time, or whether fish is ill, can make aquaculture user find the abnormal state of fish in time and make corresponding measures.

Description

TECHNICAL FIELD

[0001] The present application relates to a fish belly state abnormality recognition method based on machine vision, belonging to the field of aquaculture. BACKGROUND

[0002] As a perennial industry, aquaculture has been one of the fastest growing industries in the food industry in the past two decades. The nationwide aquaculture maintains a good development momentum, the aquaculture scale is further expanded, the structural adjustment makes new progress, and the dominant aquaculture areas and leading aquaculture varieties in the key provinces of fishery are gradually formed. The huge market demand of aquaculture drives the rapid development of aquaculture industry.

[0003] In recent years, China's aquaculture output accounts for about 2 / 3 of the world's aquaculture output, among them, the total output of freshwater aquaculture has been ranked first in the world for many years, and in recent years it has increased by about 22% per year, accounting for 42% of the total output of water products in China; According to the data of China fishery statistical yearbook, in 2015, China's fish farming output reached the highest, and then in 2016, the fish farming output fell rapidly, from the highest of 2845.77 million tons to 2670.98 million tons, with a decline of 6.14%. Since 2016, China's fish farming output has begun to recover slowly, and by 2020, China's fish farming output has recovered to 2761.36 million tons, up 1.95% year on year. China will implement the water environment health aquaculture system, play the ecological service function of aquaculture industry, and vigorously develop high-quality, characteristic, green and ecological aquatic products. By 2022, the demonstration area of healthy aquaculture will reach more than 65%, and the qualified rate of aquatic product sampling in production area will remain above 98%. With the upgrading of consumption, China is a big country of aquatic consumption, and healthy aquatic products directly determine the position and voice of China's water industry in the world.

[0004] There are still many blank places in China's aquaculture industry, which is not as mature as other industries. Because of this, its growth space is huge, and more and more people are waiting to join this family to fill in the blanks. The vast unknown field of aquaculture is waiting for people to explore and develop.

[0005] The existing fish belly state recognition method is through the naked eye observation of the breeder, which is low in efficiency and high in labor cost. The present application adopts fish belly state recognition, adopts recognition method based on machine vision, can carry out non-contact measurement, will not cause any damage to the observer and the observed person, and adopts machine vision method to carry out measurement, analysis and identification task for a long time, realizes the rapid and accurate measurement of fish belly state. Through the recognition of fish belly state by the breeder, the abnormal condition of fish can be found in time, and corresponding measures can be taken to reduce the loss, provide effective prevention and control measures for aquaculture industry, and improve the breeding efficiency and fish health level. SUMMARY

[0006] In order to overcome the defects of the prior art, the present application provides a fish belly state anomaly recognition method based on machine vision, and the technical scheme of the present application is:

[0007] A fish belly state anomaly recognition method based on machine vision, comprising the following steps:

[0008] S1, collecting video information by using an underwater waterproof camera, and constructing a fish body key point detection model based on the video information;

[0009] S2, constructing a fish body part instance segmentation model to segment each part of the fish body;

[0010] S3, adding a tracking model to track and shoot the fish body in the video information to obtain a fish body image;

[0011] S4, determining the validity of the fish to obtain a valid fish body image;

[0012] S5, recognizing the fish belly state based on the fish body image.

[0013] The step S1 specifically comprises:

[0014] S1-1, constructing a fish body key point detection image data set:

[0015] (1) collecting fish body images and performing data labeling on the fish body images to obtain labeled image data;

[0016] (2) dividing the labeled image data into a training set, a validation set and a test set according to a preset ratio to obtain the fish body key point detection image data set;

[0017] S1-2, training a fish body key point detection model based on the fish body key point detection data set:

[0018] (3) preprocessing the fish body key point detection image data set to obtain preprocessed images;

[0019] (4) inputting the preprocessed images into a key point detection network for training to obtain the fish body key point detection model.

[0020] The step S2 specifically comprises:

[0021] S2-1, preprocessing a fish body part instance segmentation image data set to obtain preprocessed images;

[0022] S2-2, inputting the preprocessed images into an instance segmentation network for training to obtain the fish body part instance segmentation model.

[0023] The step S4 is specifically:

[0024] When the fish is detected, the fish is judged for effectiveness, first, four key points m, n, a and b of the fish body are detected through the fish body key point detection model, the key point m is the eye part of the fish, the key point n is the middle part of the fish tail, the key point a is the rear side of the intersection between the lower fish body and the front fin, and the key point b is the front side of the intersection between the lower fish body and the rear fin;

[0025] A straight line is connected through the coordinates of the two key points m and n, so as to detect the inclination angle of the fish body, the fish body angle within 40 degrees is effective, and the fish image with an angle greater than 40 degrees is removed, and the fish image without detecting the key points a and b is removed;

[0026] When the angle is judged, the coordinates of m are (x1, y1) and the coordinates of n are (x2, y2) through the key point detection model, and the inclination angle of the fish is calculated through the following formula , .

[0027] The step S5 is specifically:

[0028] S5-1, draw a standard line of the fish belly in a normal state;

[0029] S5-2, draw a fish belly actual state curve;

[0030] S5-3, judge the difference between the fish belly actual state curve and the standard line, and finally output the recognition effect.

[0031] The step S5-1 is specifically:

[0032] In a normal state, the fish belly curvature of the same species of fish is similar, and one hundred fish belly images in a normal state are collected, at this time, the fish belly is convex or horizontal; the fish belly is concave in an abnormal state;

[0033] A standard line is determined in the form of a parabola determined by three points, standard key points a, b and c are set, among the standard key points a, b and c, the standard key point c is the middle point of the connecting line between the two standard key points a and b along the fish belly, the coordinates of the three standard key points a, b and c are obtained through the fish body part instance segmentation model, that is, the standard key point a is the point with the maximum horizontal coordinate among the intersection points of the front fin and the fish body, the standard key point b is the point with the minimum horizontal coordinate among the intersection points of the rear fin and the fish body, and the horizontal coordinate of the point c is obtained by averaging the horizontal coordinates of the standard key points a and b, and the maximum vertical coordinate of the fish body under the horizontal coordinate of the point c is the vertical coordinate of the point c; the values of the three standard key points a, b and c of all fish are recorded; based on the parabola formula The horizontal coordinates and the vertical coordinates of the standard key points a, b and c are brought into the parabola formula, when g>0, the greater g is, the greater the bending degree of the parabola is; when g<0, the smaller g is, the greater the bending degree is, the fish from 0 degree to 40 degree is arranged according to the degree from small to large, and the value of g presents an increasing state, when the fish is horizontal, g=0.3; when the fish is inclined by 45 degrees, g=0.7, according to the scatter diagram of g, the function formula of g is fitted: g=0.01x+0.3; x is the inclination angle of the fish body.

[0034] The step S5-2 is specifically:

[0035] The actual state curve of the fish is also a parabola, including actual key points a, b and a middle point d along the connecting line between the two actual key points a and b, the middle point d is an actual key point, the standard key points a and b coincide with the actual key points a and b; the horizontal coordinate of the point d is obtained by averaging the horizontal coordinates of the actual key points a and b, the maximum vertical coordinate of the fish body under the horizontal coordinate of the point d is the vertical coordinate of the point d, and the parabola equation is solved through the horizontal coordinates and the vertical coordinates of the three points a, b and d , that is, the actual state curve of the fish belly.

[0036] The step S5-3 is specifically:

[0037] When the quadratic term coefficient g in the parabola equation of the actual state curve of the fish is less than 0, it is indicated that the curve is concave inward, and when g is greater than 0, it is indicated that the curve is convex outward.

[0038] When g is less than 0 and the difference between the actual state curve of the fish and the standard line is greater than 20%, the fish belly abnormal state 1 is output, indicating that the fish is sick and does not eat for a long time.

[0039] When g is greater than 0 and the difference between the actual state curve of the fish and the standard line is greater than 20%, the fish belly abnormal state 2 is output, indicating that the fish belly is swollen.

[0040] The advantage of the present application is that the state of the fish belly can be recognized to find whether the fish is hungry or sick in time, so that the aquaculture user can find the abnormal state of the fish in time and take corresponding measures.

[0041] The present application promotes intelligent aquaculture, guides the deep integration of modern information technology such as artificial intelligence and aquaculture production, and promotes the mutual integration and coordinated development of the primary, secondary and tertiary industries. BRIEF DESCRIPTION OF DRAWINGS

[0042] Figure 1 is a flowchart of the present application.

[0043] Figure 2 is a position diagram of four key points on the fish.

[0044] Figure 3 is a schematic diagram of the fish belly of a fish in a normal state.

[0045] Figure 4 is a schematic diagram of the fish belly of a fish in an abnormal state.

[0046] Figure 5 is Figure 3 a schematic diagram of the position of the standard key point c.

[0047] Figure 6 is a schematic diagram of the actual state curve and the standard state curve of the present application. DETAILED DESCRIPTION

[0048] The advantages and features of the present application will become more apparent with the description of the specific embodiments. However, these embodiments are only exemplary and do not constitute any limitation on the scope of the present application. Those skilled in the art should understand that the details and forms of the technical solutions of the present application can be modified or replaced without departing from the spirit and scope of the present application, and such modifications and replacements all fall within the protection scope of the present application.

[0049] Referring to Figures 1 to 6 , the present application relates to a fish belly state abnormality recognition method based on machine vision, comprising the following steps:

[0050] S1, collecting video information by using an underwater waterproof camera, and constructing a fish body key point detection model based on the video information;

[0051] S2, constructing a fish body part instance segmentation model to segment each part of the fish body;

[0052] S3, adding a tracking model to track and shoot the fish body in the video information, and obtaining a fish body image;

[0053] S4, judging the effectiveness of the fish to obtain an effective fish body image;

[0054] S5, recognizing the fish belly state based on the fish body image.

[0055] The step S1 specifically comprises:

[0056] S1-1, constructing a fish body key point detection image data set:

[0057] (1) collecting fish body images and data labeling the fish body images to obtain labeled image data;

[0058] (2) dividing the labeled image data into a training set, a validation set and a test set according to a preset ratio to obtain the fish body key point detection image data set;

[0059] S1-2, training a fish key point detection model based on the fish key point detection dataset;

[0060] (3) Preprocessing the fish key point detection image dataset to obtain preprocessed images;

[0061] (4) Inputting the preprocessed images into a key point detection network for training to obtain the fish key point detection model.

[0062] In this embodiment, an underwater waterproof camera collects video information, and data labeling is performed on images above and beside the fish to obtain labeled image data. The labeled image data is divided into a training set, a validation set and a test set according to a preset ratio to obtain the fish key point detection image dataset. In this embodiment, a fish key point detection model is constructed through a yolopose detection network, video information is collected using an edge device, videos are cut into pictures using PotPlayer software, key point data labeling is performed on the pictures using labelme software to obtain labeled image data, the labeling information of each image is matched from the original labeling file of the dataset according to the name, and the YOLO labeling format file is converted. Finally, the training set, the validation set and the test set are divided according to the proportions of 80%, 10% and 10%, respectively, to form the fish key point detection dataset.

[0063] The yolopose algorithm is used to train the key point detection model because the model trained through this network can detect key points and detect fish targets.

[0064] The training method of the fish key point detection model includes preprocessing a fish target detection image dataset to obtain preprocessed images, inputting the preprocessed images into a yolopose network, training the yolopose network, and obtaining a fish key point detection model. In this embodiment, the yolopose network uses CSP-darknet53 as Backbone, and uses PANet to fuse features of different scales from Backbone. Next, there are 4 different scale Heads. Finally, there are 2 Decoupled Heads for predicting boxes and key points.

[0065] The fish key point detection in this embodiment has one class, fish and four key points, the Box head predicts 6 elements of each Anchor, which correspond to the bounding box, target score and confidence score of one class, respectively. There are 3 Anchors of different shapes corresponding to each grid position.

[0066] A fish has 4 associated keypoints, each with a re-identified location and confidence. So, 4 keypoints associated with an Anchor have 12 elements in total.

[0067] Therefore, for each Anchor, the Keypoint Head predicts 12 elements and the Box head predicts 6 elements. For an Anchor with n keypoints, the overall prediction vector is defined as:

[0068] ;

[0069] Keypoint confidence is trained based on the visibility flag of the keypoint. If a keypoint is visible or occluded, the Ground Truth confidence is set to 1, otherwise, if the keypoint is outside the field of view, the confidence is set to 0.

[0070] During inference, the confidence of a keypoint is kept greater than 0.5. All other predicted keypoints are masked out. The predicted keypoint confidence is not used for evaluation. However, since the network predicts all 4 keypoints for each detection, it is necessary to filter out the keypoints outside the field of view. Otherwise, there will be confidence keypoints that cause the deformed skeleton. Existing Bottom-up methods based on Heatmap do not need to do this, because the keypoints outside the field of view are not detected in the first place.

[0071] The step S2 is specifically:

[0072] S2-1, pre-process the fish body part instance segmentation image data set to obtain a pre-processed image;

[0073] S2-2, input the pre-processed image into an instance segmentation network for training to obtain the fish body part instance segmentation model, including fish head, fish body, fish tail, front fin and rear fin.

[0074] In the embodiment, the method for constructing the fish body part instance segmentation image dataset comprises: an underwater waterproof camera collects video information, and data labeling is performed on the images above and beside the fish body to obtain labeled image data; the labeled image data is divided into a training set, a verification set and a test set according to a preset ratio to obtain the fish body part instance segmentation image dataset. In the embodiment, a fish body part instance segmentation model is constructed through a yolopose detection network, video information is collected by using an edge device, the video is cut into pictures by using a PotPlayer software, fish body part data labeling is performed on the pictures by using a labelme software to obtain labeled image data, the labeling information of each image is matched from the original labeling file of the dataset according to the name, and the labeling information is converted into a YOLO labeling format file. Finally, the training set, the verification set and the test set are divided according to the ratio of 80%, 10% and 10% respectively, and finally the fish body part instance segmentation dataset is formed.

[0075] In the step S3, the fish body target tracking model is constructed based on a DeepSort algorithm, a detector obtains a bbox, generates detections, performs Kalman filtering prediction, and uses a Hungarian algorithm to match the predicted tracks and the detections in the current frame (cascade matching and IOU matching), and Kalman filtering is updated.

[0076] The step S4 specifically comprises:

[0077] When the fish is detected, the fish is judged for effectiveness. First, four key points m, n, a and b of the fish body are detected by a fish body key point detection model. The key point m is an eye part of the fish, the key point n is a middle part of a fish tail, the key point a is a rear side of a junction of a front fin of the fish body below the fish body, and the key point b is a front side of a junction of the fish body below a rear fin of the fish body.

[0078] A straight line is connected through the coordinates of the two key points m and n, so as to detect a fish body tilt angle. The fish body angle within 40 degrees is an effective fish, and the fish image with an angle greater than 40 degrees is removed. The fish image without detecting the key points a and b is removed.

[0079] When the angle is judged, the coordinates of the key point m are (x1, y1), and the coordinates of the key point n are (x2, y2). The tilt angle of the fish is calculated by the following formula: , .

[0080] The step S5 specifically comprises:

[0081] S5-1, a standard line in a normal state of a fish belly is drawn;

[0082] S5-2, Draw the actual state curve of the fish belly;

[0083] S5-3: Determine the difference between the actual state curve of the fish belly and the standard line, and finally output the recognition result.

[0084] The specific steps of S5-1 are as follows:

[0085] Under normal conditions, the degree of abdominal curvature is similar in fish of the same species. Collecting one hundred images of fish bellies in their normal state (where the abdomen is either bulging outwards or horizontal) is an example. Figure 3 As shown; when the fish's abdomen is in an abnormal state, it becomes concave, such as... Figure 4 As shown;

[0086] The standard line is determined by defining a parabola using three points, with standard keypoints a, b, and c. Keypoint c is the midpoint of the line connecting keypoints a and b along the fish's belly. Using a segmentation model of various parts of the fish body, the coordinates of keypoints a, b, and c are obtained. Specifically, keypoint a is the point with the largest x-coordinate at the junction of the pectoral fin and the body, and keypoint b is the point with the smallest x-coordinate at the junction of the posterior fin and the body. The x-coordinate of point c is obtained by averaging the x-coordinates of keypoints a and b. The largest y-coordinate of the fish body under the x-coordinate of point c is the y-coordinate of point c. The values ​​of the three standard keypoints a, b, and c for all fish are recorded. Figure 5 As shown;

[0087] Based on the parabola formula By substituting the x and y coordinates of the three standard key points a, b, and c into the parabola formula, the quadratic coefficient g can be obtained. When g > 0, the larger g is, the greater the curvature of the parabola; when g < 0, the smaller g is, the greater the curvature. Arranging fish from 0 degrees to 40 degrees in ascending order of degree, the value of g shows an increasing trend. When the fish is horizontal, g = 0.3; when the fish is tilted at 45 degrees, g = 0.7. Based on the scatter plot of g, the function formula of g can be fitted as: g = 0.01x + 0.3; where x is the tilt angle of the fish.

[0088] The specific steps of S5-2 are as follows:

[0089] The actual state curve of the fish is also a parabola, including actual key points a and b, and the midpoint d of the line connecting the two actual key points a and b along the direction of the fish's belly. The midpoint d is the actual key point, and the standard key points a and b coincide with the actual key points a and b. The x-coordinate of point d is obtained by averaging the x-coordinates of the actual key points a and b. The y-coordinate of point d is the maximum y-coordinate of the fish body under the x-coordinate of point d. The equation of the parabola is obtained by using the x-coordinates and y-coordinates of points a, b, and d. This is the actual state curve of the fish belly.

[0090] The specific steps S5-3 are as follows:

[0091] When the coefficient of the quadratic term g in the parabolic equation of the actual state curve of the fish is less than 0, it indicates that the curve is concave inward; when g is greater than 0, it indicates that the curve is convex outward.

[0092] When g<0 and the difference between the actual state curve of the fish and the standard line is greater than 20%, the output is "abnormal state 1 of the fish's abdomen", indicating that the fish is sick and has not eaten for a long time.

[0093] When g>0 and the difference between the actual state curve of the fish and the standard line is greater than 20%, the abnormal state 2 of the fish abdomen is output, indicating that the fish abdomen is swollen.

[0094] This invention uses an underwater waterproof camera to collect video information, constructs a key point detection model for the fish body and an instance segmentation model for each part of the fish body based on the video information, and uses a target tracking algorithm to track and capture images of the fish in the video information to obtain fish images. The validity of the fish is then determined, and the state of the fish's abdomen is identified based on the fish images.

[0095] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. A method for identifying anomalies in the belly state of fish based on machine vision, characterized in that, Includes the following steps: S1. Collect video information using an underwater waterproof camera, and construct a fish body key point detection model based on the video information; S2. Construct instance segmentation models for each part of the fish body and segment each part of the fish body; S3. Add a tracking model to track and capture images of the fish in the video information. S4. Determine the validity of the fish and obtain valid fish images; S5. Recognize the state of the fish's belly based on the fish image; The specific steps of step S4 are as follows: When a fish is detected, the validity of the fish is determined. First, the four key points m, n, a and b of the fish body are detected by the fish body key point detection model. Key point m is the eye of the fish, key point n is the middle part of the tail of the fish, key point a is the rear side of the junction between the front fin and the body of the fish, and key point b is the front side of the junction between the rear fin and the body of the fish. The fish's tilt angle is detected by connecting the coordinates of two key points, m and n, with a straight line. Fish with an angle within 40 degrees are considered valid, while images with angles greater than 40 degrees and images where key points a and b are not detected are discarded. In determining the angle: the key point detection model detects the coordinates of m as (x1, y1) and n as (x2, y2), and the fish's tilt angle θ is calculated using the following formula. ; The specific steps of step S5 are: S5-1, drawing the standard line for the fish belly under normal conditions; S5-2, Draw the actual state curve of the fish belly; S5-3, Determine the difference between the actual state curve of the fish belly and the standard line, and finally output the recognition effect.

2. The method for identifying abnormal fish belly states based on machine vision according to claim 1, characterized in that, Step S1 specifically includes: S1-1. Constructing a fish body keypoint detection image dataset: (1) Acquire fish images and annotate the fish images to obtain annotated image data; (2) The labeled image data is divided into a training set, a validation set and a test set according to a preset ratio to obtain the fish body key point detection image dataset; S1-2. Train the fish body key point detection model based on the aforementioned fish body key point detection dataset: (3) The fish body key point detection image dataset is preprocessed to obtain the preprocessed image; (4) Input the preprocessed image into the key point detection network for training to obtain the fish body key point detection model.

3. The method for identifying fish belly state anomalies based on machine vision according to claim 1 or 2, characterized in that, The specific steps of S2 are as follows: S2-1. Preprocess the image dataset of segmented images of various parts of the fish body to obtain the preprocessed images; S2-2. Input the preprocessed image into the instance segmentation network for training to obtain instance segmentation models for each part of the fish body.

4. The method for identifying abnormal fish belly states based on machine vision according to claim 1, characterized in that, The specific steps of S5-1 are as follows: Under normal conditions, the degree of abdominal curvature of fish of the same species is similar. Collect one hundred images of fish with normal abdominal curvature, at which time the fish's abdomen is either bulging outward or horizontal; when the fish's abdomen is abnormal, the fish's abdomen is concave inward. The standard line is determined by defining a parabola using three points. Standard key points a, b, and c are set. Among these, standard key point c is the midpoint of the line connecting two standard key points a and b along the fish's belly. The coordinates of standard key points a, b, and c are obtained by segmenting the fish body using instance models of various parts. Specifically, standard key point a is the point with the largest x-coordinate at the junction of the front fin and the body, and standard key point b is the point with the smallest x-coordinate at the junction of the rear fin and the body. The x-coordinate of point c is obtained by averaging the x-coordinates of standard key points a and b. The largest y-coordinate of the fish body under the x-coordinate of point c is the y-coordinate of point c. The values ​​of the three standard key points a, b, and c for all fish are recorded. Based on the parabola formula y=gx 2 By substituting the x and y coordinates of the three standard key points a, b, and c into the parabola formula, we can obtain the quadratic coefficient g. When g > 0, the larger g is, the greater the curvature of the parabola; when g < 0, the smaller g is, the greater the curvature. Arranging fish from 0 degrees to 40 degrees in ascending order of degree, the value of g shows an increasing trend. When the fish is horizontal, g = 0.3; when the fish is tilted at 45 degrees, g = 0.

7. Based on the scatter plot of g, we can fit the function formula of g: g = 0.01x + 0.3; where x is the tilt angle of the fish.

5. The method for identifying abnormal fish belly states based on machine vision according to claim 4, characterized in that, The specific steps of S5-2 are as follows: The actual state curve of the fish is also a parabola, including actual key points a and b, and the midpoint d of the line connecting the two actual key points a and b along the direction of the fish's belly. The midpoint d is the actual key point, and the standard key points a and b coincide with the actual key points a and b. The x-coordinate of point d is obtained by averaging the x-coordinates of the actual key points a and b. The y-coordinate of point d is the maximum y-coordinate of the fish body under the x-coordinate of point d. The equation of the parabola y=gx is obtained by using the x-coordinates and y-coordinates of points a, b, and d. 2 +hx+i represents the actual state curve of the fish belly.

6. The method for identifying abnormal fish belly states based on machine vision according to claim 1, characterized in that, The specific steps S5-3 are as follows: When the coefficient of the quadratic term g in the parabolic equation of the actual state curve of the fish is less than 0, it indicates that the curve is concave inward; when g is greater than 0, it indicates that the curve is convex outward. When g < 0 and the difference between the actual state curve of the fish and the standard line is greater than 20%, the output is "abnormal state 1 of the fish's abdomen", indicating that the fish is sick and has not eaten for a long time. When g > 0 and the difference between the actual state curve of the fish and the standard line is greater than 20%, the output "abnormal state 2 of the fish abdomen" indicates that the fish abdomen is swollen.

Images