A fish feeding decision-making method based on Transformer

Abstract

The present invention discloses a fish feeding decision method based on Transformer, including: obtaining a fish movement video, using a YOLOv5 model to detect each frame of the image to obtain a detection frame and position information of each fish; calculating the fish center of mass of each frame of the image and normalizing it: constructing a Delaunay triangulation; calculating a weight factor according to a two-dimensional Gaussian normal distribution; obtaining the fish center of mass coordinates; calculating the difference between the fish center of mass of two adjacent frames of the image to obtain a feature vector; processing the Transformer model and inputting a feature vector of a preset length into the model for training; inputting a feature vector of a preset length to be identified into the trained Transformer model to obtain a fish hunger value, comparing the fish hunger value with a preset threshold, and determining whether to feed. The method can avoid excess bait pollution or fish malnutrition, has a low amount of calculation, a high accuracy rate, can realize real-time monitoring of aquaculture, and has good generalization ability.

Description

Technical Field

[0001] The present invention belongs to the technical field of intelligent aquaculture, and in particular relates to a fish feeding decision-making method based on Transformer. Background Art

[0002] With the continuous development of fish farming, traditional feeding methods and equipment can no longer meet the growing demand. Fish feeding has certain regularities, and it is of great significance to identify the feeding status of fish in real time and use it to control feeding. However, the overall chaotic movement of fish is difficult to model, and it is often necessary to use a very complex model to describe the relationship between the movement trajectory of the fish and the hunger level of the fish. The calculation is large, and the execution efficiency of the image processing module is usually low, which cannot meet the real-time detection of the movement trajectory of the fish.

[0003] Existing feeding decision-making technologies for fish farming are usually based on traditional digital image processing and statistical machine learning methods, which are generally targeted at a certain type of specific farming environment, with poor robustness and generalization ability. For example, the traditional digital image processing method is used to binarize the entire picture, calculate the grayscale co-occurrence matrix to extract the characteristics of the fish school, and calculate the motion characteristics of the fish school and the degree of confusion of the fish school as the overall behavioral characteristics through the frame difference method, and then use the support vector machine to make a binary decision, that is, whether to feed. This method has poor robustness, and is based on the fish school as a unit. The feeding strategy is unreasonable, which is easy to cause waste or insufficient feeding. In addition, in order to achieve a higher accuracy rate, the traditional convolutional neural network often has a large model scale and too many parameters, which will result in slow calculation speed and large memory usage, and is not suitable for deployment in practical applications. Therefore, this application proposes a fish school feeding decision-making method based on Transformer. Summary of the invention

[0004] The purpose of the present invention is to propose a fish feeding decision-making method based on Transformer to address the above problems. The method combines the YOLOv5 model and the Transformer model to model the relationship between the fish movement trajectory and the hunger level, thereby realizing fish feeding decision-making, which can avoid the problem of excess bait polluting the environment or fish malnutrition. The method has low computational complexity and can be deployed on an embedded platform with limited computing resources to realize real-time monitoring of aquaculture feeding with high accuracy and good generalization ability.

[0005] To achieve the above object, the technical solution adopted by the present invention is:

[0006] The present invention proposes a fish feeding decision-making method based on Transformer, which comprises the following steps:

[0007] S1. Obtain a video of the movement of a school of fish, use the YOLOv5 model to detect each frame of the image to obtain a detection frame for each fish, and represent the position information of each fish as [x, y, w, h], where (x, y) is the center coordinate of the detection frame, i.e., the coordinate of the center of mass of the fish, w is the width of the detection frame, and h is the height of the detection frame;

[0008] S2. Calculate the fish school mass center of each frame image and perform normalization. The calculation of the fish school mass center is as follows:

[0009] S21, using the Delaunay triangulation method to construct a Delaunay triangulation network based on the centroid coordinates of all the fish in the image, the Delaunay triangulation network retains the centroid coordinates of n fish;

[0010] S22. Calculate the weight factor a of the centroid coordinates of each fish on the Delaunay triangulation network according to the two-dimensional Gaussian normal distribution i :

[0011]

[0012] Among them, μ1=0, μ2=0, σ1=1, σ2=1, ρ=0, (x i ,y i ) is the coordinate of the center of mass of the i-th (i=1,2,3…n) fish;

[0013] S23, the coordinates of the fish school's centroid (X, Y), satisfy the following formula:

[0014]

[0015] S3, calculating the speed and acceleration of the fish school movement according to the difference between the mass centers of the fish school in two adjacent frames, and generating a feature vector [p, v, a] as the behavior feature of the fish school in the latter frame of the two adjacent frames, wherein p is the mass center of the fish school in the latter frame of the two adjacent frames, v is the speed of the fish school movement, and a is the absolute value of the acceleration of the fish school movement, that is, the activity level of the fish school;

[0016] S4, processing the Transformer model and inputting the feature vector of the preset length into the Transformer model for training, wherein the processing of the Transformer model includes removing the embedding layer;

[0017] S5. Input the feature vector of the preset length to be identified into the trained Transformer model to obtain the hunger value of the fish school, and determine whether the hunger value of the fish school is greater than the preset threshold. If so, feed the fish according to the number of fish in the fish school, otherwise, do not feed the fish.

[0018] Preferably, in step S21, a Delaunay triangulation method is used to construct a Delaunay triangulation network based on the centroid coordinates of all fish in the image, as follows:

[0019] S211, treating the centroid coordinates of all fish as data points in the data point set, traversing all data points, constructing a super triangle that surrounds the most data points and putting it into a triangle linked list;

[0020] S212, randomly select one of the remaining data points except the super triangle and insert it into the triangle linked list, and determine whether the insertion point is inside the super triangle. If so, connect the insertion point with the three corner points of the super triangle to form a new convex polygon. Otherwise, connect the insertion point with the two corner points of the nearest edge of the super triangle to form a new convex polygon.

[0021] S213, determining whether the new convex polygon contains a quadrilateral, if so, optimizing by swapping the diagonals of the quadrilateral, otherwise, not optimizing;

[0022] S214, returning to step S212, until the triangle linked list contains n data points;

[0023] S215, randomly select one of the remaining data points outside the n data points and insert it into the triangle linked list, try to replace the inserted point with the n-3 data points outside the super triangle one by one to obtain a new triangle, and calculate the sum of the internal angles of the new triangle respectively. If the maximum sum of the internal angles is greater than the sum of the internal angles of the corresponding original triangle, the original triangle is replaced by the corresponding new triangle, otherwise, no replacement is performed;

[0024] S216, return to step S215, until all remaining data points are inserted and there is no edge crossing, so as to obtain the Delaunay triangulation.

[0025] Preferably, the normalization process is performed, and the formula is as follows:

[0026]

[0027] Among them, q* is the centroid of the fish school after normalization; q is the centroid of the fish school before normalization; q min is the minimum value of the fish school centroid sample; q max is the maximum value of the fish school centroid samples.

[0028] Compared with the prior art, the present invention has the following beneficial effects:

[0029] This method realizes fish feeding decision-making based on deep learning. The YOLOv5 model is used to quickly process images to obtain the position information of each fish, and the fish center of mass of each frame image is calculated. Then, the feature vector is obtained according to the difference between the fish center of mass of adjacent frames as the fish motion trajectory tensor. The feature vector of preset length is processed in combination with the Transformer model. The Transformer model does not require embedding layer encoding and additional convolution operations during image processing. It can fully model the relationship between the fish motion trajectory (such as the movement speed of the fish, the activity of the fish) and the hunger level, and obtain the fish hunger value. At the same time, feeding decisions are made based on the number of fish and the hunger level, avoiding the problem of excess bait polluting the environment or insufficient bait causing malnutrition of the fish. It can save breeding costs while ensuring the quality of fish breeding, and has low computational complexity. It can be deployed on embedded platforms with limited computing resources to realize real-time monitoring of breeding feeding with high accuracy and good generalization ability. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 The flowchart of the fish feeding decision method based on Transformer of the present invention;

[0031] Figure 2 It is the target detection diagram of the YOLOv5 model of the present invention;

[0032] Figure 3 This is a graph of the fish school centroid output result based on the Delauny triangulation network of the present invention. DETAILED DESCRIPTION

[0033] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0034] It should be noted that, unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein in the specification of this application are only for the purpose of describing specific embodiments and are not intended to limit this application.

[0035] like Figure 1-3 As shown in FIG. 1 , a fish feeding decision method based on Transformer includes the following steps:

[0036] S1. Obtain a video of the movement of a school of fish, use the YOLOv5 model to detect each frame of the image to obtain the detection frame of each fish, and represent the position information of each fish as [x, y, w, h], where (x, y) is the center coordinate of the detection frame, that is, the center of mass coordinate of the fish, w is the width of the detection frame, and h is the height of the detection frame.

[0037] The fish movement video is collected by the camera, such as extracting 100 frames of images within 10 seconds for processing (the FPS of the YOLOv5 model is 10). For each frame of the image, the YOLOv5 model is first used to perform target detection to obtain the detection frame of each fish. After processing the image, the YOLOv5 model will output the location information of each fish, that is, the location information of each fish is represented by a 4-dimensional vector [x, y, w, h]. (x, y) is the center coordinate of the detection frame, that is, the coordinate of the fish's center of mass, w is the width of the detection frame, and h is the height of the detection frame. Each fish has its corresponding detection frame. The detection results are as follows: Figure 2 As shown in the figure, the one-stage target detection is realized by the YOLOv5 model, which is faster than other models (such as the R-CNN model) and makes the detection more real-time.

[0038] S2. Calculate the fish school mass center of each frame image and perform normalization. The calculation of the fish school mass center is as follows:

[0039] S21, using the Delaunay triangulation method to construct a Delaunay triangulation network based on the centroid coordinates of all the fish in the image, the Delaunay triangulation network retains the centroid coordinates of n fish;

[0040] In one embodiment, in step S21, a Delaunay triangulation method is used to construct a Delaunay triangulation network based on the centroid coordinates of all fish in the image, as follows:

[0041] S211, treating the centroid coordinates of all fish as data points in the data point set, traversing all data points, constructing a super triangle that surrounds the most data points and putting it into a triangle linked list;

[0042] S212, randomly select one of the remaining data points except the super triangle and insert it into the triangle linked list, and determine whether the insertion point is inside the super triangle. If so, connect the insertion point with the three corner points of the super triangle to form a new convex polygon. Otherwise, connect the insertion point with the two corner points of the nearest edge of the super triangle to form a new convex polygon.

[0043] S213, determining whether the new convex polygon contains a quadrilateral, if so, optimizing by swapping the diagonals of the quadrilateral, otherwise, not optimizing;

[0044] S214, returning to step S212, until the triangle linked list contains n data points;

[0045] S215, randomly select one of the remaining data points outside the n data points and insert it into the triangle linked list, try to replace the inserted point with the n-3 data points outside the super triangle one by one to obtain a new triangle, and calculate the sum of the internal angles of the new triangle respectively. If the maximum sum of the internal angles is greater than the sum of the internal angles of the corresponding original triangle, the original triangle is replaced by the corresponding new triangle, otherwise, no replacement is performed;

[0046] S216, return to step S215, until all remaining data points are inserted and there is no edge crossing, so as to obtain the Delaunay triangulation.

[0047] like Figure 3 As shown, the Delaunay triangulation method is used to construct the Delaunay triangulation network based on the centroid coordinates of all fish in the image as follows:

[0048] (1) The centroid coordinates of all fish are used as the data point set to be triangulated. The data points in the data point set are traversed to find a super triangle containing the most data points and put it into the triangle linked list. The super triangle is the initial triangle.

[0049] (2) Select any data point from the remaining data points (data points other than the three corner points of the super triangle) and insert it into the triangle linked list. If the insertion point is inside the super triangle, connect the vertices of the super triangle (including the three corner points) to the insertion point one by one; if the insertion point is outside the super triangle, connect the vertex on the super triangle closest to the insertion point (including the two corner points) to the insertion point to form a new convex polygon.

[0050] (3) If the convex polygon formed after the insertion contains a quadrilateral, it is optimized by diagonalizing and exchanging the quadrilaterals. Exchanging diagonals is a common method in the prior art and will not be described in detail here. The minimum internal angle is maximized, and the optimized convex polygon is the new figure obtained by the insertion.

[0051] (4) Use the point-by-point insertion method to insert the remaining data points into the triangle linked list one by one until the triangle linked list includes n=8 data points.

[0052] (5) Use the replacement method to keep the triangle linked list always containing 8 data points, as follows:

[0053] Taking maximizing the internal angle as the optimization goal, when the triangle linked list includes 8 data points, the newly inserted data points will be traversed, trying to replace the five points except the three corner points of the super triangle, and calculating the maximum value of the sum of the internal angles of the five replacements. If the maximum sum of the internal angles is greater than the sum of the internal angles of the corresponding original triangle, the original triangle will be replaced with the corresponding new triangle, otherwise, no replacement will be made.

[0054] (6) Execute the above step (5) repeatedly, and ensure that there is no edge crossing, traverse all data points, and always keep 8 data points in the Delaunay triangulation. The selected 8 points are the optimal solution for maximizing the internal angle of the Delaunay triangulation. Calculating the centroid of the fish school through the data points in the Delaunay triangulation can greatly reduce the amount of calculation.

[0055] Figure 3 In the figure, the left and right figures are the Delaunay triangulation results of the two frames of images. The diamond box represents a single fish, the center of the diamond box is the centroid of the fish, and the edges constitute the Delaunay triangulation. The point * represents the centroid of the fish school obtained by the method of this article (Delaunay triangulation), the point + represents the centroid of the fish school obtained by the weighted average method of the centroid coordinates of all fish, and the point o represents the centroid of the fish school obtained by the mathematical average method of the centroid coordinates of all fish. This application uses the construction of Delaunay triangulation to obtain the centroid of the fish school with smaller calculation amount and accuracy.

[0056] S22. Calculate the weight factor a of the centroid coordinates of each fish on the Delaunay triangulation network according to the two-dimensional Gaussian normal distribution i :

[0057]

[0058] Among them, μ1=0, μ2=0, σ1=1, σ2=1, ρ=0, (x i ,y i ) is the coordinate of the center of mass of the i-th (i=1,2,3…n) fish; x i ,y i are independent and identically distributed.

[0059] S23, the coordinates of the fish school's centroid (X, Y), satisfy the following formula:

[0060]

[0061] In one embodiment, the normalization process is performed, and the formula is as follows:

[0062]

[0063] Among them, q* is the centroid of the fish school after normalization; q is the centroid of the fish school before normalization; q min is the minimum value of the fish school centroid sample; q max is the maximum value of the fish school centroid samples.

[0064] In order to prevent the data from being too different, the fish school centroid data is mapped to the [-1,0] interval and normalized.

[0065] S3. Calculate the speed and acceleration of the fish school movement according to the difference between the fish school mass centers of two adjacent frames, and generate a feature vector [p, v, a] as the behavior feature of the fish school in the latter frame of the two adjacent frames, where p is the fish school mass center of the latter frame of the two adjacent frames, v is the speed of the fish school movement, and a is the absolute value of the acceleration of the fish school movement, that is, the activity level of the fish school.

[0066] Among them, after calculating the center of mass of the fish school in each image and normalizing it, the speed of the fish school can be calculated by calculating the center of mass difference between the previous and next frames. The acceleration of the fish school can be calculated by the speed of the fish school. The absolute value of the acceleration is the activity level of the fish school. The larger a is, the more active the fish school is. The feature vector [p, v, a] is formed to describe the behavioral characteristics of the fish school.

[0067] S4. Processing the Transformer model and inputting a feature vector of a preset length into the Transformer model for training. Processing the Transformer model includes removing an embedding layer.

[0068] Among them, the Transformer model is suitable for vector operations with a relationship between the front and the back. The feature vector of the fish movement is suitable for this model, and there is no need for encoding of the embedding layer and additional convolution operations during image processing. It can give full play to the advantages of Transformer as a recurrent neural network, with fast speed, high accuracy, fast convergence, and low computational complexity. It is an existing technology to input the feature vector of a preset length into the Transformer model for training, and to use the training set and the validation set to train or validate the model to obtain a model that meets the requirements, which will not be repeated here. The feature vector of the preset length can be determined according to actual needs. For example, if 100 frames of images are processed each time, 100 frames are 100-dimensional feature vectors. The feature vector corresponding to the first frame of the image, the center of mass of the fish school is known, and v and a are both 0 by default.

[0069] S5. Input the feature vector of the preset length to be identified into the trained Transformer model to obtain the hunger value of the fish school, and determine whether the hunger value of the fish school is greater than the preset threshold. If so, feed the fish according to the number of fish in the fish school, otherwise, do not feed the fish.

[0070] This method is based on deep learning methods for modeling and image processing, makes feeding decisions based on single fish, and determines the hunger level of the fish school by analyzing the overall behavioral characteristics of the fish school over a period of time (such as 100 frames of images within 10 seconds) and the relationship between each frame of the image. When the hunger value of the fish school is greater than the preset threshold, feeding is performed based on the number of fish in the fish school (which can be counted by the number of detection boxes in the YOLOv5 model), thus avoiding waste. The introduction of the preset threshold is also more flexible than the dichotomy in the prior art.

[0071] The YOLOv5 model is used to quickly process images to obtain the position information of each fish, and the fish school centroid of each frame image is calculated. Then, the feature vector is obtained according to the difference between the fish school centroids of adjacent frames as the fish school motion trajectory tensor. The feature vector of preset length is processed in combination with the Transformer model. The Transformer model does not require embedding layer encoding and additional convolution operations during image processing. It can fully model the relationship between the fish school motion trajectory (such as the movement speed of the fish school, the activity level of the fish school) and the hunger level, and obtain the fish school hunger value. At the same time, feeding decisions are made based on the number of fish and the hunger level to avoid the problem of excess bait polluting the environment or insufficient bait causing malnutrition of the fish school. It can save breeding costs while ensuring the quality of fish breeding, and has low computational complexity. It can be deployed on embedded platforms with limited computing resources to realize real-time monitoring of breeding feeding with high accuracy and good generalization ability.

[0072] The technical features of the above-described embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above-described embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0073] The above-described embodiments only express the more specific and detailed embodiments described in this application, but they cannot be understood as limiting the scope of the patent application. It should be pointed out that for ordinary technicians in this field, several modifications and improvements can be made without departing from the concept of this application, which all belong to the protection scope of this application. Therefore, the protection scope of the patent application shall be based on the attached claims.

Claims

1. A fish feeding decision-making method based on Transformer, characterized by: The Transformer-based fish feeding decision-making method includes the following steps: S1. Obtain a video of a school of fish, use the YOLOv5 model to detect each frame of the image to obtain a detection frame for each fish, and express the position information of each fish as [x, y, w, h], where (x, y) is the center coordinate of the detection frame, i.e., the center of mass coordinate of the fish, w is the width of the detection frame, and h is the height of the detection frame; S2. Calculate the fish school mass center of each frame image and perform normalization processing. The calculation of the fish school mass center is as follows: S21, using Delaunay triangulation method to construct a Delaunay triangulation network based on the centroid coordinates of all fish in the image, wherein the Delaunay triangulation network retains the centroid coordinates of n fish; S22, calculate the weight factor a of the centroid coordinates of each fish on the Delaunay triangulation according to the two-dimensional Gaussian normal distribution i : Among them, μ1=0, μ2=0, σ1=1, σ2=1, ρ=0, (x i ,y i ) is the coordinate of the center of mass of the i-th fish, i = 1, 2, 3…n; S23. The coordinates (X, Y) of the centroid of the fish school satisfy the following formula: S3. Calculate the fish school's movement speed and fish school's movement acceleration based on the difference between the fish school's center of mass in two adjacent frames, and generate a feature vector [p, v, a] as the fish school behavior feature of the later frame image between the two adjacent frames, where p is the fish school's center of mass in the later frame image between the two adjacent frames, v is the fish school's movement speed, and a is the absolute value of the fish school's movement acceleration, i.e., the fish school's activity level; S4. Processing the Transformer model and inputting a feature vector of a preset length into the Transformer model for training, wherein the processing of the Transformer model includes removing an embedding layer; S5. Input the feature vector of the preset length to be identified into the trained Transformer model to obtain the hunger value of the fish school, and determine whether the hunger value of the fish school is greater than the preset threshold. If so, feed the fish according to the number of fish in the fish school, otherwise, do not feed the fish.

2. The Transformer-based fish feeding decision-making method according to claim 1, characterized in that: In step S21, the Delaunay triangulation method is used to construct a Delaunay triangulation network based on the centroid coordinates of all fish in the image, as follows: S211, treating the centroid coordinates of all fish as data points in the data point set, traversing all data points, constructing a super triangle that surrounds the most data points and putting it into a triangle linked list; S212: Randomly select one of the remaining data points except the super triangle and insert it into the triangle linked list. Determine whether the insertion point is inside the super triangle. If so, connect the insertion point to the three corner points of the super triangle to form a new convex polygon. Otherwise, connect the insertion point to the two corner points on the nearest edge of the super triangle to form a new convex polygon. S213. Determine whether the new convex polygon contains a quadrilateral. If so, optimize by swapping the diagonals of the quadrilateral. Otherwise, do not optimize. S214, returning to step S212, until the triangle linked list contains n data points; S215: randomly select one of the remaining data points outside the n data points and insert it into the triangle linked list. Try to replace the inserted point with the n-3 data points outside the super triangle one by one to obtain a new triangle, and calculate the interior angle sum of each new triangle. If the maximum interior angle sum is greater than the interior angle sum of the corresponding original triangle, replace the original triangle with the corresponding new triangle; otherwise, do not replace it. S216 , returning to step S215 , and continuing until all remaining data points are inserted and no edge lines cross, thereby obtaining the Delaunay triangulation.

3. The Transformer-based fish feeding decision-making method according to claim 1, characterized in that: The normalization process is as follows: Among them, q* is the centroid of the fish school after normalization; q is the centroid of the fish school before normalization; q min is the minimum value of the fish school centroid sample; q max is the maximum value of the fish school centroid samples.

Images