Fish behavior analysis method and system based on dynamic convolutional graph neural network

Abstract

A fish behavior analysis method and system based on a dynamic convolutional graph neural network includes the following steps: obtaining fish behavior videos and preprocessing them to obtain fish trajectory information; extracting fish behavior features based on the fish trajectory information; calculating fish behavior characteristic indicators based on the extracted fish behavior features, including swimming speed, angular velocity, acceleration, fish activity, and dispersion; constructing edges based on the fish behavior characteristic indicator set; and constructing a dynamic graph convolutional network for fish behavior analysis, including memory units and a multi-classifier based on hypergraph convolution. The multi-classifier based on hypergraph convolution consists of three parts: constructing a hypergraph, constructing hypergraph convolution, forming a new feature code, and finally performing classification prediction on the new feature code. The present invention can fully utilize the information contained in fish behavior video data to extract fish behavior features, and improve the performance and accuracy of the model through clustering-based memory modules and hypergraph convolution.

Description

Technical Field

[0001] The present invention belongs to the technical field of fish behavior analysis based on computer vision methods, relates to the fish farming industry, and the dynamic convolutional graph neural network technology in deep learning technology, and in particular to a fish behavior analysis method and system using a dynamic convolutional graph neural network. Background Art

[0002] Studying fish behavior is crucial for improving aquaculture efficiency and quality. This is because the fish farming industry faces several key challenges, such as sudden incidents leading to hypoxia-induced mortality and stress reactions caused by persistently deteriorating water quality. These factors can lead to physiological sub-health, illness, and mortality in fish, reducing aquaculture efficiency. Sub-health in fish populations can be reflected in various aspects of their metabolism, food intake, and reproductive cycle. For example, when fish are in poor health, their swimming speed decreases significantly, and their frequency of leaping out of the water increases. When dissolved oxygen in the water is too low, their swimming speed and acceleration tend to decrease, and their overall distribution becomes more dispersed. Therefore, analyzing fish farming conditions through continuous monitoring and quantification of fish behavior is crucial for determining whether fish populations are experiencing sub-health. Detecting and quantifying fish population status can also reflect their feeding status, which is of great significance to fish production and aquaculture.

[0003] There are many methods for monitoring and analyzing fish behavior, such as visual inspection. However, visual inspection cannot accurately record and quantify fish behavior and is closely related to the experience of the observer. With the development of computer technology, computer vision-based methods have gradually emerged to study fish behavior. Therefore, this application aims to obtain videos or images of fish life and analyze the behavioral characteristics of fish through computer vision-based methods, ultimately realizing automatic fish behavior detection and analysis, and discovering abnormal states of fish or analyzing fish feeding conditions. Summary of the Invention

[0004] The present invention aims to solve the above-mentioned technical problems existing in the prior art and provide a fish behavior analysis method and system based on a dynamic convolutional graph neural network. The method analyzes the behavior of fish schools through computer vision and is highly accurate and effective in real fish farming scenarios.

[0005] In order to achieve the above objectives, the technical solution of this application is as follows: a fish behavior analysis method based on a dynamic convolutional graph neural network, comprising the following steps:

[0006] Step 1: Acquire and preprocess fish behavior videos. First, process the fish behavior videos captured by the camera. Use Yolo-v5 and DeepSORT to track and identify fish trajectories, generating a sequence of positional information about individual fish. Yolo-v5 is a target detection algorithm, and DeepSORT is a target tracking algorithm. Using these two algorithms, we obtain fish trajectory information and record the positions of all fish in each frame in a file.

[0007] Step 2: Extract fish behavioral features based on the fish school trajectory information. These fish behavioral features primarily include the center of mass coordinates of each individual, a bounding box, and a fish ID. The center of mass coordinates of each individual are an abstract point, which can be considered a fish within the school. The bounding box is an imaginary rectangle that serves as a reference point for the fish and represents the individual's swimming posture. It is generated by object detection using Yolo-V5.

[0008] Step 3: Calculate the fish behavior characteristic index based on the extracted fish behavior characteristics. The fish behavior characteristic index includes swimming speed, angular velocity, acceleration, fish school activity, dispersion, etc. The dispersion is an indicator that can reflect the interaction between individual fish. The final obtained fish behavior characteristic index set structure is (S id ,f,i,v f ,θ f ,ACT i , ), where S id Indicates the video number in the dataset.

[0009] Step 4: Construct edges based on the set of fish behavior characteristic indicators. The edges of the graph are constructed based on the neighbor distance and Delaunay triangle. The Delaunay triangle is a triangle formed by individuals on the plane. Its numerical value can represent the distance between individuals, and the average triangle perimeter expresses the degree of dispersion of the fish school. The numerical value of the dispersion degree can be expressed as P i The perimeter of each triangle is . Therefore, the relationship between each fish in the school can be represented as a two-dimensional array.

[0010] Step five, construct a dynamic graph convolution network for fish behavior analysis. The dynamic graph convolution network for fish behavior analysis is the first method to use a memory network structure to enhance the performance of graph neural networks and apply it to solve real-world problems. The network includes two units, a memory unit and a multi-classifier based on hypergraph convolution. In the memory network structure, we innovatively proposed a construction method for keys, values, and query vectors that are suitable for our fish school status classification task, as well as a set of suitable update mechanisms, including a self-learning setting method for thresholds. The multi-classifier based on hypergraph convolution consists of three parts: constructing a hypergraph, constructing a hypergraph convolution, forming a new feature code, and finally performing classification prediction on the new feature code. Among them, the hypergraph convolution includes vertex convolution and hyperedge convolution. This application combines the intermediate features of the memory unit with the construction method of the hyperedge, and makes refined modifications to the features of the task. The vertex features in the hypergraph are formed into hypergraph features after vertex convolution, and the hypergraph and hypergraph features are connected through the memory unit.

[0011] Specifically, the contents stored in the memory unit are keys, values ​​and query vectors. The key vector is a feature vector representing the center of the cluster, which will be used to calculate the similarity with the query vector. We use the K-means algorithm for clustering, and the clustered features are the fish behavior characteristics obtained in step three. The query vector is extracted from the data during the training process. The value vector represents the information value to be returned stored in the memory unit, which describes the feature vector of the current specific classification. In this application, the key vector is equal to the value vector, and the query vector is normalized. If a query is required, the network will calculate the cosine similarity between the query vector and all the currently stored key vectors, and select a matching feature vector with the greatest similarity and return it to the model. At the same time, this returned vector will be connected to the training process as an auxiliary feature of the sample.

[0012] Furthermore, the memory unit is updated when a new query vector appears.

[0013] Furthermore, a hypergraph is constructed. Sample X is passed through a memory network to obtain the feature vector X of all samples, thereby constructing a hypergraph. In a hypergraph network, a vertex u is defined to represent a sample, and a hyperedge e is defined to represent a set of samples. The number of samples contained in a hyperedge is flexible. Specifically, Con(e) represents the set of all nodes contained in hyperedge e, and Adj(u) represents all hyperedges containing node u.

[0014]

[0015] Specifically, k e represents the number of vertices in the hyperedge e, k urepresents the hyperedge containing vertex u. We define u as the central vertex of the set Adj(u).

[0016] Furthermore, the central features of the memory cells are combined with KNN and K-means to construct a hypergraph, making full use of local and global information. Specifically, the k-1 nearest neighbors of each node u are calculated, and matching feature vectors are extracted from the memory cells based on the features of each node. These neighboring vertices and the vertex itself form the hyperedge Adj(u). In addition, the K-means algorithm is applied to the entire feature graph based on the Euclidean distance. For each vertex, the nearest S-1 clusters will be assigned to its adjacent hyperedge set, where S is the number of specified clusters. In particular, we initialize the initial feature set to the input set features. We perform the above operations for each layer of features during training.

[0017] Furthermore, a hypergraph convolution is constructed. Hypergraph convolution is divided into hyperedge convolution and vertex convolution. The vertex convolution aggregates features to hyperedges, and the hyperedge convolution aggregates features to the central vertex of the hyperedge set.

[0018] Furthermore, the formed central vertex features are used to form new feature codes, and the feature codes are classified by a classifier, and ultimately three categories are obtained: normal state, eating state, and stimulated state.

[0019] The present invention also provides a fish behavior analysis system based on a dynamic convolutional graph neural network. Specifically, the system comprises five modules: an image acquisition and preprocessing module, a fish behavior characteristic index acquisition module, a fish behavior characteristic index calculation module, a fish behavior analysis graph construction module, and a fish behavior analysis module.

[0020] Image acquisition and preprocessing module: acquires fish behavior videos and performs preprocessing; uses Yolo-v5 and DeepSORT algorithms to obtain fish trajectory information, and records the positions of all fish in each frame in a file;

[0021] Fish behavior characteristic index acquisition module: extracts individual centroid coordinates, bounding boxes, and fish number of each fish based on fish track information;

[0022] Fish behavior characteristic index calculation module: Calculates fish behavior characteristic indexes including swimming speed, angular velocity, acceleration, fish activity, dispersion degree, etc. according to the extracted fish behavior characteristics; the final obtained fish behavior characteristic index set structure is (S id ,f,i,v f ,θ f ,ACT i , ), where S idIndicates the number of the video in the dataset;

[0023] Fish behavior analysis graph construction module: construct edges based on a set of fish behavior characteristic indicators; construct graph edges based on neighbor distances and Delaunay triangles;

[0024] Fish behavior analysis module: Construct a dynamic graph convolutional network for fish behavior analysis; the dynamic graph convolutional network for fish behavior analysis includes two units, a memory unit and a multi-classifier based on hypergraph convolution; in the memory network structure, the key, value and query vector construction method suitable for the fish school status classification task is used, as well as the update mechanism, including the self-learning setting method of the threshold; the multi-classifier based on hypergraph convolution consists of three parts: constructing a hypergraph, constructing a hypergraph convolution, forming a new feature code, and finally performing classification prediction on the new feature code.

[0025] The advantages of the present invention are: the fish behavior analysis method constructed in this application is a method based on deep learning and hypergraph networks, which can effectively utilize multi-source data of fish behavior and extract multi-angle and multi-level features of fish behavior. These features can comprehensively reflect the state of the fish, such as position, direction, speed, posture, etc., to assist in subsequent analysis. The clustering-based memory unit proposed in the present invention is an innovative mechanism that can automatically cluster features of different types or stages during the training process and store the clustering results in the memory unit. In this way, the model has the ability to memorize the features of the training process, and can retrieve the most relevant or most useful features from the memory unit according to the input data during testing or application, thereby improving the robustness and generalization ability of the model. In addition, hypergraph convolution is a convolution operation based on a hypergraph structure, which can perform feature extraction and fusion in high-dimensional space. Hypergraph convolution can process data of different types and scales, and can extract features at different levels of abstraction. It can automatically select appropriate weights to assist model classification without manual setting or adjustment of parameters. The fish behavior analysis method developed in this application is an advanced and effective method that can fully utilize the information contained in fish behavior video data to extract multi-angle and multi-level features of fish behavior. It also improves the performance and accuracy of the model through clustering-based memory units and hypergraph convolution. The fish behavior analysis method developed in this application has important value and application prospects for fishery production and scientific research. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 An overall diagram of the fish behavior analysis method based on dynamic convolutional graph neural networks;

[0027] Figure 2 Schematic diagram of fish trajectory tracking method;

[0028] Figure 3 Schematic diagram of a dynamic graph convolutional network for fish behavior analysis;

[0029] Figure 4 Schematic diagram of the memory unit update mechanism;

[0030] Figure 5 Schematic diagram of hypergraph convolution;

[0031] Figure 6 Schematic diagram of the fish behavior analysis system based on dynamic convolutional graph neural network. DETAILED DESCRIPTION

[0032] The technical solution of the present invention is further described below with reference to the accompanying drawings.

[0033] Example 1

[0034] Reference Figure 1-Figure 5 , a method for constructing an industry knowledge graph based on a deep learning model containing memory blocks, including the following steps:

[0035] Step 1: Obtain and preprocess fish behavior videos. First, process the fish behavior videos captured by the camera. Use Yolo-v5 and DeepSORT to track and identify fish trajectories, obtaining a sequence of position change information for individual fish. Yolo-v5 is a target detection algorithm, and DeepSORT is a target tracking algorithm.

[0036] Specifically, the fish behavior videos collected by the camera are collected within several growth cycles of the fish (1 week, 3 weeks, 5 weeks, 7 weeks...), and three characteristic videos of fish behavior are selected from all the video data. These three fish behaviors are normal state, eating state and stimulated state. There are 60 videos in the normal state, 50 videos in the eating state, and 60 videos in the stimulated state. The duration of each video is 8 seconds. Afterwards, we pre-process the data using Mosaic data enhancement. The Mosaic data enhancement mainly splices the training data through random scaling, random cropping, and random arrangement to improve the detection effect of small targets. In addition, in order to detect the activity trajectory of fish schools, this application uses two models Yolo-v5 and DeepSORT. First, Yolo-v5 is used for target detection, the position of a single fish is detected, and a bounding box is added to it. Then, the DeepSORT model is used to track the trajectory of the fish. The entire workflow is as follows. Figure 2 As shown in the figure, the detection algorithm first obtains the bounding box, then generates the number and position information, predicts it through the Kalman filter, and uses the Hungarian algorithm to match the predicted trajectory with the position in the current frame (cascade matching and IOU matching). Finally, the Kalman filter is updated. Finally, the two algorithms are used to obtain the trajectory information of the fish school, and the position of all fish in each frame is recorded in a file.

[0037] Step 2: Extract fish behavioral features based on the fish school trajectory information. These fish behavioral features primarily include the center of mass coordinates of each individual, a bounding box, and a fish ID. The center of mass coordinates of each individual are an abstract point, which can be considered a fish within the school. The bounding box is an imaginary rectangle that serves as a reference point for the fish and represents the individual's swimming posture. It is generated by object detection using Yolo-V5.

[0038] Step 3: Calculate fish behavior characteristic indices based on the extracted fish behavior characteristics. The fish behavior characteristic indices include swimming speed, angular velocity, acceleration, fish school activity, dispersion, etc. The dispersion is an indicator that can reflect the interaction between individual fish.

[0039] For the calculation of the swimming speed of fish school, use and To represent the center of each fish's bounding box, represents the center of mass of each fish, where f represents the fish number and i represents the fish in the i-th frame. Therefore, the swimming speed is calculated as:

[0040]

[0041] Here, M represents the calculation of the swimming speed of a single fish based on the M frames before and after each frame. In order to make the speed calculation more accurate, M=3 is selected in this application. In addition, the angle of the fish school is calculated as follows:

[0042]

[0043] In addition, use Represents the size of each fish’s bounding box, and calculates the mean and variance of w and h for a single fish in three frames of video.

[0044]

[0045]

[0046]

[0047]

[0048] Regarding the calculation of fish school activity, this application completes the corrected activity calculation based on the consideration of the impact of overlapping fish schools on activity, making the activity results more accurate and reliable. The activity calculation method is as follows:

[0049]

[0050] Specifically, the It is the bbox of two adjacent frames i+m-1and bbox i+m Minimum area, the IoU i It is the bbox of two adjacent frames i+m-1 and bbox i+m The overlapping area and the bbox of two adjacent frames i+m-1 and bbox i+m The ratio of the area and the area, U i+m is the overlapping area of ​​two adjacent frames. However, the activity calculation depends on the image pixels, so the phenomenon of fish school overlap will cause errors in the activity calculation. Therefore, this application takes into account the situation of fish school overlap and calculates a correction coefficient. Finally, this correction coefficient is multiplied by the original activity to obtain the final activity.

[0051]

[0052] Wherein, N is the total number of actual fish, i and N i+1 is the number of visible fish in the two frames.

[0053] Finally, the structure of the fish behavior characteristic index set obtained according to the steps is (S id ,f,i,v f ,θ f ,ACT i , ), where S id Indicates the video number in the dataset.

[0054] Step 4: Construct edges based on the set of fish behavior characteristic indicators. The edges of the graph are constructed based on the neighbor distance and Delaunay triangle. The Delaunay triangle is a triangle formed by individuals on the plane. Its numerical value can represent the distance between individuals, and the average triangle perimeter expresses the degree of dispersion of the fish school. The numerical value of the dispersion degree can be expressed as P i is the perimeter of each triangle. Therefore, the relationship between each fish in the school can be represented as a two-dimensional array.

[0055] Step 5: Construct a dynamic graph convolutional network for fish behavior analysis. The dynamic graph convolutional network for fish behavior analysis is the first method to use a memory network structure to enhance the performance of graph neural networks and apply it to solve real-world problems. Figure 3As shown, the network includes two units, a memory unit and a multi-classifier based on hypergraph convolution. In the memory network structure, we innovatively proposed a construction method of keys, values ​​and query vectors suitable for our fish state classification task, as well as a set of suitable update mechanisms, including the self-learning setting method of thresholds. The multi-classifier based on hypergraph convolution consists of three parts: constructing a hypergraph, constructing a hypergraph convolution, forming a new feature encoding, and finally classifying and predicting the new feature encoding. Among them, the hypergraph convolution includes vertex convolution and hyperedge convolution. This application combines the intermediate features of the memory unit with the construction method of the hyperedge, and makes refined modifications to the features of the task. The vertex features in the hypergraph are formed into hypergraph features after vertex convolution, and the hypergraph and hypergraph features are connected through the memory unit.

[0056] Specifically, the contents stored in the memory unit are keys, values ​​and query vectors. The key vector is a feature vector representing the center of the cluster, which will be used to calculate the similarity with the query vector. We use the K-means algorithm for clustering, and the clustered features are the fish behavior characteristics obtained in step three. The query vector is extracted from the data during the training process. The value vector represents the information value to be returned stored in the memory unit, which describes the feature vector of the current specific classification. In this application, the key vector is equal to the value vector, and the query vector is normalized. If a query is required, the network will calculate the cosine similarity between the query vector and all the currently stored key vectors, and select a matching feature vector with the greatest similarity and return it to the model. At the same time, this returned vector will be connected to the training process as an auxiliary feature of the sample.

[0057] Furthermore, the memory unit is updated when a new query vector appears. Figure 4 As shown in Figure 2, the distance between the newly created value vector and key vector and the key feature vector determines whether the existing value vector and key vector should be updated. When the distance is less than a set threshold, the key and value vectors are updated and normalized. When the distance is greater than the set threshold, it indicates that the currently stored information cannot match the current query information. In this case, a new key-value pair is created based on the query vector value and stored in the memory unit.

[0058] Case 1:

[0059] renew:

[0060] renew:

[0061] Case 2:

[0062] Specifically, the value represents the fish behavior feature accompanying the query vector query, K near and represents the storage key vector of the memory unit with the maximum cosine similarity calculated according to the algorithm and its corresponding value vector, i represents the corresponding superscript of the fish behavior feature vector, and the hyperparameter δ is the threshold. l ] and V[n l ] is the key and value vector of the newly added memory unit, n l The table adds a new key-value pair tag. k and θ v are the update steps of key-value pairs respectively.

[0063] Furthermore, a hypergraph is constructed. Sample X is passed through a memory network to obtain the feature vector X of all samples, thereby constructing a hypergraph. In a hypergraph network, a vertex u is defined to represent a sample, and a hyperedge e is defined to represent a set of samples. The number of samples contained in a hyperedge is flexible. Specifically, Con(e) represents the set of all nodes contained in hyperedge e, and Adj(u) represents all hyperedges containing node u.

[0064]

[0065] Specifically, k e represents the number of vertices in the hyperedge e, k u represents the hyperedge containing vertex u. We define u as the central vertex of the set Adj(u).

[0066] Furthermore, the central features of the memory unit are combined with KNN and K-means to construct a hypergraph, making full use of local and global information. Specifically, the k-1 nearest neighbors of each node u are calculated, and matching feature vectors are extracted from the memory unit according to the features of each node. These neighbor vertices and the vertex itself form a hyperedge Adj(u). In addition, the K-means algorithm is applied to the entire feature graph based on the Euclidean distance. For each vertex, the nearest S-1 cluster group will be assigned to its adjacent hyperedge set. In particular, the initial feature set is the input set feature. This application performs the above operations for each layer of features during training.

[0067] Furthermore, a hypergraph convolution is constructed. Figure 5 As shown in Figure 2, hypergraph convolution is divided into hyperedge convolution and vertex convolution. The vertex convolution aggregates features to the hyperedge, and the hyperedge convolution aggregates features to the central vertex of the hyperedge set.

[0068] The present invention utilizes a multi-layer perceptron to learn a transformation matrix M from vertex features and assign weights, and then utilizes one-dimensional convolution to further extract features and perform vertex convolution.

[0069] M=MLP v (F u ) (15)

[0070]

[0071] Specifically, the F u is the input vertex feature set, MLP v represents a multilayer perceptron, Represents multiplication, Conv1d represents one-dimensional convolution, F v is the vertex convolution output.

[0072] Furthermore, hyperedge convolution aggregates hyperedge features into the central vertex features of the hyperedge set.

[0073] w=softmax(F v ·W+b) (16)

[0074]

[0075] Specifically, W and b represent the weights and biases learned from convolution, and F h is the output of the hyperedge convolution.

[0076] Furthermore, the formed central vertex features are used to form new feature codes, and the feature codes are classified by a classifier, and ultimately three categories are obtained: normal state, eating state, and stimulated state.

[0077] Example 2

[0078] This embodiment provides a fish behavior analysis system based on a dynamic convolutional graph neural network. Specifically, the system includes five modules: an image acquisition and preprocessing module, a fish behavior characteristic index acquisition module, a fish behavior characteristic index calculation module, a fish behavior analysis graph construction module, and a fish behavior analysis module.

[0079] Image acquisition and preprocessing module: acquires fish behavior videos and performs preprocessing; uses Yolo-v5 and Deepsort algorithms to obtain fish track information, and records the positions of all fish in each frame in a file;

[0080] Fish behavior characteristic index acquisition module: extracts individual centroid coordinates, bounding boxes, and fish number of each fish based on fish track information;

[0081] Fish behavior characteristic index calculation module: Calculates fish behavior characteristic indexes including swimming speed, angular velocity, acceleration, fish activity, dispersion degree, etc. according to the extracted fish behavior characteristics; the final obtained fish behavior characteristic index set structure is (Sid ,f,i,v f ,θ f ,ACT i , );

[0082] Fish behavior analysis graph construction module: construct edges based on a set of fish behavior characteristic indicators; construct graph edges based on neighbor distances and Delaunay triangles;

[0083] Fish behavior analysis module: Construct a dynamic graph convolutional network for fish behavior analysis; the dynamic graph convolutional network for fish behavior analysis includes two units, a memory unit and a multi-classifier based on hypergraph convolution; in the memory network structure, the key, value and query vector construction method suitable for the fish school status classification task is used, as well as the update mechanism, including the self-learning setting method of the threshold; the multi-classifier based on hypergraph convolution consists of three parts: constructing a hypergraph, constructing a hypergraph convolution, forming a new feature code, and finally performing classification prediction on the new feature code.

[0084] Regarding the apparatus in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0085] The above-described embodiments merely represent several implementation methods of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present invention. It should be noted that a person skilled in the art could make various modifications and improvements without departing from the spirit of the present invention, all of which fall within the scope of protection of the present application. Therefore, the scope of protection of the present invention should be determined by the appended claims.

Claims

1. A fish behavior analysis method based on a dynamic convolutional graph neural network, characterized by: The steps include: Step 1: Obtain fish behavior videos and preprocess them; use Yolo-v5 and Deepsort algorithms to obtain fish track information and record the positions of all fish in each frame in a file; Step 2: Extract the fish behavior characteristics including individual centroid coordinates, bounding box, and number of each fish based on the fish school trajectory information; Step 3: Calculate the fish behavior characteristic indicators including swimming speed, angular velocity, acceleration, fish activity, and dispersion based on the extracted fish behavior characteristics; the final structure of the fish behavior characteristic indicator set is Among them S id represents the video number in the dataset; f represents the fish number; i represents the fish in the i-th frame; v f Indicates the swimming speed of the fish; θ f Indicates the angle of the fish school; ACT i Indicates the final activity level of the fish school after correction; represents the mean value of the width w of the bounding box of a single fish in M ​​frames of video; represents the variance of the width w of a single fish bounding box in M ​​frames of video; The average height h of a single fish bounding box in M ​​frames of video; represents the variance of the height h of a single fish bounding box in M ​​frames of video; Step 4: Construct edges based on the fish behavior characteristic indicator set; construct graph edges based on neighbor distance and Delaunay triangle; Step 5: Construct a dynamic graph convolutional network for fish behavior analysis; the dynamic graph convolutional network for fish behavior analysis includes two units, a memory unit and a multi-classifier based on hypergraph convolution; in the memory network structure, the key, value and query vector construction method suitable for the fish state classification task is used, as well as the update mechanism, including the self-learning setting method of the threshold; the multi-classifier based on hypergraph convolution consists of three parts: constructing a hypergraph, constructing a hypergraph convolution, forming a new feature code, and finally performing classification prediction on the new feature code.

2. The fish behavior analysis method based on dynamic convolutional graph neural network according to claim 1, characterized in that: The method of using Yolo-v5 and Deepsort algorithms to obtain the trajectory information of the fish school as described in step 1 is as follows: using Yolo-v5 to perform target detection, detecting the position of a single fish, and adding a bounding box to it; using the DeepSORT model to track the trajectory of the fish, first obtaining the bounding box through the detection algorithm, then generating the number and position information, predicting through Kalman filtering, and using the Hungarian algorithm to match the predicted trajectory with the position in the current frame, including cascade matching and IOU matching, and finally performing Kalman filtering update; finally, using the two algorithms to obtain the trajectory information of the fish school, and recording the positions of all fish in each frame in a file.

3. The fish behavior analysis method based on dynamic convolutional graph neural network according to claim 1, characterized in that: Step 3: Calculate the fish behavior characteristic indicators including swimming speed, angular velocity, acceleration, fish activity, and dispersion. For the calculation of fish swimming speed, use and To represent the center of each fish's bounding box, represents the center of mass of each fish, where f represents the fish number, i represents the fish in the i-th frame; therefore, the swimming speed is calculated as: Where M represents the calculated value of the swimming speed of a single fish based on the M frames before and after each frame; in addition, the angle of the fish school is calculated as follows: In addition, use Indicates the size of each fish's bounding box and calculates the mean and variance of w and h for a single fish in M ​​frames of video; The activity of fish schools is calculated by correcting the effect of overlapping fish schools on activity, making the results more accurate and reliable. The activity calculation method is as follows: Specifically, the It is the bbox of two adjacent frames i+m-1 and bbox i+m Minimum area, the IoU i It is the bbox of two adjacent frames i+m-1 and bbox i+m The overlapping area and the bbox of two adjacent frames i+m-1 and bbox i+m The ratio of the area and the area, U i+m is the overlapping area of ​​two adjacent frames. However, the activity calculation depends on the image pixels, so the overlap of fish schools will cause errors in the activity calculation. Therefore, considering the overlap of fish schools, a correction coefficient is calculated, and finally the correction coefficient is multiplied by the original activity to obtain the final activity. ACT i =w*GIoU i (9) Wherein, N is the total number of actual fish, i and N i+1 is the number of visible fish in the two frames.

4. The fish behavior analysis method based on dynamic convolutional graph neural network according to claim 1, characterized in that: The memory unit described in step 5 is: the content stored in the memory unit is a key, a value, and a query vector; the key vector is a feature vector representing the center of the cluster, which will be used to calculate the similarity with the query vector; clustering is performed using the K-means algorithm, and the clustered features are the fish behavior features obtained in step 3; The query vector is extracted from the data during the training process; the value vector represents the information value stored in the memory unit to be returned, which describes the feature vector of the current specific classification; the key vector is equal to the value vector, and the query vector is normalized; if a query is required, the network will calculate the cosine similarity between the query vector and all currently stored key vectors, and select the matching feature vector with the greatest similarity and return it to the model. At the same time, this returned vector will be connected to the training process as an auxiliary feature of the sample.

5. The fish behavior analysis method based on dynamic convolutional graph neural network according to claim 1, wherein: The updating mechanism of the memory unit described in step 5 is: When a new query vector appears, it is updated; the distance between the newly created value vector and key vector and the key feature vector determines whether the original value vector and key vector value should be updated; when the distance is less than the set threshold, the key vector and value vector are updated and normalized; when the distance is greater than the set threshold, it means that the currently stored information cannot match the current query information, and a new set of key-value pairs will be created based on the value of the query vector and stored in the memory unit; Case 1: renew: renew: Case 2: Update: K[n l ]←queryV[n l ]←value(13) Specifically, the value represents the fish behavior feature accompanying the query vector query, K near and represents the storage key vector of the memory unit with the maximum cosine similarity calculated according to the algorithm and its corresponding value vector, the i represents the corresponding superscript of the fish behavior feature vector, the hyperparameter δ is the threshold; the K[n l ] and V[n l ] is the key and value vector of the newly added memory unit, n l The table adds a new key-value pair tag; the θ k and θ v are the update steps of key-value pairs respectively.

6. The fish behavior analysis method based on dynamic convolutional graph neural network according to claim 1, characterized in that: The hypergraph constructed in step 5 is: A hypergraph is constructed by obtaining the feature vectors X of all samples from the sample X through the memory network. In the hypergraph network, a vertex u is defined to represent a sample and a hyperedge e is defined to represent a sample set. The number of samples contained in the hyperedge is flexible. Specifically, Con(e) is used to represent the set of all nodes contained in the hyperedge e, and Adj(u) is used to represent all hyperedges containing node u. Specifically, k e represents the number of vertices in the hyperedge e, k u Represents the hyperedge containing vertex u; define u as the central vertex of the Adj(u) set; The central features of the memory unit are combined with KNN and K-means to construct a hypergraph, making full use of local and global information. Specifically, the k-1 nearest neighbors of each node u are calculated, and matching feature vectors are extracted from the memory unit according to the characteristics of each node. These neighbor vertices and the vertex itself form a hyperedge Adj(u). In addition, the K-means algorithm is applied to the entire feature graph based on the Euclidean distance. For each vertex, the nearest S-1 cluster group will be assigned to its adjacent hyperedge set. In particular, the initial feature set is used as the input set feature. The above operation is performed for each layer of features during training.

7. The fish behavior analysis method based on dynamic convolutional graph neural network according to claim 1, characterized in that: The construction of the hypergraph convolution described in step 5 is: Hypergraph convolution is divided into hyperedge convolution and vertex convolution; the vertex convolution aggregates features to the hyperedge, and the hyperedge convolution aggregates features to the central vertex of the hyperedge set; Use a multi-layer perceptron to learn a transformation matrix M from vertex features and assign weights, and then use one-dimensional convolution to further extract features and perform vertex convolution; M=MLP v (F u ) (15) The F u is the input vertex feature set, MLP v represents a multilayer perceptron, Represents multiplication, Conv1d represents one-dimensional convolution, F v is the vertex convolution output; Hyperedge convolution aggregates hyperedge features into the central vertex features of the hyperedge set; w=softmax(F v ·W+b) (16) W and b represent the weights and biases learned from convolution, F h is the output of the hyperedge convolution.

8. The fish behavior analysis method based on dynamic convolutional graph neural network according to claim 1, wherein: The formation of a new feature code as described in step five and the final classification prediction of the new feature code means: forming a new feature code based on the central vertex feature formed by the hyperedge convolution, and classifying the feature code through a classifier, and finally obtaining three classifications: normal state, eating state and stimulated state.

9. A fish behavior analysis system based on dynamic convolutional graph neural network, characterized in that: Include: Image acquisition and preprocessing module: acquires fish behavior videos and performs preprocessing; uses Yolo-v5 and Deepsort algorithms to obtain fish track information, and records the positions of all fish in each frame in a file; Fish behavior characteristic index acquisition module: extracts individual centroid coordinates, bounding boxes, and fish number of each fish based on fish track information; Fish behavior characteristic index calculation module: Calculates fish behavior characteristic indexes including swimming speed, angular velocity, acceleration, fish activity, and dispersion based on the extracted fish behavior characteristics; the final obtained fish behavior characteristic index set structure is Among them S id represents the video number in the dataset; f represents the fish number; i represents the fish in the i-th frame; v f Indicates the swimming speed of the fish; θ f Indicates the angle of the fish school; ACT i Indicates the final activity level of the fish school after correction; represents the mean value of the width w of the bounding box of a single fish in M ​​frames of video; represents the variance of the width w of a single fish bounding box in M ​​frames of video; The average height h of a single fish bounding box in M ​​frames of video; represents the variance of the height h of a single fish bounding box in M ​​frames of video; Fish behavior analysis graph construction module: construct edges based on a set of fish behavior characteristic indicators; construct graph edges based on neighbor distances and Delaunay triangles; Fish behavior analysis module: Construct a dynamic graph convolutional network for fish behavior analysis; the dynamic graph convolutional network for fish behavior analysis includes two units, a memory unit and a multi-classifier based on hypergraph convolution; in the memory network structure, the key, value and query vector construction method suitable for the fish school status classification task is used, as well as the update mechanism, including the self-learning setting method of the threshold; the multi-classifier based on hypergraph convolution consists of three parts: constructing a hypergraph, constructing a hypergraph convolution, forming a new feature code, and finally performing classification prediction on the new feature code.

Images