Fish behavior recognition method and system based on spatial pyramid attention

Abstract

The application belongs to the technical field of fish behavior recognition, and provides a fish behavior recognition method and system based on spatial pyramid attention, which adopts a dual-flow network with spatial pyramid attention, including a spatial flow and a motion flow, both of which use a spatial pyramid attention module to obtain spatial and temporal features of fish behavior. Specifically, the spatial pyramid attention module aggregates features from different levels of the baseline to capture the correlation between local and global information. Then, the extracted temporal features are combined with the spatial features to obtain a new fusion feature. The three features are given different weights in the final classification process through a learnable multiplier M for behavior recognition.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of fish behavior recognition, and particularly relates to a fish behavior recognition method and system based on a spatial pyramid attention. BACKGROUND

[0002] The statements in this section merely provide background information related to the present application and do not necessarily constitute the prior art.

[0003] With the increasing global demand for aquatic products, the aquaculture industry has been booming. Therefore, the welfare of fish in aquaculture is receiving more and more attention. Behavior is an important indicator of fish status in aquaculture, and fish behavior recognition can provide real-time understanding and early warning of fish status, which is very important for intensive aquaculture management. For example, timely identification of fish feeding behavior and hunger behavior can take reasonable and effective feeding strategies to improve aquaculture efficiency. By identifying abnormal behaviors such as fish fear, aquaculture managers can understand the impact of external conditions such as temperature, light, and sound on fish, and adjust the aquaculture environment in a timely manner to protect fish welfare.

[0004] Traditional fish recognition methods involve sending a person to observe fish behavior for a long time, but this method is labor-intensive and inefficient. In recent years, the development of computer vision technology has provided a low-cost and efficient method for fish behavior monitoring.

[0005] Machine learning methods for fish behavior recognition rely on handcrafted features. Beyan et al. extracted several behavior features of fish, including curvature scale space-based features, velocity and acceleration-based features, turning-based features, centroid distance functions, neighboring region features, and loop features. Principal component analysis is used to reduce the dimensionality of the features, and finally hierarchical classification and hidden Markov model clustering are used to identify abnormal behavior. Liu et al. defined a computer vision-based feeding activity index, which is generated by summing the intensity of all pixels in the difference frame obtained by subtracting two consecutive frames. The correlation coefficient between this index and the artificial observation feeding activity index reached 0.9195. Spontaneous collective behavior can also effectively evaluate the behavior state of fish. Researchers analyzed fish spontaneous collective behavior by calculating the amplitude of the flow field, interaction force, and dispersion. Spontaneous collective behavior was quantified and integrated to assess the appetite level of fish.

[0006] However, the production of manual features is usually time-consuming and laborious. In recent years, deep learning technology has become popular because it can automatically extract features through deep neural networks and has achieved remarkable results in image classification, object detection, object tracking, behavior recognition, etc. Therefore, deep learning technology has begun to be applied to aquaculture. Zhou et al. used a convolutional neural network (CNN) to evaluate the appetite level of fish. Compared with other machine learning methods, CNN has the best performance, with an accuracy of 90%. Yang et al. designed a dual attention network to realize image-based fish behavior analysis. The network obtains the spatial relationship between the regions of interest in the fish feeding image through a location attention module and a channel attention module, and then adopts various optimization strategies to train the network, and finally achieves an accuracy of 89.56% on the test set.

[0007] Although previous studies have made a lot of work on fish behavior recognition, most of the researches only define and recognize fish behavior from a global perspective. However, some fish behaviors occur between a few individuals in a local area. For example, hunger is usually a global behavior that occurs among most fish, while fear behavior is usually a local behavior that occurs between a few individuals in a local area. If only global behavior is focused on, it is difficult to identify local behavior, resulting in poor recognition effect, and vice versa. At the same time, in the actual breeding environment, due to the excessive breeding density, the phenomenon of cross occlusion between fish and fish and small area gathering often occurs, which also brings difficulties to the recognition of fish behavior. SUMMARY

[0008] To solve at least one technical problem in the above background art, the present application provides a fish behavior recognition method and system based on spatial pyramid attention, which proposes a dual-stream network based on spatial pyramid attention (SPA-TSN) to recognize fish behavior from both global and local perspectives, including a spatial stream and a motion stream, both of which use a spatial pyramid attention module to obtain the spatial and temporal features of fish behavior. The extracted temporal and spatial features are fused to obtain fused features, and the three features are used for behavior recognition after being given different weights by a learnable multiplier in the classification process.

[0009] To achieve the above purpose, the technical scheme adopted by the present application is as follows:

[0010] The first aspect of the present application provides a fish behavior recognition method based on spatial pyramid attention, comprising the following steps:

[0011] Obtain fish school behavior video data;

[0012] Combine the fish school behavior video data and the fish behavior recognition model to obtain the behavior recognition result; wherein the process of constructing the fish behavior recognition model is:

[0013] processing the fish school behavior video data into RGB images and optical flow images;

[0014] A double-stream network is adopted, for each stream, a spatial pyramid attention module is used to obtain corresponding features, the spatial stream takes the RGB image as input to extract the spatial features of fish behavior, and the motion stream takes the optical flow image as input to extract the temporal features of fish behavior, wherein the spatial pyramid attention module aggregates features from different levels of the baseline, captures the behavior semantics of fish from global and local perspectives, fuses the extracted temporal features and spatial features, obtains fusion features, and uses the three features for behavior recognition after being given different weights by a learnable multiplier in the classification process.

[0015] The second aspect of the present application provides a fish behavior recognition system based on spatial pyramid attention, comprising:

[0016] A data acquisition module is used to acquire fish school behavior video.

[0017] A fish behavior recognition module is used to obtain a behavior recognition result in combination with the fish school behavior video and a fish behavior recognition model, wherein the process of constructing the fish behavior recognition model is as follows:

[0018] processing the fish school behavior video into RGB images and optical flow images;

[0019] A double-stream network is adopted, for each stream, a spatial pyramid attention module is used to obtain corresponding features, the spatial stream takes the RGB image as input to extract the spatial features of fish behavior, and the motion stream takes the optical flow image as input to extract the temporal features of fish behavior, wherein the spatial pyramid attention module aggregates features from different levels of the baseline, captures the behavior semantics of fish from global and local perspectives, fuses the extracted temporal features and spatial features, obtains fusion features, and uses the three features for behavior recognition after being given different weights by a learnable multiplier in the classification process.

[0020] The third aspect of the present application provides a computer readable storage medium.

[0021] A computer readable storage medium has a computer program stored thereon, and the program is executed by a processor to realize the steps in the fish behavior recognition method based on spatial pyramid attention according to the first aspect.

[0022] The fourth aspect of the present application provides a computer device.

[0023] The computer device comprises a memory, a processor, and a computer program stored on the memory and executable on the processor, and the processor implements the steps in the fish behavior recognition method based on spatial pyramid attention according to the first aspect when executing the program.

[0024] Compared with the prior art, the present application has the following advantages:

[0025] 1、The PA of the present application aggregates features from different levels of Resnet-50 through a spatial pyramid module, while capturing fish behavior information from a global and local perspective. In addition, a attention module is used to enable the model to focus on the area of interest, thereby ignoring the cross-shading and small-area gathering phenomenon between fish.

[0026] 2、The double-stream fusion method proposed in the present application not only combines spatial and temporal features, but also assigns different weights to these features according to their importance to the current class. This enables the model to focus on the most relevant information for each class and achieve higher accuracy.

[0027] The advantages of the additional aspects of the present application will be partially given in the following description, partially will become obvious from the following description, or will be known by the practice of the present application. BRIEF DESCRIPTION OF DRAWINGS

[0028] The drawings accompanying the specification of this application form a part thereof, serve to provide further understanding of the present application, and together with the description of the exemplary embodiments of the present application and their description serve to explain the present application, and do not constitute improper limitations on the present application.

[0029] Figure 1 is the structure of the experimental system provided by the embodiment of the present application;

[0030] Figure 2 is the fish behavior data set provided by the embodiment of the present application; wherein, Figure 2 (a)-(d) in the table are feeding behavior, fear behavior, hunger behavior and normal behavior, respectively;

[0031] Figure 3 is the overall structure of the double-stream network based on spatial pyramid attention provided by the embodiment of the present application;

[0032] Figure 4 is the residual block provided by the embodiment of the present application;

[0033] Figure 5 is the RGB image and its corresponding optical flow image provided by the embodiment of the present application; Figure 5 (a)-(d) in the table are different RGB images, Figure 5 (e)-(h) in the table are the optical flow images corresponding to the RGB images;

[0034] Figure 6 is an attention module provided by an embodiment of the present application;

[0035] Figure 7 is a spatial pyramid module provided by an embodiment of the present application;

[0036] Figure 8 are two commonly used dual-stream fusion methods provided by an embodiment of the present application; Figure 8 in which (a) is feature-level fusion, Figure 8 in which (b) is decision-level fusion;

[0037] Figure 9 is a confusion matrix of different models provided by an embodiment of the present application; Figure 9 in which (a) is spatial stream, Figure 9 in which (b) is motion stream, Figure 9 in which (c) is decision-level fusion, Figure 9 in which (d) is feature-level fusion, Figure 9 in which (e) is the method of the present application;

[0038] Figure 10 is a precision curve of a comparative experiment provided by an embodiment of the present application;

[0039] Figure 11 is SPA-TSN attention region visualization provided by an embodiment of the present application, Figure 11 in which (a)-(d) show overall fish behavior, Figure 11 in which (e)-(f) show local fish behavior. DETAILED DESCRIPTION

[0040] The present application will be further described with reference to the drawings and examples.

[0041] It should be noted that the following detailed description is illustrative only and is not intended to limit the application as described herein. Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs.

[0042] It is to be understood that the singular forms "a," "an," and "the" include plural referents unless the context clearly dictates otherwise. It is to be understood that the terms "comprising," "including," and "having" can be used interchangeably. Furthermore, it is to be understood that where the application, or any embodiment thereof, is dependent upon a recited event occurring or not occurring, the recited event can occur or not occur.

[0043] Accurate monitoring of fish behavior is the key to improving the efficiency of intensive aquaculture. Previous studies on fish behavior recognition are mostly limited to recognizing fish group behavior from a global perspective, ignoring the importance of recognizing local area behavior. The invention proposes a dual-stream network based on spatial pyramid attention, specifically, SPA-TSN aggregates features from different levels of Resnet-50 through a spatial pyramid module, capturing fish behavior information from global and local perspectives at the same time. In addition, an attention module is used to enable the model to focus on the area of interest, thereby ignoring the cross-shading and small-area gathering phenomenon between fish.

[0044] In order to verify the method of the invention, a fish behavior dataset composed of four typical fish behaviors was collected in a real farming environment, and experiments were conducted on this dataset. The experimental results show that the accuracy of the SPA-TSN model on the test set reaches 95.904%, which is better than other advanced methods. In addition, the class activation map is visualized by grad-cam, which shows that the model proposed in the invention can identify behaviors occurring globally and locally in fish, achieving the expected goal.

[0045] Embodiment one

[0046] The embodiment provides a fish behavior recognition method based on spatial pyramid attention, which includes the following steps:

[0047] Step 1: Obtain fish group behavior dataset;

[0048] During the fifteen-day farming period, 20 segments of fish group behavior videos were continuously collected. When obtaining data, the diversity of the dataset was improved by frequently changing the shooting angle.

[0049] With the help of expert experience, the behaviors in the video are divided into four categories: feeding behavior, fear behavior, hunger behavior, and normal behavior, as shown in Table 1. Figure 2

[0050] Table 1: Characteristic description of four fish behaviors

[0051]

[0052] Step 2: Data preprocessing

[0053] In order to solve the problem of insufficient sample size, the embodiment uses data augmentation technology to expand the dataset.

[0054] ​Specifically, the number of samples in the dataset is expanded by flipping and rotating, and finally reaches 1990. The resulting samples are divided into training and test sets in a ratio of 7:3. Ensure that samples from the same video only appear in one of the subsets to avoid overfitting of the model due to specific conditions of the video (such as lighting conditions, camera angles, etc.).

[0055] The distribution of the dataset is shown in Table 2.

[0056] Table 2. Distribution of the dataset

[0057]

[0058] Step 3: Fish behavior recognition

[0059] The present application designs a fish behavior recognition algorithm based on spatial pyramid attention (SPA-TSN), which realizes accurate recognition of fish behavior. Figure 3 The overall architecture of the proposed recognition algorithm is shown.

[0060] Specifically, the fish behavior recognition model includes two parts: spatial stream CNN and motion stream CNN. For each stream, a spatial pyramid attention structure is designed to capture the behavior semantics of fish from global and local perspectives.

[0061] First, the spatial stream and the motion stream take RGB images and optical flow images as input, respectively, to extract the spatio-temporal features of fish behavior. Then, the spatial and temporal features are fused to obtain the final classification result.

[0062] The dual-stream network includes two parts: spatial stream CNN and motion stream CNN, which are used to capture the spatial and temporal features of fish, respectively. Due to the good performance of residual networks in various fields of deep learning, Resnet-50 is selected as the baseline network for each stream. Table 3 shows the overall architecture of Resnet-50.

[0063] Table 3. Overall architecture of Resnet50

[0064]

[0065] Specifically, Conv1 is a single convolutional layer, and Conv2_x-Conv_5 is stacked by different numbers of residual blocks. Figure 4 Figures (a) and (b) show two different types of residual blocks, which differ in whether a 1x1 convolutional layer is used in the residual connection. These two residual blocks jointly construct Conv2_x-Conv5_x.

[0066] Spatial stream: Randomly crop a single frame of the video, and input the normalized frame into the spatial stream network. The input dimension is 3x224x224, and the output is a fixed-length feature vector containing spatial information of fish behavior.

[0067] Optical flow: Optical flow is a method that uses the changes of pixels in the time domain and the correlation between adjacent frames to find the correspondence between the previous frame and the current frame, and then calculates the motion information of objects between adjacent frames. The traditional method usually uses Lucas-Kanade optical flow to calculate the optical flow. However, with the advent of the deep learning era, researchers have begun to try to use End-to-End network models to solve the problem of optical flow estimation. Flownet is a deep learning model designed to solve the limitations of traditional methods in terms of calculation speed and detection accuracy, and has been widely used in optical flow estimation.

[0068] Therefore, the present application uses the Flownet model to calculate the optical flow of fish school motion to obtain the time information of fish school behavior, and visualizes the calculated optical flow as an RGB picture format. Figure 5 Figures (a)-(h) show the original RGB pictures of four fish behaviors and the optical flow pictures generated with adjacent frames.

[0069] Motion stream: The motion stream network stacks 10 consecutive optical flow pictures as network input, and the preprocessing operation is consistent with the spatial stream. The input dimension is 30x224x224, and the output of the motion stream is also a fixed-length feature vector containing the time information of fish behavior.

[0070] Attention mechanism was originally applied in natural language processing to capture context relationships. In recent years, it has been widely used in computer vision to capture global correlations between local features. In this study, an attention module is proposed to focus the network on the area of interest to solve the problem of cross-occlusion and small clusters between fish. It is worth noting that the proposed attention module is a plug-and-play module that can accept any dimension input. The structure of the attention module is shown in Figure 6 .

[0071] Given an input where C is the number of channels, H and W represent the height and width of the feature map respectively, and the attention calculation operation can be represented by equations (1)-(2):

[0072]

[0073] y = x + conv(z i ) (2)

[0074] where x is the input feature, y is the output feature, and Q, K, and V correspond to query, key, and value, respectively. Q, K, and V are all generated by the input feature x after a convolution layer with a kernel size of 1x1, and the feature dimensions are all d k = c / n. n is an adjustable variable, and the purpose is to reduce the computational complexity in the self-attention operation. In Conv_3x and Conv_4x, n = 2 in this embodiment; in Conv_5x, n = 4.

[0075] The feature z calculated by formula (1) i The dimension is still d k In order to obtain the same dimension as the input feature x, let z i through a convolution layer with a kernel size of 1x1, and then residual connection with x. In this way, the attention module can be inserted anywhere in the network without destroying the original information.

[0076] Generally speaking, the deep convolutional neural network has a larger receptive field and more rich semantic information, but it often ignores important local information while having high-level visual perception. On the contrary, the bottom layer has high resolution but lacks global semantics. In order to capture both the overall behavior and the local behavior of fish, the spatial pyramid module is designed. This module aggregates features of different levels of ResNet-50 to obtain global and local fish behavior semantics at the same time, such as Figure 7 as shown by the black dashed box in the middle.

[0077] Considering that the bottom layer may introduce too much noise and higher computational complexity, this embodiment only aggregates features of three levels of Conv3_x, Conv4_x, and Conv5_x, denoted as F3, F4, and F5, respectively. Among them, n ∈R C×H×W C is the number of channels, H and W represent the height and width of the feature map, respectively, and n ∈ {3, 4, 5} represents the features from different levels of Resnet-50. The values of H, W, and C change with n. In particular, 512×28×28 F3 ∈ R 1024×14×14 F4 ∈ R 2048×7×7 .

[0078] After processing F3, F4, and F5 through the attention module, global pooling is performed. It is worth noting that a 1x1 convolution layer is added before global pooling for F3 and F4 to adjust the dimension. After global pooling, the output feature vectors Fo3, Fo4, and Fo5 corresponding to Conv3_x, Conv4_x, and Conv5_x can be represented as: on = [f n,1 ,f n,1 …,fn,N Where n∈{3,4,5}, N=2048.

[0079] F o3 ,F o4 ,F o5 The final output F is obtained after splicing. o F o ∈R 3×2048 .

[0080] Fish behavior recognition requires both spatial and temporal information; therefore, fusing spatiotemporal features is crucial for final classification. Commonly used two-stream fusion methods include feature-level fusion and decision-level fusion, such as... Figure 8 As shown, feature-level fusion first concatenates or adds the features extracted from the spatial flow and motion flow to obtain a fused feature, and then obtains the classification result. Decision-level fusion first generates a classification score for each of the spatial flow and motion flow, and then fuses the two to obtain the final classification result.

[0081] This invention proposes a dual-stream fusion method to integrate spatiotemporal information. The fusion method employs a learnable weighting scheme to combine spatial and temporal features. The weighting scheme is learned during training using a cross-entropy loss function and a backpropagation algorithm.

[0082] For a specific video clip input, spatial and temporal information are extracted using spatial flow and motion flow, respectively, denoted as F. o,rgb and F o,flow Then, F is obtained by adding elements together. o,rgb ,F o,flow Generate a new fusion feature F o,fusion As shown in equation (3):

[0083] F o,fusion =F o,rgb +F o,flow (3)

[0084] in

[0085] For F o,fusion ,F o,rgb ,F o,flow Each of them is passed through a fully connected layer to generate its own classification score C. fusion C rgb C flow Subsequently, a trainable multiplier M was designed. fusion M rgb M flow The weights of the three features in the final classification result are automatically adjusted, as shown in equation (4):

[0086]

[0087] wherein, is the predicted classification result, and represents the matrix multiplication of the corresponding element multiplication.

[0088] The loss function is selected as the cross-entropy function, as shown in equation (5).

[0089]

[0090] wherein y is the real classification label, n is the batch size, and x is the number of classes.

[0091] The experimental object in this experiment is the D1 pool of the No. 1 aquaculture workshop of a certain aquaculture company in Shandong Province. These fish have been cultured in the circulating water system for about 6 months before the experiment. The fish population is fed by an automatic feeder, which feeds five times a day.

[0092] The fish behavior recognition experiment is carried out in a real industrial circulating water aquaculture environment, as shown in Figure 1 The breeding pond is a square with a side length of 6.85 m and a height of 1 m, with a water depth of about 0.8 m. The water temperature is about 25°C, and the pH value is about 8. The RGB camera is located above the breeding pond, about 2 m from the ground, continuously collecting fish behavior videos, with a video resolution of 1920*1080 and a rate of 30 frames / second. The deep learning model is deployed on two Nvidia GTX 1080Ti servers and implemented using the Pytorch framework in the Linux Ubuntu 20.04.3LTS environment.

[0093] In the invention, four evaluation indicators-accuracy, precision, recall rate and specificity-are used to evaluate the performance of the fish behavior recognition method proposed in the invention. The accuracy refers to the proportion of correctly classified samples to the total number of samples. The precision refers to the proportion of correctly predicted samples of a certain class to the total number of predicted samples. The recall rate refers to the proportion of correctly predicted samples of a certain class to the actual number of samples, while the specificity refers to the proportion of true negative samples in negative samples. The calculation methods of accuracy, precision, recall rate and specificity are shown in equations (6)-(9):

[0094]

[0095]

[0096]

[0097]

[0098] Among them, TP, FN, FP, and TN represent true positive, true negative, false positive, and false negative respectively.

[0099] In this section, a large number of experiments were conducted on the fish behavior dataset to evaluate the performance of the algorithm proposed in this invention. First, ablation experiments were carried out to verify the effectiveness of each module proposed in Section 2. Next, the model of this invention was compared with other advanced methods. Finally, the experimental results of the model were discussed.

[0100] To improve the performance of the model, this invention adopted the transfer learning method. The model was pre-trained on the UCF101 dataset. UCF101 is a real human behavior recognition dataset collected from YouTube, providing 13,320 videos from 101 action categories. After the pre-training stage on UCF101, the model was continued to be trained using the fish behavior dataset of this invention. The main hyperparameters during the training process are shown in Table 4.

[0101] Table 4 Main Hyperparameters

[0102]

[0103]

[0104] To verify the effectiveness of the three operations (spatial pyramid module, attention module, and two-stream fusion method) proposed in this invention, the following ablation experiments were designed. First, the baseline network of each stream was tested, and then the spatial pyramid and attention modules were added in turn. Finally, this process was repeated with two-stream input.

[0105] The experimental results are shown in Table 5. First, the dataset of this invention was executed on the spatial stream baseline and the motion stream baseline (named RBG only and Flow only respectively), and the accuracies of the test set were 86.348% and 91.809% respectively. After adding the spatial pyramid module (Spatial Pyramid, SP), the accuracies on the test set reached 90.444% and 94.198% respectively. After further adding the attention module (Attention), the accuracies of the test set finally reached 91.468% and 95.222%. Then, two-stream input was adopted, and the accuracies of the test set in the three cases were 92.491%, 94.888%, and 95.904% respectively.

[0106] Table 5 Ablation Experiment Results

[0107] <00​​​​Next, experiments were conducted to evaluate the effectiveness of the fusion method proposed in the present application. Three different fusion methods were tested: feature-level fusion, decision-level fusion, and the dual-stream fusion method proposed in the present application. The results are shown in Table 6, with the accuracy rates of the three methods being 93.857%, 94.368%, and 95.904%, respectively. Surprisingly, the accuracy rates of the feature-level fusion and decision-level fusion methods were even lower than that of the motion stream after adding the spatial pyramid module and attention module. To explore the reasons for these results, the confusion matrix plots under the spatial stream, motion stream, and three different dual-stream fusion methods were plotted and analyzed, as shown in FIG. 6. Figure 9 The number of correct predictions is described on the diagonal line, with darker colors indicating higher confidence and lighter colors indicating lower confidence.

[0110] Analysis of the same can find that the feeding behavior is more dependent on spatial distribution information, and its accuracy on the spatial stream is higher, while the fear behavior is more dependent on the motion information on the time scale, and its performance on the motion stream is better. For decision-level fusion, the results are very similar to those when only using the motion stream, which indicates that the motion stream obtains a higher weight during fusion, and the spatial distribution information provided by the spatial stream is ignored. On the other hand, although the feature-level fusion method fuses temporal features and spatial features, it does not achieve the expected results. It is speculated that feature-level fusion may lose the dependence of certain behaviors on a particular stream (e.g., fear behavior is more dependent on the motion stream), resulting in poor recognition accuracy.

[0111] In comparison, the dual-stream fusion method proposed in the present application not only combines spatial and temporal features, but also assigns different weights to these features according to their importance to the current class. This enables the model to focus on the most relevant information for each class and achieve higher accuracy. The experimental results show that this fusion method has the highest accuracy among all fusion methods, with an accuracy rate of 95.904% on the fish behavior dataset. This result demonstrates the effectiveness of the proposed dual-stream fusion method in improving the performance of behavior recognition models.

[0112] To verify the advancement of the proposed SPA-TSN model, comparative experiments were conducted with other advanced fish behavior recognition deep learning algorithms, including AlexNet, ResNet-50, ResNet-101, ResNet152, DAN-EfficientNet-B2, MobileNet_v2, C3D, and DSC3D. All the above models were pre-trained on UCF-101, and the experimental results are shown in Table 7. The results show that the accuracy of AlexNet, ResNet-50, ResNet-101, ResNet-152, DAN-EfficientNet-B2, and MobileNetV2-SENet are all lower than the motion flow baseline (91.809%). This may be due to the fact that these models only use single-frame RGB images as input, ignoring the time information that is crucial for fish behavior. It is also worth noting that the ResNet-101 model outperforms ResNet-50 and ResNet-152 on the test set. To explore the reason, the precision curves of different models on the test set are generated, as shown in Figure 10 It can be seen that AlexNet has poor accuracy due to its shallow network structure. However, despite the deeper network structure of ResNet-152, it shows poor performance and is difficult to converge even after 40 iterations of training. This indicates that increasing the depth of the network can improve the efficiency of the model, but an excessively deep network structure may lead to overfitting, making the network difficult to train. Compared with other models that consider time information (C3D and DSC3D), the accuracy of the proposed method is improved by 3.071% and 2.028%, respectively.

[0113] The results of the comparative experiments show that the proposed method has the best performance, which is mainly due to the following two factors. First, the dual-stream network structure is used to capture the spatio-temporal information of fish behavior. Second, the spatial pyramid attention module can extract the behavior features of fish from both global and local perspectives.

[0114] Table 7. Comparative experimental results

[0115]

[0116]

[0117] The SPA-TSN model is a deep learning algorithm designed to recognize fish behavior by considering both global and local perspectives. Compared with the most advanced methods, the SPA-TSN model shows superior recognition accuracy. To further evaluate the effectiveness of the model, the class activation map of the model is visualized, as shown in Figure 11 Some typical images of group behavior and local behavior are selected from the dataset for visualization. Figure 11Figures (a)-(d) show the overall fish behavior, Figure 11 Figures (e)-(f) show the local fish behavior.

[0118] It is worth noting that in Figure 11 In figure (f), the model of the present application accurately detects that one fish at the bottom of the image exhibits a fearful behavior, which indicates that the model has the ability to detect local behavior. At the same time, the model of the present application also identifies the fearful behavior of a group of fish at the top right of the image, which also proves the ability of the model to capture global behavior.

[0119] Embodiment Two

[0120] The present embodiment provides a fish behavior recognition system based on spatial pyramid attention, comprising:

[0121] A data acquisition module for acquiring a school of fish behavior video;

[0122] A fish behavior recognition module for obtaining a behavior recognition result by combining the school of fish behavior video and a fish behavior recognition model; wherein the process of constructing the fish behavior recognition model is:

[0123] Processing the school of fish behavior video into an RGB image and an optical flow image;

[0124] Using a double-stream network, for each stream, using a spatial pyramid attention module to obtain the corresponding features, the spatial stream extracts the spatial features of the fish behavior by taking the RGB image as the input, and the motion stream extracts the temporal features of the fish behavior by taking the optical flow image as the input, wherein the spatial pyramid attention module aggregates features from different levels of the baseline, captures the behavior semantics of the fish from a global and local perspective, fuses the extracted temporal features and spatial features, obtains the fused features, and uses the three features in the classification process after being given different weights by a learnable multiplier for behavior recognition.

[0125] Embodiment Three

[0126] The present embodiment provides a computer readable storage medium having a computer program stored thereon, the program being executed by a processor to implement the steps of the fish behavior recognition method based on spatial pyramid attention of embodiment one.

[0127] Embodiment Four

[0128] The present embodiment provides a computer device, comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, wherein the processor executes the program to implement the steps of the fish behavior recognition method based on spatial pyramid attention of embodiment one.

[0129] Those skilled in the art will appreciate that embodiments of the application can be readily used as software, hardware, or a combination of software and hardware. In a software embodiment, the methods can be tangibly embodied in a machine-readable storage medium having stored thereon instructions that can be used to program a processing system to perform the methods. The term "processor," as used herein can refer to one or more processors capable of executing a software routine created to perform one or more processes described herein. The processor can be implemented as one or more central processing units (CPUs), one or more microprocessors, one or more microcomputers, one or more microcontrollers, one or more digital signal processors, one or more graphics processing units (GPUs), one or more processing cores, one or more processing units, one or more processing circuits, one or more processing devices, one or more processors, one or more processing means, or any combination thereof. The processor can be implemented as a single-chip processor, a multi-chip processor, a multi-core processor, or any combination thereof.

[0130] The present application is described in reference to the flowchart and / or block diagrams of the methods, apparatus (systems) and computer program products according to embodiments of the present application. It will be understood that each block of the flowchart and / or block diagrams, and combinations of blocks in the flowchart and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general purpose computer, special purpose computer, embedded processing apparatus, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart and / or block diagram block or blocks. Figure 1 one or more functions specified in the flowchart and / or block diagram block or blocks. Figure 1 one or more functions specified in the flowchart and / or block diagram block or blocks.

[0131] These computer program instructions can also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instructions which implement the function specified in the flowchart and / or block diagram block or blocks. Figure 1 one or more functions specified in the flowchart and / or block diagram block or blocks. Figure 1 one or more functions specified in the flowchart and / or block diagram block or blocks.

[0132] These computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart and / or block diagram block or blocks. Figure 1 one or more functions specified in the flowchart and / or block diagram block or blocks. Figure 1 one or more functions specified in the flowchart and / or block diagram block or blocks.

[0133] It will be understood by those within the art that the methods of the embodiments described above can be implemented in software, hardware, or a combination thereof. In a software implementation, the various processes and functions described above can be implemented as one or more computer programs tangibly embodied in a computer-readable storage medium, such as a magnetic disk, a magnetic tape, an optical disk, a memory, a memory card, or the like. The program code can be in the form of a source code, an object code, a script, machine code, or the like.

[0134] The above description is only the preferred embodiment of the present application, and is not intended to limit the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A fish behavior recognition method based on spatial pyramid attention, characterized in that, Includes the following steps: Acquire video data of fish behavior; The behavior recognition result is obtained by combining video data of fish school behavior and a fish behavior recognition model; wherein, the process of constructing the fish behavior recognition model is as follows: The video data of fish behavior was processed into RGB images and optical flow images; A dual-stream network is adopted. For each stream, a spatial pyramid attention module is used to obtain the corresponding features. The spatial stream extracts the spatial features of fish behavior from RGB images as input, and the motion stream extracts the temporal features of fish behavior from optical flow images as input. ResNet-50 is used as the baseline network for each stream. The spatial pyramid attention module aggregates features from different levels of the baseline to capture the semantics of fish behavior from both global and local perspectives. The extracted temporal features are fused with the spatial features to obtain fused features. The three features are then used for behavior recognition after being assigned different weights in the classification process through a learnable multiplier. In the spatial pyramid attention module, when features from different levels of the baseline are aggregated, only the features from the third, fourth and fifth levels are aggregated. After processing by the attention module, global pooling is performed. After global pooling, the feature vectors output by the three levels are concatenated to obtain the final output.

2. The fish behavior recognition method based on spatial pyramid attention as described in claim 1, characterized in that, After acquiring video data of fish behavior, data augmentation techniques were used to expand the dataset.

3. The fish behavior recognition method based on spatial pyramid attention as described in claim 1, characterized in that, Processing fish school behavior video data into RGB images and optical flow images includes: using the FlowNet model to calculate the optical flow of fish school movement to obtain temporal information of fish school behavior, and visualizing the calculated optical flow into RGB image format.

4. The fish behavior recognition method based on spatial pyramid attention as described in claim 1, characterized in that, The learnable multiplier is trained using a cross-entropy loss function and a backpropagation algorithm.

5. The fish behavior recognition method based on spatial pyramid attention as described in claim 1, characterized in that, The behavioral recognition results include eating behavior, fear behavior, hunger behavior, and normal behavior.

6. A fish behavior recognition system based on spatial pyramid attention, characterized in that, include: The data acquisition module is used to acquire videos of fish behavior. A fish behavior recognition module is used to combine fish behavior videos and a fish behavior recognition model to obtain behavior recognition results; wherein, the process of constructing the fish behavior recognition model is as follows: The video of fish school behavior was processed into RGB images and optical flow images; A dual-stream network is adopted. For each stream, a spatial pyramid attention module is used to obtain the corresponding features. The spatial stream extracts the spatial features of fish behavior from RGB images as input, and the motion stream extracts the temporal features of fish behavior from optical flow images as input. ResNet-50 is used as the baseline network for each stream. The spatial pyramid attention module aggregates features from different levels of the baseline to capture the semantics of fish behavior from both global and local perspectives. The extracted temporal features are fused with the spatial features to obtain fused features. The three features are then used for behavior recognition after being assigned different weights in the classification process through a learnable multiplier. In the spatial pyramid attention module, when features from different levels of the baseline are aggregated, only the features from the third, fourth and fifth levels are aggregated. After processing by the attention module, global pooling is performed. After global pooling, the feature vectors output by the three levels are concatenated to obtain the final output.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the steps in the fish behavior recognition method based on spatial pyramid attention as described in any one of claims 1-5.

8. A computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps in the fish behavior recognition method based on spatial pyramid attention as described in any one of claims 1-5.

Images