An animal behavior recognition method based on spatio-temporal attention mechanism

Abstract

The present invention provides an animal behavior recognition method based on a spatio-temporal attention mechanism, comprising the following steps: (1) Using a hardware platform with motion detection, collecting and recording video data of each target animal; (2) The video data is wirelessly transmitted to a remote host, where sparse sampling of the video image is first performed, the data is converted from a video stream to an image sequence, and then preprocessing such as tensor splicing and numerical normalization is carried out; (3) Training the YOLOv5s model, detecting the target and outputting the height, width and coordinates of the target regression box, segmenting the image, and outputting each sub-image sequence. The present invention combines the algorithm model with the hardware platform to form a complete system from video acquisition, target detection to behavior recognition; at the same time, the attention mechanism is used to reasonably allocate the weight ratios of each feature, and the best embedding method of the CBAM attention module in the TSM model is studied, improving the accuracy of behavior recognition.

Description

Technical Field

[0001] The present invention relates to the technical field of intelligent behavior detection, and particularly relates to an animal behavior recognition method based on a spatio-temporal attention mechanism. Background Art

[0002] Nowadays, the global aquaculture industry is developing towards large-scale and standardized directions, and factory farming methods are becoming more and more common. In the factory farming mode, the intelligent detection of livestock behavior is particularly important. Before the clinical and subclinical signs of animal diseases appear, there are usually changes in the animals' own behaviors. Using non-contact computer vision technology to monitor and identify the abnormal behaviors of livestock can reduce the emaciation and even death of livestock, thereby controlling economic losses and improving breeding efficiency. Therefore, intelligent behavior detection is a technology urgently needed in the aquaculture industry.

[0003] In the aspect of computer vision, the research on intelligent behavior detection can be mainly divided into two categories: image-based analysis and video-based analysis. The image-based animal behavior analysis method usually studies the postures of animals, using only single-frame images as the data source, mainly including methods such as image classification and region classification. The video-based animal behavior analysis method not only focuses on the spatial position features of animal behaviors in single-frame images, but also considers the temporal features of image changes during the action duration, mainly including methods such as 3D convolutional models and two-stream models.

[0004] However, the image-based processing method can only achieve the classification of static behaviors and is difficult to extract dynamic features; while the video-based processing method is still immature and has low accuracy in the behavior recognition tasks with multiple categories and not obvious spatial features. And the calculation of optical flow estimation and 3D convolution is complex and computationally intensive, making it difficult to meet the real-time requirements.

[0005] In summary, the existing technologies mainly have problems of large computational amount and low accuracy, and the main reasons include incomplete feature extraction, unreasonable model structure, overly complex algorithm process, and difficult video acquisition, etc. Summary of the Invention

[0006] The purpose of the present invention is to provide an animal behavior recognition method that collects video data of target animals, realizes behavior recognition, target detection and segmentation, and thus conducts video classification and recognition.

[0007] To this end, the present invention adopts the following technical solutions:

[0008] An animal behavior recognition method based on a spatio-temporal attention mechanism, comprising the following steps:

[0009] (1) Using a hardware platform with motion detection to collect and record the video data of each target animal;

[0010] (2) The video data is wirelessly transmitted to the remote host. First, sparse sampling of the video images is performed, and the data is converted from a video stream to an image sequence. Then, preprocessing such as tensor splicing and numerical normalization is carried out.

[0011] (3) The YOLOv5s model is trained to detect the target and output the height, width, and coordinates of the target regression box, segment the image, and output the sub-image sequence.

[0012] (4) For a sub-image sequence data, the TSM model based on the spatio-temporal attention mechanism is used for feature extraction and classification recognition. The channel attention module is used to enhance the temporal features, and the spatial attention module is used to enhance the spatial features to allocate the weight ratios of each feature. And the improved overall model is trained end-to-end to obtain the classification results of animal behavior features. The trained model is used in the actual scenario of behavior recognition.

[0013] Furthermore: The hardware platform for motion detection in the above step (1) includes an intelligent vehicle platform, which is equipped with a target detection module, a data processing module, a driving device, and a camera. The target detection module is used to search for the target animal, and the camera is used to record the video data.

[0014] Furthermore: The video data in the above step (2) is sent to the remote host via the same wireless network segment, and uniform sampling is performed to output a video frame sequence. Each frame image is read, a time dimension is added and spliced into a tensor format, and z-score numerical normalization is performed on each dimension value.

[0015] Furthermore: In the sparse sampling of the video images in the above step (2), the video is evenly divided into several segments, and the duration of each segment is a rated millisecond time period. One frame image is randomly selected from each video segment as the animal behavior image data of this time period, and all the selected images generate sequence data in chronological order.

[0016] Furthermore: The YOLOv5s model in the above step (3) uses the parameters trained through the dataset to output the height, width, and coordinates of the target regression box.

[0017] Furthermore: All the sampled image sequences are segmented according to the regression box, and a sub-image sequence with only a single target in the picture is output.

[0018] Furthermore: The improved overall model in the above step (4) uses Resnet50 as the backbone network, and adds a TSM module and a CBAM attention module, including a convolutional layer, a max pooling layer, a first module, a second module, a third module, a fourth module, an average pooling layer, and a fully connected layer connected in series in turn;

[0019] The convolutional layer uses 64 7×7 convolutional kernels with a stride of 2;

[0020] The max pooling layer uses a 3×3 pooling kernel with a stride of 2;

[0021] The first module uses 3 identical Bottleneck series structures. Each Bottleneck includes a TSM module, a 1×1×64 convolutional layer, a 3×3×64 convolutional layer, and a 1×1×256 convolutional layer connected in series in sequence;

[0022] The second module uses 4 identical Bottleneck series structures. Each Bottleneck includes a TSM module, a channel attention module, a spatial attention module, a 1×1×128 convolutional layer, a 3×3×128 convolutional layer, and a 1×1×512 convolutional layer connected in series in sequence;

[0023] The third module uses 6 identical Bottleneck series structures. Each Bottleneck includes a TSM module, a channel attention module, a spatial attention module, a 1×1×256 convolutional layer, a 3×3×256 convolutional layer, and a 1×1×1024 convolutional layer connected in series in sequence;

[0024] The fourth module uses 3 identical Bottleneck series structures. Each Bottleneck includes a TSM module, a 1×1×512 convolutional layer, a 3×3×512 convolutional layer, and a 1×1×2048 convolutional layer connected in series in sequence;

[0025] The average pooling layer uses a 7×7 pooling kernel;

[0026] The fully connected layer is combined with the Softmax function to output the classification result.

[0027] Furthermore: The TSM module of the improved model in step (4) above includes two calculation paths. One is the main line, which directly sends the spatial feature map to the next convolutional network layer; the other is the branch line, which performs a shift operation on some channels in the time series, extracts features using 2D convolution, and finally sends them to the next layer in the form of a residual;

[0028] The CBAM attention module of the improved model includes a channel attention module and a spatial attention module connected in series in sequence;

[0029] Channel attention module: The input feature map F(H×W×C) is respectively fed into the global max-pooling layer and global average-pooling layer based on width and height to obtain two feature vectors of 1×1×C; the two feature vectors are respectively fed into a two-layer neural network MLP. The number of neurons in the first layer is C / r, where r is the reduction rate, and the activation function is Relu. The number of neurons in the second layer is C; the two feature vectors output by the MLP are concatenated by channels, and after passing through the sigmoid activation function, the final channel attention weight, i.e., M_c, is generated; finally, the multiplication operation based on channels is performed on M_c and the input feature map F, and the output feature map F’ is obtained.

[0030] Spatial attention module: The input is the feature map F’ output by the channel attention module. First, global max-pooling and global average-pooling based on channels are performed to obtain two feature maps of H×W×1; the two feature maps are concatenated by channels and passed through a 7×7 convolutional layer to reduce the dimension to 1 channel, i.e., H×W×1; after passing through the sigmoid activation function, the spatial attention weight, i.e., M_s, is generated; finally, M_s and the input feature of this module are multiplied according to the spatial position to obtain the finally generated feature.

[0031] Furthermore, for the improved overall model in step (4) above, the formula for calculating the loss function is as follows: where C is the total number of categories, y i represents the label, and G i represents the score of the video belonging to each category;

[0032]

[0033] In the above formula, G is the fusion function, which adopts a fully connected layer and a Softmax function. The input is the output after the main network processes each video frame sampled in the time series, and the output is the probability that the entire video data is classified into each category.

[0034] Compared with the prior art, the present invention has the following beneficial effects:

[0035] The present invention combines the algorithm model with the hardware platform to form a complete system from video acquisition, target detection to behavior recognition; at the same time, the attention mechanism is used to reasonably allocate the weight ratios of each feature, and the best embedding method of the CBAM attention module in the TSM model is studied, improving the accuracy of behavior recognition. The model structure designed by the present invention is simple and lightweight. Compared with other behavior recognition models such as 3D convolutional networks and two-stream models, it has a small amount of calculation and high real-time performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 It is a schematic flow chart of the present invention;

[0037] Figure 2Schematic diagram of the Y-TSM model network structure of the present invention;

[0038] Figure 3 Effect diagram of object detection of the YOLOv5s model of the present invention;

[0039] Figure 4 Structural diagram of the TSM module of the present invention;

[0040] Figure 5 Structural diagram of the channel attention module of the present invention;

[0041] Figure 6 Structural diagram of the spatial attention module of the present invention;

[0042] Figure 7 Effect diagram of motion detection by frame difference method of the present invention;

[0043] Figure 8 Accuracy rate curve of the training set in the training process of the improved TSM model of the present invention;

[0044] Figure 9 Loss function value curve in the training process of the improved TSM model of the present invention. Specific implementation manners

[0045] The present invention will be further described below in conjunction with the drawings and embodiments, but it shall not be used as a basis for limiting the present invention.

[0046] As Figures 1-9 shown, a method for animal behavior recognition based on spatio-temporal attention mechanism includes the following steps:

[0047] (1) Using a hardware platform with motion detection, collect and record video data of each target animal;

[0048] (2) The video data is wirelessly transmitted to a remote host. First, sparse sampling of the video image is performed, and the data is converted from a video stream to an image sequence, and then preprocessing such as tensor splicing and numerical normalization is performed;

[0049] (3) Train the YOLOv5s model, detect the target and output the height, width and coordinates of the target regression box, segment the image, and output each sub-image sequence;

[0050] (4) For a sub-image sequence data, use the TSM model based on spatio-temporal attention mechanism to perform feature extraction and classification recognition, use the channel attention module to enhance the temporal features, use the spatial attention module to enhance the spatial features, so as to allocate the weight ratio of each feature, and perform end-to-end training on the improved overall model to obtain the animal behavior feature classification result, and the trained model is used in the actual scenario of behavior recognition.

[0051] AsFigure 1 As shown, the hardware platform for motion detection in the above step (1) includes an intelligent vehicle platform, which is equipped with a target detection module, a data processing module, a driving device, and a camera. The target detection module is used to search for target animals, and the camera is used to record video data.

[0052] In this embodiment, the target detection module includes a lidar, and the data processing module includes a Raspberry Pi and an STM32, which cooperate with the camera to complete the above collection and record the video data of each target animal. Among them, the overall vehicle structure of the intelligent vehicle platform adopts an aluminum frame and an iron chassis, is equipped with four Mecanum wheels, and is independently driven by four DC motors and installed with encoders to measure the rotational speed.

[0053] Specifically, the video data of the camera in the above step (2) is sent to the remote host via the same wireless network segment; first, sparse sampling is performed, and the video is evenly divided into several segments, each segment having a duration of 250 milliseconds. A frame of image is randomly selected from each video segment as the animal behavior image data for this period. All the selected images are generated into sequence data in chronological order; each frame of image in the sequence is read, a time dimension is added and stitched into a tensor format, and the values of each dimension are normalized by z-score, as shown in Formula 1:

[0054] Among them

[0055] As Figure 2 shown, in this embodiment, the Y-TSM model structure is adopted in the above steps (3) and (4) for offline training, and its specific configuration is as follows:

[0056] First: Hardware environment:

[0057] CPU: Intel i7-7700K;

[0058] GPU: NVIDIA GTX 1080 Ti;

[0059] Memory: 16G;

[0060] Second: Software environment:

[0061] OS: Ubuntu 16.04;

[0062] Python: 3.7;

[0063] CUDA: 10.2.89;

[0064] cuDNN: 7.5.0;

[0065] Torch: 1.11.0;

[0066] As Figure 3 shown, specifically, the YOLOv5s model in step (3) above uses the parameters trained by the dataset to output the height, width, and coordinates of the target regression box. At the same time, all sampled image sequences are segmented according to the regression box, and a sub-image sequence with only a single target in the picture is output.

[0067] The specific experimental situation is as follows:

[0068] The dataset used to train YOLOv5s is from the sow monitoring videos taken at the breeding site, with a total of 900 images, divided into 520 training images, 80 validation images, and 300 test images. The training parameter epochs is 300 times, batch_size is 4, and the model is optimized using the mini-batch stochastic gradient descent method. It should be noted that the experimental results based on the above experiments show that the false detection rate of the object detection based on YOLOv5s is 0, and the missed detection rate is 0.67%, and the object detection effect is as Figure 3 shown.

[0069] Specifically, the improved overall model in step (4) above uses Resnet50 as the backbone network, and adds a TSM module and a CBAM attention module, including a convolutional layer, a max pooling layer, a first module, a second module, a third module, a fourth module, an average pooling layer, and a fully connected layer connected in series in sequence;

[0070] The convolutional layer uses 64 7×7 convolutional kernels with a stride of 2;

[0071] The max pooling layer uses a 3×3 pooling kernel with a stride of 2;

[0072] The first module uses 3 identical Bottleneck series structures, and each Bottleneck includes a TSM module, a 1×1×64 convolutional layer, a 3×3×64 convolutional layer, and a 1×1×256 convolutional layer connected in series in sequence;

[0073] The second module uses 4 identical Bottleneck series structures, and each Bottleneck includes a TSM module, a channel attention module, a spatial attention module, a 1×1×128 convolutional layer, a 3×3×128 convolutional layer, and a 1×1×512 convolutional layer connected in series in sequence;

[0074] The third module uses 6 identical Bottleneck series structures, and each Bottleneck includes a TSM module, a channel attention module, a spatial attention module, a 1×1×256 convolutional layer, a 3×3×256 convolutional layer, and a 1×1×1024 convolutional layer connected in series in sequence;

[0075] The fourth module uses three identical Bottleneck series structures. Each Bottleneck includes a TSM module, a 1×1×512 convolutional layer, a 3×3×512 convolutional layer, and a 1×1×2048 convolutional layer connected in series in sequence.

[0076] The average pooling layer uses a pooling kernel of 7×7.

[0077] The fully connected layer, in conjunction with the Softmax function, outputs the classification result.

[0078] As Figure 4 shown, the TSM module of the improved model in step (4) above contains two calculation paths. One is the main path, which directly sends the spatial feature map to the next convolutional network layer. The other is the branch path, which performs a shift operation on some channels in the time series, extracts features using 2D convolution, and finally sends them to the next layer in the form of a residual.

[0079] Among them, the CBAM attention module of the improved model includes a channel attention module and a spatial attention module connected in series in sequence.

[0080] As Figure 5 shown, the channel attention module sends the input feature map F(H×W×C) to the global max pooling layer and the global average pooling layer based on width and height respectively, obtaining two feature vectors of 1×1×C. The two feature vectors are respectively sent to a two-layer neural network MLP. The number of neurons in the first layer is C / r, where r is the reduction rate, the activation function is Relu, and the number of neurons in the second layer is C. The two feature vectors output by the MLP are concatenated by channel, and after passing through the sigmoid activation function, the final channel attention weight, namely M_c, is generated. Finally, a channel-based multiplication operation is performed on M_c and the input feature map F, and the output feature map F’ is obtained.

[0081] As Figure 6 shown, the spatial attention module takes the feature map F’ output by the channel attention module as the input. First, global max pooling and global average pooling based on channels are performed, obtaining two feature maps of H×W×1. These two feature maps are concatenated by channel, passed through a 7×7 convolutional layer, and reduced to 1 channel, that is, H×W×1. After passing through the sigmoid activation function, the spatial attention weight, namely M_s, is generated. Finally, M_s and the input feature of this module are multiplied according to the spatial position to obtain the finally generated feature.

[0082] Furthermore: For the overall improved model in step (4) above, the formula for calculating the loss function is shown in Equation 2: Among them, C is the total number of categories, y i represents the label, and G i represents the score of the video belonging to each category;

[0083]

[0084] In the above formula, G is the fusion function, which adopts a fully connected layer and a Softmax function. The input is the output after each video frame sampled on the time series passes through the backbone network, and the output is the probability that the entire video data is classified into each category.

[0085] The specific experimental process is as follows:

[0086] The dataset used to train the improved TSM model is sourced from the monitoring videos of sows taken at the breeding site. First, calculate the loss, and then use the mini-batch stochastic gradient descent method for optimization. For the original video data captured, the three-frame difference method is used for preliminary motion detection to screen and eliminate most of the sleeping behaviors. Calculate the difference image Dn' according to Equation 3, where fn+1(x, y), fn(x, y), and fn-1(x, y) are the gray values of the corresponding pixel points of the three frames of images. Then, through threshold processing, median filtering, dilation, and connectivity analysis, the discrimination result is output to determine whether there are moving targets in the picture.

[0087] D′ n (x, y) = |f n+1 (x, y) - f n (x, y)| ∩ |f n (x, y) - f n-1 (x, y)| (3)

[0088] Since the lighting in the farm uses lights and the brightness remains basically unchanged, a method with a fixed threshold is adopted. After testing, the threshold is set to 10, the aperture of the median filter is 3*3, the dilation kernel is 5*5, and the number of iterations is 2. Such a motion detection effect is the best, and it can filter out most of the videos of sows sleeping, as Figure 7 shown. Where a is the original image of the current frame and b is the frame difference image.

[0089] Construct datasets with different durations for single-segment videos and conduct comparative experiments to optimize the configuration of the sparse sampling frequency during actual application, which can improve the recognition accuracy of the present invention. The frame rate of the generated dataset is consistent with the original video and is set to 24. When the frame difference method detects a moving target, 200 consecutive frames of images are saved as a single-segment video, that is, each video has a duration of about 8 seconds; 50 consecutive frames of images are saved as a single-segment video, that is, each video has a duration of about 2 seconds. Sort out the pre-screened video data, divide the corresponding datasets according to the behaviors of the sows, and select six frequently occurring behaviors as the recognition targets: chewing behavior, masticating behavior, drinking behavior, eating behavior, exploring behavior, and sleeping behavior.

[0090] Divide the training set into 540 videos and the test set into 115 videos, with each video having a duration of 2 seconds, as shown in Table 1. Set the number of iterations to 50, the batch_size to 4, and the dropout ratio to 0.5. Adjust the image size, and fill in the black edges for images that do not meet the square shape; also use left - right flipping to expand the data volume and improve the richness of training samples. Set the initial learning rate to 0.001, which is set to 0.0001 after 10 iterations and 0.00001 after 20 iterations, and then remain unchanged to accelerate the training speed in the initial stage.

[0091] Table 1 Dataset

[0092]

[0093] And combine the obtained experimental data to draw the corresponding curve graphs, from which the accuracy curve of the training set as shown in Figure 8 is obtained, and the curve graph of the loss function value as shown in Figure 9 is obtained, where the light - colored curve is the true value and the dark - colored curve is the smoothed curve. The model basically reaches convergence after 20 iterations, and the accuracy is 95.54%.

[0094] Conduct a comparative experiment on the dataset with different single - video durations, and the results are shown in Table 2. Judging the behavior types in units of 2 seconds has the highest accuracy and is more suitable for setting as the video duration of the behavior data of sows.

[0095] Table 2 Comparative Experiment

[0096]

[0097] Specifically, insert the CBAM attention module into different blocks of the Resnet50 backbone network to achieve feature enhancement of action behaviors, conduct ablation experiments, and test the accuracy of behavior recognition to explore the best embedding method of the attention module. The results are shown in Table 3.

[0098] Table 3 Ablation Experiment

[0099]

[0100] Conduct verification experiments on the two models with the best performance in the ablation experiment on different datasets. Based on the experimental results in Table 4, it shows that using the attention mechanism in the 2nd and 3rd blocks has the best effect and the highest comprehensive accuracy.

[0101] Table 4 Verification Experiment

[0102]

[0103] Therefore, the animal behavior recognition method collects video data of the target animal by using the hardware platform for motion detection, and transmits the data to the remote host through the wireless network. Sparse sampling and preprocessing are carried out on the remote host to ensure the sufficiency of the corresponding features required for video acquisition. At the same time, the Y-TSM model is used to achieve behavior recognition, the YOLOv5s model is trained for target detection and segmentation, and the improved TSM model is trained for video classification and recognition.

[0104] Moreover, the attention mechanism based on the animal behavior recognition method reasonably allocates the weight ratio of each feature, and is equipped with the best embedding method of the corresponding CBAM attention module in the TSM model, so that the animal behavior recognition method can improve the accuracy of behavior recognition.

[0105] The above embodiments are only a relatively optimal technical solution of the present invention. Those skilled in the art should understand that without departing from the principle and essence of the present invention, modifications or replacements can be made to the technical solutions or parameters in the embodiments, and all should be covered within the protection scope of the present invention.

Claims

1. An animal behavior recognition method based on a spatio-temporal attention mechanism, characterized in that: Including the following steps: (1) Using a hardware platform with motion detection to collect and record the video data of each target animal; (2) The video data is wirelessly transmitted to a remote host. First, sparse sampling of the video image is performed, and the data is converted from a video stream to an image sequence, and then preprocessing such as tensor splicing and numerical normalization is carried out; (3) Training the YOLOv5s model, detecting the target and outputting the height, width and coordinates of the target regression box, segmenting the image, and outputting each sub-image sequence; (4) For a sub-image sequence data, using the TSM model based on the spatio-temporal attention mechanism for feature extraction and classification recognition, enhancing the temporal features with a channel attention module and enhancing the spatial features with a spatial attention module to allocate the weight ratios of each feature, and performing end-to-end training on the improved overall model to obtain the classification result of animal behavior features. The trained model is used for the actual scenario of behavior recognition; The improved overall model described in step (4) above uses Resnet50 as the backbone network, adding a TSM module and a CBAM attention module, including a convolutional layer, a max pooling layer, a first module, a second module, a third module, a fourth module, an average pooling layer, and a fully connected layer connected in series in turn; The convolutional layer uses 64 7×7 convolutional kernels with a stride of 2; The max pooling layer uses a 3×3 pooling kernel with a stride of 2; The first module uses 3 identical Bottleneck series structures, and each Bottleneck includes a TSM module, a 1×1×64 convolutional layer, a 3×3×64 convolutional layer, and a 1×1×256 convolutional layer connected in series in turn; The second module uses 4 identical Bottleneck series structures, and each Bottleneck includes a TSM module, a channel attention module, a spatial attention module, a 1×1×128 convolutional layer, a 3×3×128 convolutional layer, and a 1×1×512 convolutional layer connected in series in turn; The third module uses 6 identical Bottleneck series structures, and each Bottleneck includes a TSM module, a channel attention module, a spatial attention module, a 1×1×256 convolutional layer, a 3×3×256 convolutional layer, and a 1×1×1024 convolutional layer connected in series in turn; The fourth module uses 3 identical Bottleneck series structures, and each Bottleneck includes a TSM module, a 1×1×512 convolutional layer, a 3×3×512 convolutional layer, and a 1×1×2048 convolutional layer connected in series in turn; The average pooling layer uses a 7×7 pooling kernel; The fully connected layer cooperates with the Softmax function to output the classification result.

2. The animal behavior recognition method based on a spatio-temporal attention mechanism according to claim 1, wherein: The hardware platform for motion detection described in step (1) above includes an intelligent vehicle platform, and the intelligent vehicle platform is equipped with a target detection module, a data processing module, a driving device, and a camera. The target animal is searched through the target detection module, and the video data is recorded through the camera.

3. The animal behavior recognition method based on spatio-temporal attention mechanism according to claim 1, wherein: The video data described in step (2) above is sent to a remote host via the same wireless network segment, uniformly sampled to output a video frame sequence, each frame image is read, a time dimension is added and stitched into a tensor format, and z-score numerical normalization is performed on the values of each dimension.

4. The animal behavior recognition method based on the spatio-temporal attention mechanism according to claim 1, wherein: For the sparse sampling of the video images described in step (2) above, the video is evenly divided into several segments, each segment having a duration of a rated millisecond time period, and one frame image is randomly selected from each video segment as the animal behavior image data for that period. All the selected images generate sequence data in chronological order.

5. The animal behavior recognition method based on the spatio-temporal attention mechanism according to claim 1, wherein: The regression boxes described in step (3) above are the output results of the YOLOv5s model. According to the height, width, and coordinate information of the regression boxes, all the sampled image sequences are segmented, and a sub-image sequence with only a single target in the picture is output.

6. The animal behavior recognition method based on the spatio-temporal attention mechanism according to claim 1, wherein: The TSM module of the improved model in step (4) above includes two calculation paths. One is the main path, which directly sends the spatial feature map to the next convolutional network layer. The other is the branch path, which performs a shift operation on some channels in the time series, extracts features using 2D convolution, and finally sends them to the next layer in the form of a residual. The CBAM attention module of the improved model includes a channel attention module and a spatial attention module connected in series in sequence. The channel attention module sends the input feature map F(H×W×C) to the global max pooling layer and global average pooling layer based on width and height respectively, to obtain two 1×1×C feature vectors. The two feature vectors are respectively sent to a two-layer neural network MLP. The number of neurons in the first layer is C / r, where r is the reduction rate, the activation function is Relu, and the number of neurons in the second layer is C. The two feature vectors output by the MLP are concatenated by channel, and after passing through the sigmoid activation function, the final channel attention weight, i.e., M_c, is generated. Finally, a channel-based multiplication operation is performed on M_c and the input feature map F to output the feature map F'. The spatial attention module takes the feature map F' output by the channel attention module as the input: First, global max pooling and global average pooling based on channels are performed to obtain two H×W×1 feature maps. These two feature maps are concatenated by channel and passed through a 7×7 convolutional layer to reduce the dimension to 1 channel, i.e., H×W×1. After passing through the sigmoid activation function, the spatial attention weight, i.e., M_s, is generated. Finally, M_s and the input feature of this module are multiplied according to the spatial position to obtain the finally generated feature.

7. The animal behavior recognition method based on the spatio-temporal attention mechanism according to claim 5, wherein: For the improved overall model described in step (4) above, the formula for calculating the loss function is as follows: where C is the total number of categories, and y o represents the label, and G o represents the score of the video belonging to each category; In the above formula, G is the fusion function, which uses a fully connected layer and a Softmax function. The input is the output after the main network of each video frame sampled in the time series, and the output is the probability that the entire video data is classified into each category.

Images