Animal behavior recognition method fusing pose and rgb data
By utilizing the dual-stream input architecture and multi-scale attention adaptive module of the RPNet network, the problem of insufficient fusion of single-modal data and multi-scale features is solved, achieving efficient and robust animal behavior recognition.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- KUNMING UNIV OF SCI & TECH
- Filing Date
- 2025-09-12
- Publication Date
- 2026-07-24
AI Technical Summary
Existing animal behavior recognition methods rely on single-modal data, which makes it difficult to fully reflect the true state of animal behavior. They also lack multi-scale feature fusion and have huge computational overhead in multi-animal behavior recognition, which limits their practical application.
The RPNet network is designed with a dual-stream input architecture, including Pose-Path and RGB-Path. It combines ResNet3D+, the multi-scale attention adaptive module MASC, and an adaptive detection head to utilize multimodal information by fusing keypoint information and RGB features across layers.
It significantly improves the accuracy and robustness of animal behavior recognition, enabling it to better capture detailed action information, integrate semantic information, reduce computational overhead, and adapt to complex environments.
Smart Images
Figure CN121259908B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to a method for animal behavior recognition that integrates posture and RGB data, belonging to the field of computer vision. Background Technology
[0002] Animal behavior recognition has significant application value in agricultural animal health monitoring, especially for precision farming, animal behavior analysis, and health assessment. With the continuous development of artificial intelligence technology, video-based animal behavior recognition has become a research hotspot. Traditional animal behavior recognition methods often rely on single-modal information such as RGB images, optical flow, and audio signals, but these methods face numerous challenges in complex agricultural environments. In particular, the performance of existing methods is often limited under the influence of complex background environments, occlusion, lighting variations, and individual animal differences.
[0003] In recent years, action recognition methods based on pose estimation have attracted increasing attention due to their high robustness and accuracy. Skeleton keypoint data is extracted by a pose estimator and represented as a series of joint coordinate sequences. This representation can effectively capture animal motion information and avoid the interference of environmental changes on recognition accuracy. Common Graph Convolutional Network (GCN) methods recognize actions by treating joint coordinates as nodes, constructing a graph, and propagating information through graph convolution. Although GCNs perform well in processing skeleton sequences, they have poor coordinate perception, especially when applying different pose estimators, as the distribution of coordinates may change, leading to a significant decrease in recognition accuracy. Furthermore, GCN-based methods also suffer from interoperability and scalability issues. GCNs typically operate on irregular skeleton graphs, and these methods are difficult to effectively fuse with other modalities (such as RGB images, optical flow maps, etc.), especially in the early stages of the network. Moreover, since the complexity of GCNs is linearly related to the number of joints in an animal, the computational cost of GCNs is enormous in multi-animal behavior recognition tasks, limiting their widespread adoption in practical applications.
[0004] To overcome the aforementioned limitations, this invention designs an animal behavior recognition method that fuses pose and RGB data—RPNet. Unlike traditional GCN methods, RPNet consists of two paths: Pose-Path and RGB-Path, used to extract keypoint information and RGB video features, respectively. Furthermore, by fusing keypoint information and RGB features across layers, it fully utilizes multimodal information. Summary of the Invention
[0005] The purpose of this invention is to provide an animal behavior recognition method that integrates posture and RGB data, aiming to solve the technical problems of existing methods that rely on single-modal data, making it difficult to fully reflect the true behavioral state of animals and the lack of multi-scale feature fusion.
[0006] To achieve the above objectives, the technical solution of the present invention is: an animal behavior recognition method that integrates posture and RGB data, comprising the following steps:
[0007] Step 1: Construct the animal behavior recognition network RPNet, which integrates pose and RGB data; wherein, the RPNet adopts a dual-stream input architecture, including two paths, Pose-Path and RGB-Path, and combines ResNet3D+, the multi-scale attention adaptive module MASC, and an adaptive detection head;
[0008] Step 2: Input the video to be recognized for animal behavior into RPNet. Process the skeleton information in the video through Pose-Path and process the RGB video frames in the video through RGB-Path. In both Pose-Path and RGB-Path, use ResNet3D+ to extract spatiotemporal features to obtain Pose-Path spatiotemporal features and RGB-Path spatiotemporal features respectively.
[0009] Step 3: The Pose-Path spatiotemporal features and the RGB-Path spatiotemporal features are fused using the Multi-Scale Attention Adaptive Module (MASC) to obtain the fused Pose-Path enhanced spatiotemporal features and RGB-Path enhanced spatiotemporal features, respectively. The MASC introduces an attention mechanism to assign weights to features at different scales, thereby achieving feature fusion.
[0010] Step 4: Perform action recognition on the Pose-Path enhanced spatiotemporal features and the RGB-Path enhanced spatiotemporal features using the adaptive detection head to obtain action recognition results; wherein, the adaptive detection head includes two paths, one path for processing the Pose-Path enhanced spatiotemporal features and the other path for processing the RGB-Path enhanced spatiotemporal features.
[0011] Optionally, the process of processing the skeleton information in the video using the Pose-Path specifically involves:
[0012] First, a pose estimator is used to extract 2D pose data from the video, and a 3D heatmap is regenerated as the input for Pose-Path. A 2D pose is represented as a heatmap of size K×H×W, where K is the number of keypoints, H and W are the height and width of the frame, respectively. At the skeleton keypoints, only coordinate triplets are present. , ,c k In the case of ), a keypoint heatmap J is obtained by constructing K Gaussian plots centered on each keypoint, expressed as:
[0013]
[0014] In the formula, J kij This represents the response value of the k-th keypoint at position (i, j) in the heatmap; (i, j) represents the coordinate position of a pixel on the heatmap, where i is the row and j is the column. Controlling the variance of the Gaussian mapping; , ) and c k These represent the position and confidence score of the k-th joint, respectively.
[0015] Then, create a branch heatmap L with the following expression:
[0016]
[0017] In the formula, L kij This represents the heatmap value of the k-th branch at pixel position (i, j); a k b k Represents the key points at both ends of the k-th branch; D is the distance from the calculation point (i, j) to the line segment seg[a k ,b k The distance is a function of ];
[0018] Secondly, all key point heatmaps J and branch heatmaps L are superimposed along the time dimension to obtain a three-dimensional heatmap with a size of K×T×H×W, where T represents the time dimension and is the number of frames in the input video sequence.
[0019] Finally, the obtained 3D heatmap is optimized, specifically as follows:
[0020] Find the bounding boxes of the animals in all frames, which encompass all 2D pose information;
[0021] All frames are cropped according to the target bounding box and adjusted to a preset target size to reduce the size of the 3D heatmap;
[0022] The video is divided into several segments, and one frame is randomly selected from each segment to reduce redundancy in the 3D heatmap.
[0023] Optionally, processing the RGB video frames in the video using the RGB-Path specifically involves:
[0024] RGB-Path takes the original RGB frames of the video to be used for animal behavior recognition as input, uses ResNet3D+ to extract features, learns visual information from each frame, and transforms the visual information into high-dimensional spatiotemporal features.
[0025] Optionally, the ResNet3D+ is an improvement based on the traditional ResNet network by introducing nonlocal modules. The ResNet3D+ consists of convolutional layers, batch normalization layers, residual blocks, and nonlocal blocks.
[0026] Optionally, the multi-scale attention adaptive module (MASC) specifically comprises:
[0027] First, the input feature maps at different scales , and Dimensionality reduction is performed using a 1×1 dimensionality-reducing convolutional layer to obtain the fused feature map. The expression is:
[0028]
[0029] In the formula, the symbol || represents the feature map concatenation operation, and down indicates that the concatenated features are reduced in dimensionality by a 1×1 dimensionality reduction convolutional layer;
[0030] Then, a channel attention mechanism (CA) is introduced to process the fused feature map. We perform weighting to obtain the weighted feature map. The expression is:
[0031]
[0032] In the formula, CA(⋅) represents the channel attention mechanism;
[0033] Next, the weighted feature map Parallel multi-scale convolution operations are performed using 3×3, 5×5, and 7×7 convolution kernels to generate multi-scale fused feature maps. The expression is:
[0034]
[0035] In the formula, Conv 3×3 Conv 5×5 Conv 7×7 These represent convolutions of 3×3, 5×5, and 7×7, respectively.
[0036] Secondly, a spatial attention mechanism (SAM) is introduced to assign weights to the features at each spatial location, resulting in a weighted multi-scale fused feature map. The expression is:
[0037]
[0038] In the formula, SA(∙) represents the operation of spatial attention mechanism;
[0039] Finally, an upsampling convolutional layer is used to process the weighted multi-scale fused feature map. Perform an upsampling operation, the upsampling operation will Restore to the original spatial dimensions and output the final feature map. The expression is:
[0040]
[0041] In the formula, up(∙) represents performing an upsampling operation.
[0042] Optionally, the adaptive detection head specifically comprises:
[0043] First, the adaptive detection head includes two paths, one of which processes the input of the Pose-Path. Another path is used to process the input of the RGB-Path. Both paths pass through an adaptive pooling layer, and the size of the output feature map is automatically adjusted according to the size of the input. Let the size of the output of the adaptive pooling layer be defined as H1×W1×D1. Then, the pooled feature map is obtained after processing by the adaptive pooling layer. and The expression is:
[0044]
[0045]
[0046] In the formula, AdaptiveAvgPool3D(∙) represents pooling an input tensor of arbitrary size to a specified size H1×W1×D1, where D1 represents the depth of the output, and H1 and W1 represent the spatial dimensions;
[0047] Secondly, the feature map after pooling and Processing with a Dropout layer outputs a feature map. and Represented as:
[0048]
[0049]
[0050] In the formula, p is the probability of Dropout;
[0051] Then, the feature maps after the Dropout layer are transformed by a fully connected layer to finally obtain the classification score for each action, expressed as:
[0052]
[0053]
[0054] In the formula, A classification score is assigned to each action in the RGB-Path. Classify and score each action in the Pose-Path. and The fully connected layer weights for the RGB-Path and Pose-Path paths are respectively, b rgb and b pose These are learnable parameters, representing the biases of the RGB-Path and Pose-Path paths, respectively;
[0055] Finally, the classification scores of each action in the RGB-Path and Pose-Path are fused to obtain the final action recognition result.
[0056] Compared with the prior art, the beneficial effects of the present invention are:
[0057] (1) The present invention designs an animal behavior recognition RPNet that integrates pose and RGB data. The RPNet is divided into two paths, Pose-Path and RGB-Path, which are used to extract key point information and RGB video features respectively. At the same time, by fusing key point information and RGB features across layers, multimodal information is fully utilized.
[0058] (2) This invention designs a Multi-scale Attention Adaptive Fusion Module (MASC), which aims to solve the problem of lack of flexible weighting of features at different scales in traditional multi-scale fusion methods. The core of this module is the introduction of a learnable attention mechanism that can dynamically assign weights to convolutional features at different scales, thereby achieving refined fusion of multi-scale features. In this way, the MASC module can not only better capture low-level detailed information, but also effectively integrate high-level semantic information, significantly improving the adaptability of feature representation and task performance. Attached Figure Description
[0059] Figure 1 The original video clips collected for constructing the dataset of this invention;
[0060] Figure 2The video clips with classification labels obtained after data preprocessing for this invention;
[0061] Figure 3 Network structure diagram of the behavior recognition method that integrates RGB and pose data designed for this invention;
[0062] Figure 4 This is a structural diagram of the multi-scale attention adaptive fusion module of the present invention;
[0063] Figure 5 This is a structural diagram of the adaptive detection head of the present invention;
[0064] Figure 6 This is a structural diagram of the ResNet3D+ module of the present invention. Detailed Implementation
[0065] The present invention will be further described below with reference to the accompanying drawings and specific embodiments.
[0066] Example 1: As Figure 3 As shown, an animal behavior recognition method that integrates pose and RGB data includes the following steps:
[0067] Step 1: Construct the animal behavior recognition network RPNet, which integrates pose and RGB data; wherein, the RPNet adopts a dual-stream input architecture, including two paths, Pose-Path and RGB-Path, and combines ResNet3D+, the multi-scale attention adaptive module MASC, and an adaptive detection head;
[0068] Step 2: Input the video to be recognized for animal behavior into RPNet. Process the skeleton information in the video through Pose-Path and process the RGB video frames in the video through RGB-Path. In both Pose-Path and RGB-Path, use ResNet3D+ to extract spatiotemporal features to obtain Pose-Path spatiotemporal features and RGB-Path spatiotemporal features respectively.
[0069] Optionally, the videos to be used for animal behavior recognition are derived from a collected dataset containing 598 video clips. These clips cover various typical animal behaviors (cows and horses), such as lameness, walking, standing, and running, and possess rich spatiotemporal information, comprehensively reflecting the dynamic changes of animals in different situations. The data is primarily collected from internet platforms to obtain videos of animal (horse and cow) behavior in real-world environments. Figure 1 As shown.
[0070] Optionally, the data in the dataset may be preprocessed, including:
[0071] By using a YOLOv8 object detection model pre-trained on large-scale public datasets such as COCO, the huge computational overhead of retraining the model was avoided, ensuring the model's generalization performance and detection stability. Secondly, rigorous image preprocessing was performed on the original video frames, including uniformly adjusting the images to a standard size of 640×640 pixels and padding non-square images to ensure the input data format met the model requirements. Simultaneously, accurate target bounding box data was prepared for model training using manual and semi-automatic annotation methods.
[0072] In the target detection stage, the YOLOv8 model processes video frames to generate multiple candidate bounding boxes containing the target category and its corresponding spatial location. Frames that do not detect any target animal are directly excluded; when multiple bounding boxes appear, redundant or low-quality candidate results are filtered out by analyzing the overlapping area between the bounding boxes and the detection confidence (threshold set to 0.9). Finally, the selected high-quality video frames are uniformly stored in a designated directory, providing reliable data support for subsequent pose recognition and behavior analysis.
[0073] To further improve data processing efficiency and ensure the consistency of training data, this embodiment utilizes a Python script combined with the moviepy library to implement automatic video data cropping. The script automatically traverses all video files in a specified directory and divides the video into multiple segments according to a set time window (5 seconds). After this automated process, video data segments with uniform duration are obtained. The cow behavior video segments include 67 instances of lameness, 8 instances of running, 85 instances of standing, and 54 instances of walking; the horse behavior video segments include 87 instances of lameness, 57 instances of running, 35 instances of standing, and 205 instances of walking. Figure 2 As shown.
[0074] Optionally, the process of processing the skeleton information in the video using the Pose-Path specifically involves:
[0075] First, a pose estimator is used to extract 2D pose data from the video, and a 3D heatmap is regenerated as the input for Pose-Path. A 2D pose is represented as a heatmap of size K×H×W, where K is the number of keypoints, H and W are the height and width of the frame, respectively. At the skeleton keypoints, only coordinate triplets are present. , ,c k In the case of ), a keypoint heatmap J is obtained by constructing K Gaussian plots centered on each keypoint, expressed as:
[0076]
[0077] In the formula, J kijThis represents the response value of the k-th keypoint at position (i, j) in the heatmap; (i, j) represents the coordinate position of a pixel on the heatmap, where i is the row and j is the column. Controlling the variance of the Gaussian mapping; , ) and c k These represent the position and confidence score of the k-th joint, respectively.
[0078]
[0079] In the formula, L kij This represents the heatmap value of the k-th branch at pixel position (i, j); a k b k Represents the key points at both ends of the k-th branch; D is the distance from the calculation point (i, j) to the line segment seg[a k ,b k The distance is a function of ];
[0080] Secondly, all key point heatmaps J and branch heatmaps L are superimposed along the time dimension to obtain a three-dimensional heatmap with a size of K×T×H×W, where T represents the time dimension and is the number of frames in the input video sequence.
[0081] Finally, the obtained 3D heatmap is optimized, specifically as follows:
[0082] Find the bounding boxes of the animals in all frames, which encompass all 2D pose information;
[0083] All frames are cropped according to the target bounding box and adjusted to the preset target size to reduce the size of the 3D heat map and improve computational efficiency.
[0084] The video is evenly divided into several segments, and one frame is randomly selected from each segment to reduce the redundancy of the 3D heatmap and thus maintain the global dynamism of the video.
[0085] Optionally, processing the RGB video frames in the video using the RGB-Path specifically involves:
[0086] RGB-Path takes the original RGB frames of the video to be used for animal behavior recognition as input, extracts features using ResNet3D+, learns visual information from each frame, and transforms the visual information into high-dimensional spatiotemporal features. In this way, RGB-Path can capture dynamic visual information in the video, including the animal's appearance, movements, and environmental changes.
[0087] Optionally, ResNet3D+ is an improvement on the traditional ResNet network by introducing nonlocal modules. ResNet3D+ consists of convolutional layers, batch normalization layers, residual blocks, and nonlocal blocks, such as... Figure 6 As shown.
[0088] Specifically, the ResNet3D+ network architecture has four residual groups (Residual group 1-4), and each residual group consists of two residual blocks, as shown below. Figure 6 As shown. The network begins with convolutional layers, batch normalization layers, and ReLU, followed by four residual groups. The first convolutional layer has a time step of 1 to avoid prematurely sampling temporal information. Each residual group consists of two residual blocks, as shown. Figure 6 As shown. The residual blocks contain skip connections and element-wise addition operations. Batch normalization is applied after all convolutional layers. For residual groups 2 through 4, both the temporal and spatial strides are set to 2 in the first convolutional layer to perform downsampling. The residual groups are followed by an average pooling layer, a flattening layer, and a fully connected layer.
[0089] Understandably, in animal behavior recognition tasks, traditional convolutional neural networks (CNNs) often fail to effectively process spatiotemporal features in videos. ResNet3D, by introducing a temporal dimension into convolutional operations, can better capture temporal dependencies between video frames. Therefore, this embodiment designs ResNet3D+ as a feature extraction module to extract spatiotemporal features of animal behavior. Furthermore, the overall architecture of ResNet3D+ follows standard ResNet design principles, namely, avoiding the vanishing gradient problem in deep networks through residual connections.
[0090] It is understood that the RPNet in this embodiment adopts a dual-stream input architecture to process skeleton information and RGB video frames separately, enabling RPNet to extract key features from different modalities and fully utilize the complementary advantages of pose estimation and visual information.
[0091] Step 3: The Pose-Path spatiotemporal features and the RGB-Path spatiotemporal features are fused using the Multi-Scale Attention Adaptive Module (MASC) to obtain the fused Pose-Path enhanced spatiotemporal features and RGB-Path enhanced spatiotemporal features, respectively. The MASC introduces an attention mechanism to assign weights to features at different scales, thereby achieving feature fusion.
[0092] Optionally, the multi-scale attention adaptive module (MASC) specifically comprises:
[0093] First, in order to reduce computational cost and improve the efficiency of feature fusion, feature maps of different scales are processed. , and Dimensionality reduction is achieved by using a 1×1 dimensionality-reducing convolutional layer to reduce the number of channels for each scale feature, resulting in a fused feature map. The expression is:
[0094]
[0095] In the formula, the symbol || represents the feature map concatenation operation, and down indicates that the concatenated features are reduced in dimensionality by a 1×1 dimensionality reduction convolutional layer;
[0096] Then, the feature map after dimensionality reduction It still contains information from different scales, but the importance of its channels may vary significantly. To enhance the expressive power of key channels and suppress redundant information, a channel attention mechanism (CA) is introduced. This mechanism learns the weight distribution between channels to improve the expressive power of the fused feature map. We perform weighting to obtain the weighted feature map. The expression is:
[0097]
[0098] In the formula, CA(⋅) represents the channel attention mechanism, which adjusts the importance of channels according to the weight of each channel.
[0099] Next, in order to further extract multi-scale features, the weighted feature map is processed. Parallel use of 3×3, 5×5, and 7×7 convolutional kernels for multi-scale convolution operations enables the capture of local features at different scales, generating multi-scale fused feature maps. The expression is:
[0100]
[0101] In the formula, Conv 3×3 Conv 5×5 Conv 7×7 These represent convolutions of 3×3, 5×5, and 7×7, respectively.
[0102] Understandably, multi-scale convolution operations can effectively capture information about objects at different scales in the input image, enhancing the model's adaptability to complex scenes.
[0103] Secondly, the results of multi-scale convolution operations While incorporating features from different scales, some redundant information or excessive focus on background regions may still exist. Therefore, a spatial attention mechanism (SAM) is introduced to assign weights to features at each spatial location, resulting in a weighted multi-scale fused feature map. The expression is:
[0104]
[0105] In the formula, SA(∙) represents the operation of spatial attention mechanism;
[0106] Understandably, the spatial attention mechanism weights the feature map according to the importance of each spatial location, thereby enabling the model to focus more on key regions in the image and suppress the influence of irrelevant regions.
[0107] Finally, an upsampling convolutional layer is used to process the weighted multi-scale fused feature map. Perform an upsampling operation, the upsampling operation will Restore to the original spatial dimensions and output the final feature map. The expression is:
[0108]
[0109] In the formula, up(∙) represents performing an upsampling operation, resulting in the final feature map. It is an optimized feature that integrates features from different scales and is weighted by channel and spatial attention.
[0110] Understandably, both the Pose-Path and RGB-Path use ResNet3D+ to extract spatiotemporal features, capturing spatial information through spatial convolution and dynamic changes through temporal convolution. Each path uses max pooling to reduce spatial dimensionality, thereby alleviating computational burden and preserving the most important spatiotemporal features. However, using RGB video or 2D pose information alone is insufficient to fully capture subtle changes in motion. Therefore, this embodiment designs a Multi-Scale Attention Adaptive Fusion (MASC) module for cross-layer feature fusion. The core of the MASC module lies in introducing a learnable attention mechanism that dynamically assigns weights to convolutional features at different scales, thereby achieving refined fusion of multi-scale features, such as... Figure 4 As shown. In this way, the MASC module can not only better capture low-level detailed information, but also effectively integrate high-level semantic information, significantly improving the adaptability of feature representation and task performance.
[0111] Step 4: Perform action recognition on the Pose-Path enhanced spatiotemporal features and the RGB-Path enhanced spatiotemporal features using the adaptive detection head to obtain action recognition results; wherein, the adaptive detection head includes two paths, one path for processing the Pose-Path enhanced spatiotemporal features and the other path for processing the RGB-Path enhanced spatiotemporal features.
[0112] Optionally, the adaptive detection head specifically comprises:
[0113] First, the adaptive detection head includes two paths, one of which processes the input of the Pose-Path. Another path is used to process the input of the RGB-Path. ,like Figure 5 As shown, both paths undergo adaptive pooling layer operations. The size of the output feature map is automatically adjusted according to the size of the input, reducing dimensionality while preserving spatial information. Let the size of the output of the adaptive pooling layer be defined as H1×W1×D1. Then, the pooled feature map is obtained after processing by the adaptive pooling layer. and The expression is:
[0114]
[0115]
[0116] In the formula, AdaptiveAvgPool3D(∙) represents pooling an input tensor of arbitrary size to a specified size H1×W1×D1, where D1 represents the depth of the output, and H1 and W1 represent the spatial dimensions;
[0117] Secondly, to prevent overfitting, a Dropout layer is added to randomly discard feature maps, thereby enhancing the model's generalization ability. (Feature map after pooling) and Processing with a Dropout layer outputs a feature map. and Represented as:
[0118]
[0119]
[0120] In the formula, p is the probability of Dropout;
[0121] Then, the feature maps after the Dropout layer are transformed by a fully connected layer to finally obtain the classification score for each action, expressed as:
[0122]
[0123]
[0124] In the formula, A classification score is assigned to each action in the RGB-Path. Classify and score each action in the Pose-Path. and The fully connected layer weights for the RGB-Path and Pose-Path paths are respectively, b rgb and b pose These are learnable parameters, representing the biases of the RGB-Path and Pose-Path paths, respectively;
[0125] Finally, the classification scores of each action in the RGB-Path and Pose-Path are fused to obtain the final action recognition result.
[0126] It is understandable that the adaptive detection head designed in this embodiment generates the final action classification result by utilizing features obtained from the Pose path and RGB path, combined with an adaptive mechanism. It can dynamically adjust computing resources and focus on the most critical areas, thereby more accurately identifying different actions.
[0127] In summary, the RPNet designed in this invention employs a two-stream structure to process skeleton keypoint data and RGB data separately, achieving comprehensive capture of animal spatiotemporal behavioral features through complementary advantages. To enhance the model's ability in spatiotemporal feature extraction, an improved ResNet3D+ module is introduced into the backbone network. Addressing the inconsistencies and local feature loss issues across different modalities at multiple scales, this invention designs a multi-scale attention adaptive fusion module (MASC). This module dynamically adjusts the weights of features at each scale, achieving efficient fusion of key information from the two paths, thereby significantly improving the overall accuracy and robustness of behavior recognition.
[0128] The specific embodiments of the present invention have been described in detail above with reference to the accompanying drawings. However, the present invention is not limited to the above embodiments. Within the scope of knowledge possessed by those skilled in the art, various changes can be made without departing from the spirit of the present invention.
Claims
1. A method for animal behavior recognition that integrates posture and RGB data, characterized in that, The method includes the following steps: Step 1: Construct the animal behavior recognition network RPNet, which integrates pose and RGB data; wherein, the RPNet adopts a dual-stream input architecture, including two paths, Pose-Path and RGB-Path, and combines ResNet3D+, the multi-scale attention adaptive module MASC, and an adaptive detection head; Step 2: Input the video to be recognized for animal behavior into RPNet. Process the skeleton information in the video through Pose-Path and process the RGB video frames in the video through RGB-Path. In both Pose-Path and RGB-Path, use ResNet3D+ to extract spatiotemporal features to obtain Pose-Path spatiotemporal features and RGB-Path spatiotemporal features respectively. Step 3: The Pose-Path spatiotemporal features and the RGB-Path spatiotemporal features are fused using the Multi-Scale Attention Adaptive Module (MASC) to obtain the fused Pose-Path enhanced spatiotemporal features and RGB-Path enhanced spatiotemporal features, respectively. The MASC introduces an attention mechanism to assign weights to features at different scales, thereby achieving feature fusion. Step 4: Perform action recognition on the Pose-Path enhanced spatiotemporal features and the RGB-Path enhanced spatiotemporal features using the adaptive detection head to obtain action recognition results; wherein, the adaptive detection head includes two paths, one path for processing the Pose-Path enhanced spatiotemporal features and the other path for processing the RGB-Path enhanced spatiotemporal features; The multi-scale attention adaptive module MASC is specifically as follows: First, the input feature maps at different scales , and Dimensionality reduction is performed using a 1×1 dimensionality-reducing convolutional layer to obtain the fused feature map. The expression is: ； In the formula, the symbol || represents the feature map concatenation operation, and down indicates that the concatenated features are reduced in dimensionality by a 1×1 dimensionality reduction convolutional layer; Then, a channel attention mechanism (CA) is introduced to process the fused feature map. We perform weighting to obtain the weighted feature map. The expression is: ； In the formula, CA( ) represents the channel attention mechanism; Next, the weighted feature map Parallel multi-scale convolution operations are performed using 3×3, 5×5, and 7×7 convolution kernels to generate multi-scale fused feature maps. The expression is: ； In the formula, Conv 3×3 Conv 5×5 Conv 7×7 These represent convolutions of 3×3, 5×5, and 7×7, respectively. Secondly, a spatial attention mechanism (SAM) is introduced to assign weights to the features at each spatial location, resulting in a weighted multi-scale fused feature map. The expression is: ； In the formula SA( This indicates the execution of spatial attention mechanisms. Finally, an upsampling convolutional layer is used to process the weighted multi-scale fused feature map. Perform an upsampling operation, the upsampling operation will Restore to the original spatial dimensions and output the final feature map. The expression is: ； In the formula, up( ) indicates that an upsampling operation is performed.
2. The animal behavior recognition method that integrates posture and RGB data according to claim 1, characterized in that, The specific steps of processing the skeleton information in the video using the Pose-Path are as follows: First, a pose estimator is used to extract 2D pose data from the video, and a 3D heatmap is regenerated as the input for Pose-Path. A 2D pose is represented as a heatmap of size K×H×W, where K is the number of keypoints, H and W are the height and width of the frame, respectively. At the skeleton keypoints, only coordinate triplets are present. , ,c k In the case of ), a keypoint heatmap J is obtained by constructing K Gaussian plots centered on each keypoint, expressed as: ； In the formula, J kij This represents the response value of the k-th key point at position (i, j) in the heatmap; (i, j) represents the coordinates of a pixel on the heatmap, where i is the row and j is the column; Controlling the variance of the Gaussian mapping; , ) and c k These represent the position and confidence score of the k-th joint, respectively. Then, create a branch heatmap L with the following expression: ； In the formula, L kij This represents the heatmap value of the k-th branch at pixel position (i, j); a k b k Represents the key points at both ends of the k-th branch; D is the distance from the calculation point (i, j) to the line segment seg[a k ,b k The distance is a function of ]; Secondly, all key point heatmaps J and branch heatmaps L are superimposed along the time dimension to obtain a three-dimensional heatmap with a size of K×T×H×W, where T represents the time dimension and is the number of frames in the input video sequence. Finally, the obtained 3D heatmap is optimized, specifically as follows: Find the bounding boxes of the animals in all frames, which encompass all 2D pose information; All frames are cropped according to the target bounding box and adjusted to a preset target size to reduce the size of the 3D heatmap; The video is divided into several segments, and one frame is randomly selected from each segment to reduce redundancy in the 3D heatmap.
3. The animal behavior recognition method that integrates posture and RGB data according to claim 1, characterized in that, The specific steps of processing RGB video frames in the video using the RGB-Path are as follows: RGB-Path takes the original RGB frames of the video to be used for animal behavior recognition as input, uses ResNet3D+ to extract features, learns visual information from each frame, and transforms the visual information into high-dimensional spatiotemporal features.
4. The animal behavior recognition method that integrates posture and RGB data according to claim 3, characterized in that, The ResNet3D+ is an improvement on the traditional ResNet network by introducing nonlocal modules. The ResNet3D+ consists of convolutional layers, batch normalization layers, residual blocks, and nonlocal blocks.
5. The animal behavior recognition method based on the fusion of posture and RGB data according to claim 1, characterized in that, The adaptive detection head is specifically: First, the adaptive detection head includes two paths, one of which processes the input of the Pose-Path. Another path is used to process the input of the RGB-Path. Both paths pass through an adaptive pooling layer, and the size of the output feature map is automatically adjusted according to the size of the input. Let the size of the output of the adaptive pooling layer be defined as H1×W1×D1. Then, the pooled feature map is obtained after processing by the adaptive pooling layer. and The expression is: ； ； In the formula, AdaptiveAvgPool3D(∙) represents pooling an input tensor of arbitrary size to a specified size H1×W1×D1, where D1 represents the depth of the output, and H1 and W1 represent the spatial dimensions; Secondly, the feature map after pooling and Processing with a Dropout layer outputs a feature map. and Represented as: ； ； In the formula, p is the probability of Dropout; Then, the feature maps after the Dropout layer are transformed by a fully connected layer to finally obtain the classification score for each action, expressed as: ； ； In the formula, A classification score is assigned to each action in the RGB-Path. Classify and score each action in the Pose-Path. and The fully connected layer weights for the RGB-Path and Pose-Path paths are respectively, b rgb and b pose These are learnable parameters, representing the biases of the RGB-Path and Pose-Path paths, respectively; Finally, the classification scores of each action in the RGB-Path and Pose-Path are fused to obtain the final action recognition result.