Animal behavior recognition method based on pose-transformer network

Abstract

The application discloses an animal behavior recognition method based on a Pose-Transformer network, which comprises the following steps: first, collecting an animal behavior video and pre-processing the animal behavior video; then, constructing a Pose-Transformer network to recognize the animal behavior in the video. The network construction steps comprise the following steps: 1) obtaining the position of a joint of an animal body part, generating joint coordinates and positions of the animal; for a skeleton sequence of a T0 frame of the input video, the skeleton sequence is divided into a posture block sequence; 2) in the Transformer network, the posture block obtained in the step 1) is subjected to space-time Transformer Encoder, wherein: the time sequence Transformer Encoder and the space Transformer Encoder respectively give corresponding weights to skeleton joints of different posture blocks according to the different correlation intensity between joints corresponding to different behaviors of the animal in time sequence and space, so as to represent the time-space correlation degree between joints of different behaviors; 3) the time sequence information of the posture block sequence obtained in the step 2) is modeled through a posture block aggregation layer, and then the behavior classification result of the animal is obtained through global average pooling and a full connection layer.

Description

Technical Field

[0001] This invention relates to the field of computer vision, specifically to an animal behavior recognition method based on a Pose-Transformer network. Background Technology

[0002] Species diversity is closely related to human life, and the development of biological populations has a vital impact on human life. Automatically identifying the behavior of animals in zoos and nature reserves can help managers to rationally manage animal diets, understand animal emotions and physical conditions, and thus provide strong technical support for scientific decision-making in the protection of endangered animals.

[0003] Early animal protection relied primarily on manual monitoring and handwritten records. Later, various sensors were placed on animals' bodies to monitor abnormal behavior. However, the act of wearing sensors itself can be disruptive to the animals, causing them to exhibit unusual behaviors. With the development of video surveillance technology, the application of surveillance videos to automatically identify animal behavior has attracted increasing attention from researchers. However, it is still in the exploratory stage, and research on animal behavior recognition using image processing techniques is still lacking.

[0004] Animal behavior is often a continuous process, difficult to capture in a single image. Skeleton-based animal behavior recognition methods only need to identify behavior based on the skeletal joint information, effectively avoiding the influence of background and complex environments, thus improving the accuracy of animal behavior recognition. Different animal behaviors are not only related to the information of individual joints in the animal skeleton, but also to the spatial correlation between different joints within the same skeleton, and the temporal correlation between joints in the same location across different frames in a skeleton sequence. Therefore, extracting the spatiotemporal correlation features of animal behavior is of great significance for improving the performance of animal recognition methods. Summary of the Invention

[0005] Studies have shown that joints in animal movement are interconnected in both time and space, and extracting the spatiotemporal correlation features of animal behavior is crucial for improving the performance of animal recognition methods. However, existing animal behavior recognition methods cannot effectively capture the interconnectedness information of joints between frames in animal behavior videos, leading to an inability to distinguish between similar behaviors. This is especially true when a behavior consists of multiple sub-actions, which can easily result in misidentification.

[0006] Based on the above reasons, this invention constructs a Pose-Transformer deep neural network model and applies it to animal behavior recognition;

[0007] This invention divides the skeleton sequence into a pose block sequence and constructs a Transformer-based spatiotemporal encoder for the pose blocks to capture the spatiotemporal correlation features between animal body joints.

[0008] This invention uses an aggregation operation to obtain the temporal relationship between the various sub-actions of a behavior by the spatiotemporal correlation features of the posture block, thereby distinguishing similar animal behaviors.

[0009] The specific steps of this invention include:

[0010] 1) Obtain the positions of the joints of the animal's body parts, and generate the animal's joint coordinates and confidence (Deeplabcut can be used); for the skeleton sequence of the input T0 frame, divide it into a pose block sequence;

[0011] 2) Perform a spatiotemporal Transformer Encoder on the pose blocks obtained in step 1). The temporal Transformer Encoder and the spatial Transformer Encoder assign different weights to the skeleton joints of different pose blocks in the temporal and spatial directions, respectively, based on the different correlation strengths between the joints corresponding to different behaviors, in order to characterize the spatiotemporal correlation degree between the joints of different behaviors.

[0012] 3) The temporal information of the pose block sequence obtained in step 2) is modeled through a pose block aggregation layer, and the animal behavior classification results are obtained through global average pooling and fully connected layers.

[0013] In step 2),

[0014] 2.1) Add a position encoder after the Embedding layer to convert the joint information of the animal skeleton sequence of each pose block into a Transformer vector representation. The generated vectors include a query vector Q, a key vector K, and a value vector V.

[0015] 2.2) Obtain the spatial correlation between joints. Input the vector obtained in step 2.1) into the Spatial Transformer Encoder layer. The Spatial Transformer Encoder layer consists of N stacked Spatial Transformer Encoder blocks to obtain the spatial correlation features of joints for each animal skeleton in each pose block;

[0016] 2.3) Obtain the temporal correlation between joints. Input the vector obtained in step 2.2) into the temporal Transformer Encoder layer. The temporal Transformer Encoder layer consists of N stacked temporal Transformer Encoder blocks to obtain the temporal correlation features of the joints of the animal skeleton sequence for each pose block;

[0017] In step 3)

[0018] 3.1) After performing the Concate operation on the spatial correlation features and temporal correlation features obtained in steps 2.2) and 2.3), the temporal correlation of continuous posture block sequences is analyzed by posture block aggregation operation, that is, the correlation between posture blocks and postures. Postures with high correlation are aggregated, which helps to distinguish two different animal behaviors containing similar postures.

[0019] 3.2) Analyze the temporal correlation of consecutive attitude block sequences, i.e., the correlation between attitude blocks and corresponding attitudes, through attitude block aggregation operations;

[0020] 3.3) Based on the correlation of the pose blocks obtained in step 3.2), the animal behavior classification results are obtained through global average pooling and fully connected layers.

[0021] The beneficial effects of this invention are as follows:

[0022] (1) This invention proposes an animal behavior recognition method based on a Pose-Transformer deep neural network model. The main steps include: dividing the video skeleton sequence into a pose block sequence, spatiotemporal Transformer encoding of the pose blocks, and aggregation operation of the spatiotemporal features of the pose blocks, which improves the accuracy of animal behavior recognition.

[0023] (2) The present invention constructs a Transformer-based spatiotemporal encoder for the attitude block, wherein: the spatial Transformer encoder extracts the correlation features between related joints in space, and the temporal Transformer encoder extracts the correlation features between related joints in time, which are used to obtain the spatiotemporal linkage features between joints of animal behavior.

[0024] (3) The present invention obtains the temporal relationship between the various sub-actions of a behavior by aggregating the spatiotemporal correlation features of the posture block, which can better distinguish different animal behaviors containing similar sub-actions. Attached Figure Description

[0025] Figure 1 This is a structural diagram of the animal behavior recognition network of the present invention;

[0026] Figure 2 A schematic diagram illustrating the method for calculating the association weights between joints in a Spatial Transformer Encoder;

[0027] Figure 3 A schematic diagram illustrating the method for calculating the association weights between key points in a Temporal Transformer Encoder;

[0028] Figure 4 The experimental results of the accuracy of animal behavior recognition are shown in the figure, which shows the different numbers of frames n in a pose block.

[0029] Figure 5 The experimental results are shown as the number of iterations versus accuracy.

[0030] Figure 6 The figure shows the experimental results for the number of iterations versus the loss function. Detailed Implementation

[0031] To address the shortcomings of existing methods in identifying the temporal correlation between keypoints in different frames of a skeleton sequence and in distinguishing different animal behaviors containing similar sub-actions, this invention proposes an animal behavior recognition method based on a Pose-Transformer deep neural network model. The network structure is as follows: Figure 1 As shown, the steps include:

[0032] 1) Use Deeplabcut to obtain the positions of the joints of the animal's body parts, generate the joint coordinates and confidence of the animal, and divide the skeleton sequence of the input T0 frame into a pose block sequence.

[0033] For the skeleton sequence of input frame T0, each frame's skeleton sequence contains k0 key points. This is divided into a sequence of pose blocks, with n consecutive frames forming one pose block. Therefore, each pose block has n × k0 key points, and the number of pose blocks is...

[0034]

[0035] 2) Perform a spatiotemporal Transformer Encoder on the pose blocks obtained in step 1). The temporal Transformer Encoder and the spatial Transformer Encoder assign different weights to the skeleton joints of different pose blocks in the temporal and spatial directions, respectively, based on the different correlation strengths between the joints corresponding to different behaviors, in order to characterize the spatiotemporal correlation degree between the joints of different behaviors.

[0036] 2.1) Add a position encoder after the Embedding layer to convert the joint information of the animal skeleton sequence of each pose block into a Transformer vector representation. The generated vectors include a query vector Q, a key vector K, and a value vector V.

[0037] 2.2) Obtain the spatial correlation between joints. Input the vector obtained in step 2.1) into the Spatial Transformer Encoder layer. The Spatial Transformer Encoder layer consists of N stacked Spatial Transformer Encoder blocks to obtain the spatial correlation features of each animal skeleton joint for each pose block. The steps include:

[0038] 2.2.1) The animal skeleton is divided into 5 different parts, namely the spine and 4 legs. Since there are connections between the joints of each part, the connections between the joints of different body parts are small. In order to reduce the amount of computation, this invention calculates the spatial correlation between the joints of each part, such as the spatial correlation of the 3 joints of the animal leg.

[0039] 2.2.2) Construct a spatial self-attention mechanism, using the spatial self-attention module to calculate the spatial correlation between the joints of each part divided in step 2.2.1), and assign different weights to the skeleton joints. The correlation weight calculation formula of the self-attention mechanism is as shown in equation (1):

[0040]

[0041] Where, q i =Q(i), k i =K(i), v i =V(i), Q, K, V are the query vector, key vector, and value vector of the Transformer obtained in step 2.1), respectively, and their inner products are... (Here, T stands for "transpose").

[0042] like Figure 2 As shown, Formula (1) calculates the spatial association weight between the i-th node of an animal skeleton in a frame and the other joints in the same part of the same animal skeleton.

[0043] 2.2.3) The self-attention module in step 2.2.2) is followed by a feedforward network layer and a normalization layer, which together form the Spatial Transformer Encoder block;

[0044] 2.2.4) A spatial Transformer Encoder layer is formed by stacking N Spatial Transformer Encoder blocks to obtain the spatial association features of the joints of each animal skeleton in each pose block;

[0045] 2.3) Obtain the temporal correlation between joints. Input the vector obtained in step 2.1) into the temporal Transformer Encoder layer. The temporal Transformer Encoder layer consists of N stacked temporal Transformer Encoder blocks to obtain the temporal correlation features of the joints of the animal skeleton sequence for each pose block. The steps include:

[0046] 2.3.1) Temporal self-attention mechanism for key points in animal skeleton sequences that construct pose blocks;

[0047] 2.3.2) Within a pose block, a temporal self-attention module is used to calculate the temporal correlation between a keypoint in a frame of the pose block and keypoints at the same position in other frames of the pose block, assigning different weights to the skeleton keypoints. The correlation weight calculation formula for the self-attention mechanism is shown in equation (2):

[0048]

[0049] Where, q i =Q(i), k i =K(i), v i =V(i), Q, K, V are the query vector, key vector, and value vector of the Transformer obtained in step 2.1), respectively, and their inner products are... (Here, T stands for "transpose").

[0050] like Figure 3 As shown, formula (2) calculates the temporal association weight between the i-th node of the animal skeleton in a certain frame of a pose block and the i-th node of the skeleton in the remaining frames of the pose block.

[0051] 2.3.3) The self-attention module in step 2.3.1) is followed by a feedforward network layer and a normalization layer, which together form the Temporal Transformer Encoder block;

[0052] 2.3.4) A temporal Transformer Encoder layer is formed by stacking N Temporal Transformer Encoder blocks to obtain the temporal association features of the joints of the skeleton sequence of each pose block;

[0053] 3) The temporal information of the pose block sequence obtained in step 2) is modeled through a pose block aggregation layer, and the animal behavior classification results are obtained through global average pooling and fully connected layers.

[0054] 3.1) After performing the Concate operation on the spatial correlation features and temporal correlation features obtained in steps 2.2) and 2.3), the temporal correlation of continuous posture block sequences is analyzed by posture block aggregation operation, that is, the correlation between posture blocks and postures. Postures with high correlation are aggregated, which helps to distinguish two different animal behaviors containing similar postures.

[0055] 3.2) Analyze the temporal correlation of consecutive attitude block sequences, i.e., the correlation between attitude blocks and corresponding attitudes, through attitude block aggregation operations. The steps include:

[0056] 3.2.1) Calculate the output of each attention head and perform a Concat operation on the output of each attention head, as shown in formula (3);

[0057]

[0058] in, (1≤i≤h) represents the output of the i-th attention head.

[0059] 3.2.2) Apply formula (4) to perform a convolution operation on XAttn obtained in step 3.2.1) to obtain the multi-head attention output of the pose block aggregation module;

[0060]

[0061] in, This represents a two-dimensional convolution with a kernel size of 1×k1.

[0062] 3.2.3) Apply formula (5) to perform a convolution operation on the multi-attention head output XSTTA obtained in step 3.2.2) to finally achieve the aggregation of pose blocks.

[0063]

[0064] in, This represents a two-dimensional convolution with a kernel size of k2×1.

[0065] 3.3) Based on the correlation of the pose blocks obtained in step 3.2), the animal behavior classification results are obtained through global average pooling and fully connected layers.

[0066] Experimental Analysis

[0067] (1) Experimental comparison

[0068] To demonstrate the advancement of the proposed method, a comparative experiment was conducted on a constructed video dataset, comparing the proposed animal behavior recognition method with existing spatiotemporal deep neural network behavior recognition methods. Accuracy was used as the evaluation metric, and the experimental results are shown in Table 1.

[0069] Table 1 Comparison of experimental results

[0070]

[0071] (2) Determining the number n of attitude blocks

[0072] To achieve optimal recognition results during pose block segmentation, this invention sets the number of different pose blocks, n, and conducts experiments. The experimental results are attached. Figure 4 As can be seen from the figure, the accuracy of animal behavior recognition decreases when the number of pose blocks, n, is too large or too small. Analysis of the experimental videos revealed that a sub-action in an animal behavior typically lasts 8-12 frames, which is likely why the accuracy of animal behavior recognition is higher when the number of pose blocks, n, is around 10.

[0073] (3) Convergence of the model

[0074] As attached Figure 4 As shown, with the increase of the number of iterations, the accuracy of the animal behavior recognition method of the present invention gradually increases. When the number of epochs increases to 30, the accuracy of the animal behavior recognition method basically stabilizes at the maximum value; as shown... Figure 5 As shown, when the Epoch increases to 30, the loss function of the animal behavior recognition method begins to converge, indicating that the convergence speed of the method of the present invention is relatively fast.

Claims

1. An animal behavior recognition method based on Pose-Transformer network, which first collects animal behavior videos and then preprocesses the animal behavior videos; Then, a recognition network is constructed to identify animal behaviors in videos, characterized by: The recognition network is a Pose-Transformer network, and its construction steps include: 1) Obtain the location of the joints of the animal's body parts, and generate the joint coordinates and confidence scores of the animal; for the skeleton sequence of the T0 frame of the input video, divide it into a pose block sequence; 2) In the Transformer network, the pose blocks obtained in step 1) are subjected to a spatiotemporal Transformer Encoder, wherein: the temporal Transformer Encoder and the spatial Transformer Encoder assign corresponding weights to the skeleton joints of different pose blocks in the temporal and spatial order, respectively, based on the different correlation strengths between the joints corresponding to different animal behaviors, to characterize the spatiotemporal correlation degree between the joints of different behaviors. 3) The temporal information of the pose block sequence obtained in step 2) is modeled through a pose block aggregation layer, and then the animal behavior classification result is obtained through global average pooling and a fully connected layer; In step 2), 2.1) After the embedding layer of the Transformer network, a positional encoder, positionalEmbedding, is added to convert the joint information of the animal skeleton sequence of each pose block into a Transformer vector representation. The generated vector includes a query vector Q, a key vector K, and a value vector V. 2.2) Obtaining spatial correlations between joints: Input the vector obtained in step 2.1) into the spatial TransformerEncoder layer and process it to obtain the spatial correlation features of joints of each animal skeleton in each pose block; the spatial Transformer Encoder layer consists of N stacked Spatial Transformer Encoder blocks. 2.3) Obtain the temporal correlation between joints: Input the vector obtained in step 2.1) into the temporal Transformer Encoder layer and process it to obtain the temporal correlation features of the joints of the animal skeleton sequence of each pose block; the temporal Transformer Encoder layer consists of N stacked Temporal Transformer Encoder blocks. In step 3), 3.1) After performing a concatenation operation on the spatial association features and temporal association features of the joints obtained in step 2), use them as input to the attitude block aggregation submodule; 3.2) Analyze the temporal correlation of consecutive attitude block sequences, i.e., the correlation between attitude blocks and corresponding attitudes, through attitude block aggregation operations; 3.3) Based on the correlation of the pose blocks obtained in step 3.2), the animal behavior classification results are obtained through global average pooling and fully connected layers.

2. The animal behavior recognition method based on Pose-Transformer network according to claim 1, characterized in that: In step 1), for the skeleton sequence of input frame T0, each frame's skeleton sequence contains k0 key points. The skeleton sequence is divided into a sequence of pose blocks, with n consecutive frames forming a pose block. Therefore, each pose block has n×k0 key points, and the number of pose blocks is...

3. The animal behavior recognition method based on Pose-Transformer network according to claim 1, characterized in that: In step 2.2), calculating the spatial correlation between joints of consecutive skeletal sequences corresponding to animal behaviors within the same pose block's animal skeletal sequence includes the following steps: 2.2.1) Divide the animal skeleton into 5 different parts: the spine and 4 legs; 2.2.2) Constructing the spatial Transformer Encoder layer of the Transformer network: First, the Spatial Self-Attention module is followed by the FeedForward network layer and the Add&Norm residual connection / normalization layer to form the Spatial Transformer Encoder block. Then, N Spatial Transformer Encoder blocks are stacked to form a Spatial Transformer Encoder layer; 2.2.3) In the spatial Transformer Encoder layer, the spatial self-attention module is used to calculate the spatial correlation between the joints of each part of the animal skeleton and assign corresponding weights to the skeleton joints. Finally, the spatial association features of the joints of each animal skeleton in each pose block are obtained.

4. The animal behavior recognition method based on a Pose-Transformer network according to claim 3, characterized in that... In step 2.2.3), the formula for calculating the association weights of the self-attention mechanism is as shown in equation (1): Where, q i =Q(i),k i =K(i),v i =V(i), Q,K,V are the query vector, key vector, and value vector obtained in step 2.1), respectively, and their inner product is V(i). Here, T stands for "transpose;". The spatial association weight of the i-th node of an animal skeleton in a frame with the other joints in the same part is calculated using formula (1).

5. The animal behavior recognition method based on Pose-Transformer network according to claim 3, characterized in that: In step 2.3), calculating the temporal correlation between joints of consecutive skeletal sequences corresponding to animal behaviors within the same pose block's animal skeletal sequence includes the following steps: 2.3.1) Constructing the temporal Transformer Encoder layer of the Transformer network: First, a feedforward network layer and a residual connection / normalization layer Add&Norm are followed after the temporal self-attention module to form the Temporal Transformer Encoder block; Then, N Temporal Transformer Encoder blocks are stacked to form a temporal Transformer Encoder layer; 2.3.2) In the temporal Transformer Encoder layer, for a pose block, the temporal self-attention module is used to calculate the temporal correlation between a certain key point in a certain frame of the pose block and the key points at the same position in the other frames of the pose block, and to assign different weights to the skeleton key points. Finally, the temporal correlation features of the joints of the skeleton sequence of each pose block are obtained.

6. The animal behavior recognition method based on Pose-Transformer network according to claim 5, characterized in that... In step 2.3.2), the formula for calculating the association weights of the self-attention mechanism is as shown in equation (2): Where, q i =Q(i),k i =K(i),v i =V(i), Q,K,V are the query vector, key vector, and value vector obtained in step 2.1), respectively, and their inner product is V(i). Here, T stands for "transpose;". The temporal association weight between the i-th node of the animal skeleton in a certain frame of a pose block and the i-th node of the skeleton in the remaining frames of the pose block is calculated using formula (2).

7. The animal behavior recognition method based on Pose-Transformer network according to claim 1, characterized in that: In step 3.2), the temporal correlation of consecutive attitude block sequences, i.e., the correlation between attitude blocks and corresponding attitudes, is analyzed through attitude block aggregation operations. The steps include: 3.2.1) Calculate the output of each attention head and perform a Concat operation on the output of each attention head, as shown in formula (3); in, (1≤i≤h) represents the output of the i-th attention head; 3.2.2) Apply formula (4) to X obtained in step 3.2.1). Attn Perform a convolution operation to obtain the multi-head attention output of the pose block aggregation module: in, This represents a two-dimensional convolution with a kernel size of 1×k1; 3.2.3) Apply formula (5) to the X obtained in step 3.2.2). STTA The output of the multi-attention head is used for convolution operations to ultimately achieve the aggregation of pose blocks: in, This represents a two-dimensional convolution with a kernel size of k2×1.

Images