Wild animal posture estimation method based on dynamic prompt and semi-supervised multi-mode learning

Abstract

The invention discloses a wild animal attitude estimation method based on dynamic prompt and semi-supervised multi-mode learning, and relates to the technical field of computer vision and animal behavior analysis, and the method comprises the steps: firstly carrying out the motion state analysis of an input video, so as to extract parameters such as the centroid velocity, the acceleration and the steering angle of a moving material; the method comprises the following steps: generating a dynamic text prompt containing directional semantics, constructing a cross-modal fusion time sequence convolutional network, fusing visual features and text prompt features through a space alignment mask mechanism and a cross attention module, capturing time sequence dependence by using a bidirectional ConvGRU network to realize attitude estimation, and finally, based on a semi-supervised multi-task learning framework, carrying out dynamic text prompt processing on the basis of the semi-supervised multi-task learning framework. A joint loss function including supervision loss, time difference loss, attitude PCA reconstruction loss, multi-modal alignment loss and motion consistency loss is adopted, a loss weight is adaptively adjusted in combination with prediction uncertainty to optimize network parameters, a smooth 2D / 3D animal skeleton is finally output, real-time reasoning can be realized on edge equipment, and the real-time reasoning efficiency is improved. Ecological analysis software is compatible, high precision is still kept under the condition of low-label data, and the performance is remarkably improved especially in an animal sharp turning scene.

Description

Technical Field

[0001] This invention relates to the fields of computer vision and intelligent wildlife monitoring technology, specifically a wildlife pose estimation method based on dynamic cueing and semi-supervised multimodal learning, which is used to achieve accurate estimation of wildlife pose under low annotation cost conditions. Background Technology

[0002] Wildlife behavior research is a core requirement of ecology and conservation biology, and posture estimation, as the technical foundation for behavioral quantification, is of great significance for understanding animal ecological habits and assessing population health. Traditional manual observation methods have limitations such as interfering with natural animal behavior and low data collection efficiency.

[0003] With the development of computer vision technology, deep learning-based pose estimation methods have achieved remarkable results in laboratory environments. However, in real-world applications in the field, existing technologies face severe challenges: high annotation costs: annotating a single frame of a reliable animal skeleton requires 3-5 minutes from experienced ecologists, and dense sampling and annotation are difficult for video data of rare species; severe environmental interference: factors such as occlusion, changes in lighting, and camouflage colors cause key parts to be invisible in more than 30% of video frames; high motion complexity: instantaneous maneuvers such as sharp turns and jumps bring motion blur and sudden changes in viewpoint, which traditional single-frame or shallow temporal models cannot accurately capture; limited modal information: existing methods mainly rely on visual information and lack effective utilization of behavioral textual knowledge.

[0004] In particular, existing fully supervised methods fail directly in scenarios with rare species where labeled data is less than 5%, while traditional semi-supervised methods are inadequate in modeling long-range dependencies related to occlusion and reconstruction, resulting in significant trajectory jitter. However, multimodal information can address the problems of semi-supervised methods. Furthermore, static cues cannot provide effective directional priors for complex motions, leading to a significant drop in estimation accuracy in scenarios with sudden maneuvers such as sharp turns. Summary of the Invention

[0005] In view of the above-mentioned deficiencies of the prior art, the technical problem to be solved by the present invention is to provide a wildlife pose estimation method based on dynamic cueing and semi-supervised multimodal learning, which can achieve accurate estimation of wildlife pose under low-label conditions, and maintain robustness, especially in complex scenarios such as occlusion and sharp turns.

[0006] To achieve the above objectives, this invention provides a method for wild animal pose estimation based on dynamic prompts and semi-supervised multimodal learning, characterized by the following steps:

[0007] Step 1: Analyze the motion state of the input video and generate dynamic text prompts;

[0008] Step 2: Construct a cross-modal fusion temporal convolutional network to perform pose estimation by combining visual features and textual cues;

[0009] Step 3: Optimize network parameters based on a semi-supervised multi-task learning framework and output 2D / 3D animal skeletons.

[0010] Furthermore, step one specifically includes:

[0011] A1. Calculate the motion field between video frames using the Farneback dense optical flow algorithm, and extract the animal's center of mass velocity v, acceleration a, and turning angle θ;

[0012] A2. Construct a five-state classifier based on motion parameters to classify motion states into five categories: steady speed, acceleration, deceleration, left turn, and right turn;

[0013] A3. Based on the historical motion trajectory of 0.5 seconds, perform linear extrapolation and quadratic curve correction to predict the motion trajectory of the next 5 frames;

[0014] A4. Based on the motion state, retrieve the corresponding text description from the pre-built 120 motion cue word library and generate dynamic cue containing directional semantics;

[0015] A5. Introduce a 4×77-dimensional learnable prefix vector to enable dynamic prompts to optimize the text encoder through gradient backpropagation.

[0016] Furthermore, the specific steps for generating dynamic prompts described in step A4 of step one are as follows:

[0017] B1. Generate directional descriptive text for 17 key animal points, in the form of "location + direction + action", including "tail base to the left" and "right shoulder retracted";

[0018] B2. Combine species name with movement state to generate complete prompts, such as "Siberian tiger is turning right, right forelimb extended";

[0019] B3. Dynamic prompt text input CLIP text encoder, output 77×512-dimensional text feature vector.

[0020] Furthermore, step two specifically includes:

[0021] C1, the visual branch uses the CLIP-ResNet50 backbone to extract image features and outputs 49 visual tokens;

[0022] C2. Text branch receives the text features after dynamic prompt encoding;

[0023] C3. Align visual and text tokens through a spatial alignment masking mechanism, and then fuse them through a cross-attention module;

[0024] C4. The fused features are input into a bidirectional ConvGRU network to capture temporal dependencies, and the final output is a key point heatmap and an uncertainty map.

[0025] Furthermore, step three specifically includes:

[0026] D1. Construct the joint loss function for the five tasks:

[0027] L_total=λ_sup×L_sup+λ_td×L_td+λ_pca×L_pca+λ_align×L_align+λ_mot×L_mot

[0028] in:

[0029] L_sup is the supervision loss, which includes heatmap MSE loss and L1 coordinate regression loss;

[0030] L_td is the temporal difference loss, which is generated by calculating the Euclidean distance between the predicted poses of adjacent frames. The loss occurs when the displacement exceeds the maximum allowable displacement threshold.

[0031] L_pca is the pose PCA reconstruction loss, which constrains the predicted pose to a low-dimensional reasonable subspace and calculates the pixel error between the original prediction and the reconstructed coordinates.

[0032] L_align is the multimodal alignment loss, which uses InfoNCE contrastive learning to align visual-text features;

[0033] L_mot is the motion consistency loss, which penalizes cases where the extrapolation distance between the predicted pose and the trajectory exceeds 3σ.

[0034] D2. Adaptive adjustment of loss weights based on prediction uncertainty: The larger the uncertainty σ, the higher the weights of λ_td and λ_mot, thus strengthening the physical constraints;

[0035] D3. The output posture sequence is post-processed using an extended Kalman filter. A state-space model is constructed by combining the dynamic model and the observation model to obtain a smooth 2D / 3D animal skeleton.

[0036] Furthermore, the adaptive weight adjustment described in D2 is specifically implemented as follows:

[0037] λ_i=λ_i_base×(1+α×σ 2 )

[0038] in:

[0039] λ_i_base represents the base weights of the i-th loss term;

[0040] α is the weighting adjustment coefficient;

[0041] σ 2 To predict the variance, pose prediction is performed on the same frame using multiple independently trained networks, and the variance of the prediction results is calculated.

[0042] Furthermore, the time difference loss function is defined as:

[0043] L_td=∑∑max(0,||p_{k,t}-p_{k,t-1}|| 2 -δ 2 )

[0044] Where p_{k,t} is the predicted position of the k-th keypoint in frame t, and δ is the maximum allowable displacement threshold.

[0045] Furthermore, the attitude PCA loss function compresses the multi-view attitude coordinates into a low-dimensional three-dimensional subspace using principal component analysis, and calculates the pixel error between the original predicted and reconstructed coordinates.

[0046] The beneficial effects of this invention are: it can achieve high-precision estimation of wild animal postures using only 10-20% of the labeled data, achieving PCK@0.2 = 86.7% on the self-built LOTE-18k dataset, and improving the PCK metric by 18.6% compared to baseline methods on the sharp turn subset. Through dynamic cue generation and cross-modal fusion mechanisms, it effectively solves the posture estimation problem in occlusion and complex motion scenarios, providing reliable technical support for wild animal behavior research. Attached Figure Description

[0047] Figure 1 This is a schematic diagram of the overall process of Embodiment 1 of the present invention;

[0048] Figure 2 This is a schematic diagram of the motion state analysis and dynamic prompt generation process;

[0049] Figure 3 This is a schematic diagram of a cross-modal fusion network structure;

[0050] Figure 4 This is a schematic diagram of a semi-supervised multi-task learning framework. Detailed Implementation

[0051] The present invention will be further described below with reference to the accompanying drawings and embodiments:

[0052] This embodiment uses the estimation of Siberian tiger posture from field surveillance video as an example to illustrate the specific implementation process of the present invention.

[0053] like Figure 1 As shown, this embodiment provides a method for wild animal pose estimation based on dynamic prompts and semi-supervised multimodal learning, including the following steps:

[0054] like Figure 2 The analysis shown describes how to perform motion state analysis on the input video and generate dynamic text prompts. Step one includes:

[0055] A0. The LOTE-18k and AP-10K datasets are integrated to form an integrated dataset;

[0056] A1. Seventeen key points of the animal were marked using the COCO-Annotator tool, including key parts such as the nose tip, eyes, ears, neck, limb joints, and tail root.

[0057] A2. Divide the dataset into training, validation, and test sets in an 8:1:1 ratio to ensure a reasonable data distribution.

[0058] A3. Construct a visual encoder based on CLIP-ResNet50, fine-tune it on a wildlife dataset, and set the learning rate to 1e-5;

[0059] A4. Construct a motion state classifier and use a lightweight CNN architecture to process optical flow features to achieve five state classifications: steady speed, acceleration, deceleration, left turn, and right turn.

[0060] A5. Pre-build a vocabulary of 120 motion cue words and generate dynamic text descriptions based on motion status and species information;

[0061] A6. Design a cross-modal fusion network, using a bidirectional ConvGRU structure for temporal modeling, with a hidden layer dimension of 512 and 3 network layers.

[0062] A7. Construct a semi-supervised multi-task loss function, including supervision loss, temporal difference loss, pose PCA loss, multimodal alignment loss, and motion consistency loss;

[0063] A8. The Farneback dense optical flow algorithm is used to calculate the motion field between video frames and extract the animal's center of mass velocity, acceleration and turning angle parameters.

[0064] A9. The motion state classifier contains two convolutional layers and two fully connected layers. It takes an optical flow feature map as input and outputs the probabilities of five types of motion states.

[0065] A10. Based on the historical motion trajectory of 0.5 seconds, a weighted linear regression is performed to predict the motion trajectory of the next 5 frames, providing prior information for attitude estimation.

[0066] A11. The dynamic prompt generation module automatically generates descriptive text such as "The Siberian tiger is turning right and extending its right forelimb" based on the movement status and species information.

[0067] A12. Introduce a 4×77-dimensional learnable prefix vector and optimize the text encoding process through gradient backpropagation.

[0068] Furthermore, such as Figure 3 The diagram shows the construction of a cross-modal fusion temporal convolutional network, which combines visual features and textual cues for pose estimation. Step two includes:

[0069] B1. Visual feature extraction uses CLIP-ResNet50 backbone. The input image is normalized to 256×256 and the output is 49 512-dimensional visual tokens.

[0070] B2. Text feature extraction uses the BERT encoder to process dynamic prompts, outputs text features and reduces the dimensionality to 512 dimensions to align with visual features;

[0071] B3. The cross-modal attention mechanism uses visual features as queries and text features as keys and values ​​to achieve deep feature fusion.

[0072] B4. The temporal modeling network uses a bidirectional ConvGRU to process a 5-frame sequence, effectively capturing the temporal dependencies of motion.

[0073] Furthermore, such as Figure 4 The diagram shows the optimization of network parameters based on a semi-supervised multi-task learning framework, outputting a 2D / 3D animal skeleton. Step three includes:

[0074] C1, the supervised loss combines the heatmap MSE loss and L1 coordinate regression loss, and is calculated only on the labeled data;

[0075] C2. Temporal difference loss constrains the displacement of key points between adjacent frames to avoid unreasonable attitude jumps;

[0076] C3. Attitude PCA loss uses principal component analysis to constrain and predict attitude within a reasonable subspace.

[0077] C4. The multimodal alignment loss uses InfoNCE contrastive learning to align visual and textual feature representations;

[0078] C5. Motion consistency loss is based on trajectory prediction results and strengthens kinematic physical constraints;

[0079] C6. The AdamW optimizer is used for model training with an initial learning rate of 1e-4 and the CosineAnnealingLR scheduling strategy.

[0080] C7. Adaptively adjust the loss weight based on the prediction uncertainty, and strengthen the physical constraint loss when the uncertainty is high;

[0081] C8. Post-processing uses extended Kalman filtering to smooth the attitude sequence, and the state vector contains the coordinates of 17 key points.

[0082] C9 accelerates the inference phase with model quantization, graph optimization, and other processing, and supports deployment on edge devices.

[0083] Experimental results show that on the LOTE-18k test set, the proposed method achieves 86.7% PCK@0.2, an improvement of 18.3% compared to the DeepLabCut baseline. It performs exceptionally well in complex scenes such as sharp turns and occlusion, with improvements of 18.6% and 12.4%, respectively. The system achieves 30fps real-time inference on an RTX 3080 and 15fps on a Jetson Orin edge device.

[0084] In practical applications, this method has processed over 1,000 hours of surveillance video in the Northeast Tiger Nature Reserve, achieving an accuracy rate of over 85% in identifying behaviors such as foraging, socializing, and vigilance. The output data can be directly imported into professional behavioral analysis software.

[0085] In summary, this embodiment verifies the effectiveness and advancement of the present invention in the field of wildlife posture estimation through systematic technical implementation, providing reliable technical support for ecological protection and research.

[0086] The preferred embodiments of the present invention have been described in detail above. It should be understood that those skilled in the art can make numerous modifications and variations based on the concept of the present invention without creative effort. Therefore, all technical solutions that can be obtained by those skilled in the art based on the concept of the present invention through logical analysis, reasoning, or limited experimentation on the basis of existing technology should be within the scope of protection defined by the claims.

Claims

1. A method for wild animal pose estimation based on dynamic prompts and semi-supervised multimodal learning, characterized in that, Includes the following steps: Step 1: Analyze the motion state of the input video and generate dynamic text prompts; Step 2: Construct a cross-modal fusion temporal convolutional network to perform pose estimation by combining visual features and textual cues; Step 3: Optimize network parameters based on a semi-supervised multi-task learning framework and output 2D / 3D animal skeletons.

2. The method for wild animal pose estimation based on dynamic prompts and semi-supervised multimodal learning as described in claim 1, characterized in that, Step one specifically includes: A1. Calculate the motion field between video frames using the Farneback dense optical flow algorithm, and extract the animal's center of mass velocity v, acceleration a, and turning angle θ; A2. Construct a five-state classifier based on motion parameters to classify motion states into five categories: steady speed, acceleration, deceleration, left turn, and right turn; A3. Based on the historical motion trajectory of 0.5 seconds, perform linear extrapolation and quadratic curve correction to predict the motion trajectory of the next 5 frames; A4. Based on the motion state, retrieve the corresponding text description from the pre-built 120 motion cue word library and generate dynamic cue containing directional semantics; A5. Introduce a 4×77-dimensional learnable prefix vector to enable dynamic prompts to optimize the text encoder through gradient backpropagation.

3. The method for wild animal pose estimation based on dynamic prompts and semi-supervised multimodal learning as described in claim 2, characterized in that, The specific steps for generating dynamic prompts as described in A4 are as follows: B1. Generate directional descriptive text for 17 key animal points, in the form of "location + direction + action", including "tail base to the left" and "right shoulder retracted"; B2. Combine species name with movement state to generate complete prompts, such as "Siberian tiger is turning right, right forelimb extended"; B3. Dynamic prompt text input CLIP text encoder, output 77×512-dimensional text feature vector.

4. The method for wild animal pose estimation based on dynamic prompting and semi-supervised multimodal learning as described in claim 1, characterized in that, Step two specifically includes: C1, the visual branch uses the CLIP-ResNet50 backbone to extract image features and outputs 49 visual tokens; C2. Text branch receives the text features after dynamic prompt encoding; C3. Align visual and text tokens through a spatial alignment masking mechanism, and then fuse them through a cross-attention module; C4. The fused features are input into a bidirectional ConvGRU network to capture temporal dependencies, and the final output is a key point heatmap and an uncertainty map.

5. The method for wild animal pose estimation based on dynamic prompts and semi-supervised multimodal learning as described in claim 1, characterized in that, Step three specifically includes: D1. Construct the joint loss function for the five tasks: L_total=λ_sup×L_sup+λ_td×L_td+λ_pca×L_pca+λ_align×L_align+λ_mot×L_mot in: L_sup is the supervision loss, which includes heatmap MSE loss and L1 coordinate regression loss; L_td is the temporal difference loss, which is generated by calculating the Euclidean distance between the predicted poses of adjacent frames. The loss occurs when the displacement exceeds the maximum allowable displacement threshold. L_pca is the pose PCA reconstruction loss, which constrains the predicted pose to a low-dimensional reasonable subspace and calculates the pixel error between the original prediction and the reconstructed coordinates. L_align is the multimodal alignment loss, which uses InfoNCE contrastive learning to align visual-text features; L_mot is the motion consistency loss, which penalizes cases where the extrapolation distance between the predicted pose and the trajectory exceeds 3σ. D2. Adaptive adjustment of loss weights based on prediction uncertainty: The larger the uncertainty σ, the higher the weights of λ_td and λ_mot, thus strengthening the physical constraints; D3. The output posture sequence is post-processed using an extended Kalman filter. A state-space model is constructed by combining the dynamic model and the observation model to obtain a smooth 2D / 3D animal skeleton.

6. The method for wild animal pose estimation based on dynamic prompts and semi-supervised multimodal learning as described in claim 5, characterized in that, The adaptive weight adjustment described in D2 is specifically implemented as follows: λ_i=λ_i_base×(1+α×σ 2 ) in: λ_i_base represents the base weights of the i-th loss term; α is the weighting adjustment coefficient; σ 2 To predict the variance, pose prediction is performed on the same frame using multiple independently trained networks, and the variance of the prediction results is calculated.

7. The method for wild animal pose estimation based on dynamic prompts and semi-supervised multimodal learning as described in claim 5, characterized in that, The time difference loss function is defined as: L_td=∑∑max(0,||p_{k,t}-p_{k,t-1}|| 2 -δ 2 ) Where p_{k,t} is the predicted position of the k-th keypoint in frame t, and δ is the maximum allowable displacement threshold.

8. The method for wild animal pose estimation based on dynamic prompts and semi-supervised multimodal learning as described in claim 5, characterized in that, The attitude PCA loss function uses principal component analysis to compress multi-view attitude coordinates into a low-dimensional three-dimensional subspace and calculates the pixel error between the original predicted and reconstructed coordinates.

Images