Forest musk deer posture recognition method based on multi-modal semantic learning

Abstract

The application discloses a forest musk deer posture recognition method based on multi-modal semantic learning, relates to the technical field of animal behavior analysis and computer vision, and comprises the following steps: acquiring a forest musk deer clipping image and a global feature map with different resolutions of a to-be-recognized forest musk deer image; obtaining a global feature vector based on the global feature map; constructing a local feature learning model, inputting the forest musk deer clipping image into the local feature learning model, obtaining a plurality of channel heat maps, decoding the channel heat maps to obtain two-dimensional coordinates and confidence of a plurality of forest musk deer skeleton key points; obtaining a topological semantic feature vector and a multi-frame skeleton sequence based on the two-dimensional coordinates and the confidence; extracting a space-time joint feature of the multi-frame skeleton sequence, fusing the global feature vector, the topological semantic feature vector and the space-time joint feature to obtain a multi-modal feature fusion vector, obtaining a probability distribution of different actions of the forest musk deer based on the multi-modal feature fusion vector, and further obtaining a forest musk deer posture recognition result; and the problem that a backbone network of an existing forest musk deer posture recognition method is not optimized for forest musk deer biological characteristics, resulting in insufficient feature discrimination, and the problem that a heat map method is sensitive to occlusion, resulting in a high key point missing detection rate.

Description

Technical Field

[0001] This invention relates to the fields of animal behavior analysis and computer vision technology, specifically to a method for recognizing the posture of the musk deer based on multimodal semantic learning. Background Technology

[0002] In the musk deer farming industry, understanding the posture and movements of musk deer is crucial for monitoring their health, behavioral habits, and optimizing the farming environment. Animal posture estimation and behavior recognition are among the core technologies of smart farming. Currently, musk deer posture recognition methods typically employ a deep learning-based multi-animal posture estimation framework. The technical approach involves: collecting video data of musk deer in farming scenarios, manually annotating key points, extracting image features using a general backbone network (such as ResNet), locating the coordinates of key points through heatmap regression, and finally tracking the trajectory based on temporal information. This method has achieved good recognition results in experimental scenarios involving rodents and other animals. However, this method has the following shortcomings:

[0003] 1) Using general image classification networks such as ResNet as the backbone of feature extraction, these networks are pre-trained on large-scale natural image datasets and have not been adapted to the unique fur color, texture and body shape characteristics of the musk deer, resulting in limited discriminative power of the extracted visual features, especially in breeding environments with complex backgrounds and drastic changes in lighting, where the feature expression ability is insufficient.

[0004] 2) Musk deer breeding enclosures often contain obstructions such as feeding troughs and perches, and there is mutual obstruction among musk deer. Existing heatmap methods generate Gaussian heatmaps for supervised training based on key point locations. When the obstruction rate exceeds 30%, the missed detection rate of key points increases sharply, the positioning accuracy decreases significantly, and the robustness of attitude estimation is seriously affected. Summary of the Invention

[0005] To address the problems of insufficient feature discrimination due to the backbone network not being optimized for the biometric features of musk deer in existing musk deer pose recognition methods, and high missed detection rates of key points due to the sensitivity of heatmap methods to occlusion, this invention provides a musk deer pose recognition method based on multimodal semantic learning. The method includes:

[0006] Obtain the cropped image of the musk deer and global feature maps at different resolutions from the image of the musk deer to be identified;

[0007] The global feature vector is obtained based on the global feature map;

[0008] A local feature learning model is constructed, comprising a multi-resolution parallel backbone network, a linear mapping module, and a regression head network connected in sequence. The multi-resolution parallel backbone network includes several resolution branch channels with different resolutions, and the linear mapping module includes several first linear layers.

[0009] The cropped image of the musk deer is input into the local feature learning model. Based on the multi-resolution parallel backbone network, local feature maps of each resolution branch channel are obtained. Multi-scale feature fusion is performed on the local feature maps through cross-resolution connections to obtain fused features. The linear mapping module enhances the fused features to obtain enhanced features. Based on the regression head network, the enhanced features are analyzed to obtain several channel heatmaps. The channel heatmaps are decoded to obtain several musk deer skeleton key points of each frame of the cropped image of the musk deer, as well as the two-dimensional coordinates and confidence scores corresponding to each musk deer skeleton key point.

[0010] A topological semantic feature vector is obtained based on the two-dimensional coordinates and the confidence level, and a multi-frame skeleton sequence is obtained based on all the two-dimensional coordinates of the cropped images of the musk deer in multiple frames;

[0011] The spatiotemporal joint features of the multi-frame skeleton sequence are extracted, and the global feature vector, the topological semantic feature vector and the spatiotemporal joint features are fused to obtain a multimodal feature fusion vector. Based on the multimodal feature fusion vector and the pre-trained musk deer action recognition model, the probability distribution of different musk deer actions is obtained, and the musk deer posture recognition result is obtained based on the probability distribution.

[0012] This method, based on the classic high-resolution network (HRNet), introduces a multi-resolution feature parallel fusion mechanism and a self-attention enhancement strategy: The multi-resolution parallel backbone network adopts the HRNet multi-resolution parallel architecture, maintaining four resolution branches from high to low. Each branch interacts with information through cross-resolution connections and feature addition, forming a unified and multi-level feature representation, realizing multi-scale feature fusion; The linear mapping module maps multiple independent linear layers into query (Q), key (K), and value (V) matrices. After feature fusion, a scaling dot product self-attention mechanism is introduced to enhance the focusing ability on key pose regions, significantly improving the localization ability of easily occluded and small-scale key points in the musk deer's pose. It is suitable for real-time pose estimation tasks in complex breeding scenarios, making it more suitable for learning local features of musk deer pose, accurately capturing key point information of various parts of the musk deer's body, and learning the local pose semantics of the musk deer.

[0013] Furthermore, the specific steps for obtaining the enhanced features include:

[0014] The first linear layer maps the fused features into a first query matrix, a first key matrix, and a value matrix;

[0015] A first attention weight matrix is ​​obtained based on the first query matrix and the first key matrix;

[0016] The value matrix is ​​weighted based on the first attention weight matrix to obtain a weighted value matrix. The weighted value matrix is ​​then residually connected to the fused feature to obtain the enhanced feature.

[0017] The formula for calculating the first attention weight matrix is ​​as follows:

[0018] ;

[0019] The formula for calculating the enhanced features is as follows:

[0020] ;

[0021] in, This represents the first attention weight matrix. This represents the Softmax function. This represents the first query matrix. Represents the first bond matrix. Indicates transpose. Indicates the scaling factor. Represents a value matrix, Indicates enhanced features, This indicates the fusion feature.

[0022] Introducing attention weight matrix The model adaptively focuses on regions with significant motion or structural criticality, preserving original feature information through residual connections and reducing feature degradation caused by the attention mechanism. The linear mapping module employs a multi-head attention mechanism, dividing the first query matrix, the first key matrix, and the value matrix into multiple parallel subspaces along the channel dimension for independent scaling dot product attention calculations, and then concatenating the outputs of each attention head. This mechanism enables the model to simultaneously capture multiple dependencies of the musk deer's local pose from different feature subspaces, significantly improving the robustness of localization for complex occluded key points.

[0023] Furthermore, the specific steps for obtaining the global feature vector based on the global feature map include:

[0024] A global feature learning model is constructed, which includes a fusion module, a self-attention sub-module, and an output module connected in sequence. The self-attention sub-module includes a multilayer perceptron, a second linear layer, and a softmax function layer connected in sequence.

[0025] The global feature map is input into the global feature learning model. The fusion module concatenates the global feature maps of different resolutions to generate multi-scale fusion features. The multi-scale fusion features are then subjected to convolution and normalization processes to obtain the first feature.

[0026] The multilayer perceptron and the second linear layer perform a nonlinear transformation on the first feature to generate a second query matrix and a second key matrix. The Softmax function layer normalizes the second query matrix and the second key matrix to generate a second attention weight matrix. The second attention weight matrix is ​​then multiplied element-wise with the first feature to obtain an attention-weighted feature.

[0027] The output module adds the attention-weighted feature to the first feature by performing residual addition to obtain the global feature vector.

[0028] The fusion module merges global feature maps from multiple channels through channel concatenation to form a unified multi-scale feature representation. After convolution and normalization, the features are reshaped and their dimensions are unified, providing structured input for subsequent attention calculations. Normalization generates an attention weight matrix that adaptively focuses on salient regions in the image related to the musk deer target (such as moving parts and key pose regions). The attention matrix is ​​then multiplied element-wise with another convolutionally processed feature to enhance key features and suppress non-key features. To maintain gradient flow and feature stability, a residual connection structure is used to add the attention-weighted features to the original fused features, avoiding information loss and outputting a highly discriminative global feature vector. This feature will be used for subsequent local pose estimation and action understanding tasks. Through multi-scale feature fusion and self-attention enhancement mechanisms, this model significantly improves its ability to model the overall appearance, pose, and scene context of the musk deer, while maintaining real-time inference speed and enhancing the robustness of target localization and semantic extraction.

[0029] Furthermore, the specific steps for obtaining the topological semantic feature vector based on the two-dimensional coordinates and the confidence level include:

[0030] A topological feature learning model is constructed, which includes a Chebyshev graph convolutional module and a temporal convolutional network connected in sequence. The Chebyshev graph convolutional module includes several parallel Chebyshev graph convolutional layers.

[0031] Based on the confidence level and the preset filtering threshold, the key points of the musk deer skeleton are screened to obtain target points. Based on the target points and the two-dimensional coordinates of the target points, a single-frame graph structure is obtained, and the single-frame graph structure is input into the topological feature learning model.

[0032] The Chebyshev graph convolutional layer performs a K-order approximation of the normalized Laplacian operator using Chebyshev polynomials, aggregates features in the local neighborhood, and captures multi-level spatial semantic information from local joint constraints to the overall pose configuration in a hierarchical manner, and obtains spatial topological features based on the multi-level spatial semantic information.

[0033] Based on the spatial topological features, a graph feature sequence is obtained. The graph feature sequence of R consecutive frames is input into the temporal convolutional network to obtain temporal features, where R represents an integer greater than or equal to 1.

[0034] The spatial topological features and the temporal features are fused to obtain the topological semantic feature vector;

[0035] The operational formula for the Chebyshev diagram convolutional layer is as follows:

[0036] ;

[0037] in, Indicates the first The output features of the layer Indicates the first The input features of the layer Indicates the first The weight matrix of the layer, Indicates the network layer index. This represents the Sigmoid function. Indicates the local neighborhood range. Indicates the summation index. Represents the Chebyshev polynomial. This represents the normalized Laplace matrix.

[0038] Graph Convolutional Networks (GCNs) are employed to analyze the topological structure of the musk deer's postures, enabling fine-grained topological semantic learning. Keypoints of the musk deer are treated as nodes in a graph, and the connections between these keypoints are represented as edges. GCNs are used to process this topological structure, allowing for in-depth analysis of the spatial relationships and action logic between different parts of the musk deer, thus leading to a better understanding of its behavioral intentions.

[0039] This model takes the coordinates and confidence scores of multiple anatomical keypoints of the musk deer, output by an improved HRNet network, as input. It abstracts the pose of each frame into a dynamic graph structure, where nodes represent keypoints, and edges are defined based on anatomical connections (such as natural links like the neck-shoulder-elbow-wrist) and real-time spatial proximity. Edge weights are dynamically calculated using a Gaussian kernel function to construct an adjacency matrix that adjusts with pose changes. During the construction process, the input keypoints are first filtered by confidence (threshold > 0.5) to ensure sufficient reliability of the information used in the graph construction, thereby enhancing the model's robustness in occluded and blurred scenes.

[0040] In terms of spatial topology learning, the module employs multi-layer Chebyshev graph convolution (ChebGCN) to process the single-frame graph structure. This method uses a Chebyshev polynomial to perform a K-order approximation of the normalized Laplacian operator. This design enables the model to capture multi-level semantic information from local joint constraints to overall pose configuration, and to explicitly model long-range dependencies such as limb support and trunk balance.

[0041] To simultaneously capture the temporal dynamics of posture, the model employs a parallel temporal modeling path. Continuous multi-frame feature sequences are cached and input into a Temporal Convolutional Network (TCN). The TCN uses dilated convolutional kernels with variable receptive fields to adaptively learn motion patterns at different time scales—from subtle head swaying to sustained strides—thereby extracting temporal features that characterize the evolution of movement.

[0042] Finally, the spatial topological features and temporal dynamic features of a single frame are deeply fused with a multilayer perceptron through residual connections to generate a compact and semantically rich topological feature vector (e.g., 256-dimensional). This vector not only encodes the spatial constraints and coordination relationships of the musk deer's posture but also implies the intent and logic of the action, providing an interpretable structured representation for subsequent high-precision action recognition and understanding. The entire module, through sparse graph operations and lightweight design, ensures real-time inference capabilities on edge devices, completing end-to-end topological learning from keypoint coordinates to high-level semantics.

[0043] Furthermore, the method for obtaining the spatiotemporal joint features includes: mapping each frame of the multi-frame skeleton sequence into a single-frame skeleton graph with key points of the musk deer skeleton as nodes and the anatomical connection relationships of each key point of the musk deer skeleton as edges; combining the single-frame skeleton graphs into a multi-frame spatiotemporal skeleton graph based on the time order; extracting the temporal and spatial features of the multi-frame spatiotemporal skeleton graph; and fusing the temporal and spatial features to obtain the spatiotemporal joint features.

[0044] The core of the musk deer action recognition model is a sequence understanding model with a Spatial-Temporal Graph Attention Network (ST-GAT) as its backbone. This module receives a series of optimized musk deer skeleton sequences from the front-end feature fusion module, naturally mapping the key points of each frame's skeleton to graph nodes, and constructing a unified multi-frame spatiotemporal skeleton graph based on anatomical connections and temporal order. Through an embedded multi-head attention mechanism, the network adaptively focuses on significantly moving body parts (such as the head and limbs) in the spatial dimension and captures the transitions and key rhythms of actions in the temporal dimension, thereby automatically learning and extracting highly discriminative spatiotemporal joint features. Finally, these high-level features are mapped to probability distributions of different types of actions by a classifier, completing a full and automated parsing process from the original image to pose coordinates and then to high-level behavioral semantics, providing immediate and accurate behavioral judgment criteria for breeding management.

[0045] Furthermore, an accuracy evaluation index is obtained based on the multi-frame skeleton sequence, and the accuracy evaluation index is used to evaluate the fusion accuracy of the multi-frame skeleton sequence;

[0046] The formula for calculating the accuracy evaluation index is as follows:

[0047] ;

[0048] in, Indicates the accuracy evaluation index. This represents the index of key points on the musk deer skeleton. Indicates the first The Euclidean distance between the predicted and true values ​​of key points on a musk deer skeleton Indicates the first Normalization factor corresponding to the key point type of the forest musk deer skeleton Indicates the target scale. This represents the Kronecker indicator function. Indicates the first Visible markers of key points on the skeleton of a forest musk deer.

[0049] Furthermore, based on the YOLO model, the cropped image of the musk deer to be identified and global feature maps at different resolutions are obtained. The YOLO model includes a backbone network for extracting global feature maps of the musk deer image to be identified, a neck network and detection head for fusing features, and a detection module for extracting the cropped image of the musk deer.

[0050] The front portion of the backbone network has its parameters frozen during training, while the rear portion of the backbone network, the neck network, and the detection head are fine-tuned using the target object dataset.

[0051] Training strategy based on the YOLO model: Pre-train on the COCO dataset → Freeze the front layers → Fine-tune the last 3 layers using musk deer data, focusing on foreground classification. The reason for freezing the front feature extraction layers of the backbone network and fine-tuning the later layers is: This is an efficient training strategy based on transfer learning theory, computational efficiency, and the specific task requirements of musk deer. Freezing the front layers of the backbone network (i.e., the shallow layers and some middle layers) is because these layers have learned powerful general visual features during pre-training (e.g., the COCO dataset), such as edges, corners, textures, and basic shapes. These features are effective and transferable for recognizing any object (including musk deer). Freezing them can prevent catastrophic forgetting or overfitting on the relatively small musk deer dataset, while significantly reducing the number of parameters and computational cost during training. Fine-tuning the later parts of the network (such as the Neck and the detection head) is necessary because these high-level modules are responsible for task-specific combinations and interpretations of basic features: the Neck (feature pyramid network) focuses on multi-scale feature fusion and needs to adapt to the scale changes of the musk deer at different distances and postures; the detection head is directly responsible for classification and localization, and must specifically learn the unique morphological features of the musk deer (such as specific body shape and coat color) and their contextual relationships in the breeding environment. This model can quickly identify individual musk deer in complex breeding environment images and obtain their overall appearance, location, and other global features.

[0052] Existing models are typically pre-trained on large-scale human motion datasets or general animal datasets. Due to the lack of high-quality skeletal keypoint datasets specifically for musk deer, these models cannot effectively capture the unique movement patterns of musk deer, such as their signature vertical take-off (used for marking territory or responding to fright) and stationary vigilance (remaining still for extended periods with highly alert head posture). Furthermore, as a ruminant, the skeletal topology and kinematic constraints (such as limb gait cycles and spinal flexibility) of musk deer do not match the distribution of existing model training data, leading to domain shifts, low recognition accuracy, and poor generalization ability when transferring applications to other models.

[0053] This method constructs a kinematic constraint model specific to the musk deer and embeds this prior knowledge into the model in the form of differentiable constraints. At the same time, it forms hard constraints in the loss function, so that the features learned by the model may be consistent with the real biological movement mechanism, thereby achieving accurate and robust identification of the complex behavior of the musk deer.

[0054] Furthermore, the musk deer action recognition model is trained based on musk deer video sequences. The specific steps for training the musk deer action recognition model based on musk deer video sequences include:

[0055] Extract the key points of the two-dimensional skeleton of the musk deer from the video sequence and construct the skeletal topology map of the musk deer;

[0056] A prior knowledge base of musk deer kinematics is constructed, which includes several biomechanical constraints.

[0057] The biomechanical constraints are transformed into a loss function term;

[0058] The musk deer action recognition model is trained based on the musk deer skeletal topology map and the loss function term.

[0059] The biomechanical constraints include phase period constraints, contact geometry constraints, and spinal flexibility constraints.

[0060] The specific steps for constructing the phase period constraint include: constructing phase variables based on sine functions, and constructing a mapping model between the trajectory features of key points on the limbs of the musk deer and the phase variables;

[0061] The formula for calculating the phase variable is:

[0062] ;

[0063] in, Indicates the first The phase variable of the frame, Represents angular frequency. Indicates the initial phase. Indicates the image frame index;

[0064] The calculation formula for the mapping model is:

[0065] ;

[0066] in, Indicates the first Predicted joint angles for frames. , and All of these represent learnable parameters. This represents the order of the Fourier series. Indicates the order index of the Fourier series;

[0067] The specific steps for constructing the contact geometric constraints include: constructing the ground equation for the ground in the musk deer breeding scenario; obtaining the relative distance between the hoof and the ground based on the key points of the musk deer's hoof and the ground equation; constructing a soft threshold function based on the relative distance; and determining the motion state of the musk deer's hoof and the ground based on the soft threshold function. If the motion state is a contact state, the velocity and acceleration of the musk deer's hoof in the vertical direction are constrained to approach zero. If the motion state is a swinging state, the motion trajectory of the musk deer's hoof is constrained to follow a preset swinging trajectory curve.

[0068] The formula for calculating the soft threshold function is as follows:

[0069] ;

[0070] ;

[0071] in, Represents the soft threshold function. This represents the Sigmoid function. Indicates the contact threshold. Indicates the first The relative distance between frames, Indicates the first The coordinates of the key points of the musk deer's hooves in the frame in the vertical direction. Indicates smoothness, Index of key points on the hooves of the musk deer;

[0072] The specific steps for constructing the spinal flexibility constraint include: constructing a sequence of key points along the spine of the musk deer, obtaining the Euclidean distance and direction vector between any two adjacent key points, and obtaining the bending angle between any three adjacent key points; the Euclidean distance constraint is within a preset distance range, and the bending angle is less than or equal to the maximum allowable bending angle.

[0073] Phase period constraint: Based on the kinematics theory of ruminants, a periodic model of the support phase and swing phase of the limbs of the musk deer under walking or specific actions is defined, and the mapping relationship between the joint angles and angular velocities of the limbs and the periodic phase is established.

[0074] Contact geometry constraints: Construct a geometric model of the contact surface between the key points of the hoof and the ground, and define the continuity and nonlinear contact force constraints of the position, velocity and acceleration of the hoof during the contact and takeoff processes.

[0075] Spinal flexibility restraint: Based on the physiological structure of the musk deer's spine, restrictions are established on the relative position and rotation angle of adjacent key points in front and behind the trunk center to restrain excessive twisting of posture.

[0076] The constructed phase period constraint, contact geometry constraint, and spinal flexibility constraint are transformed into differentiable loss function terms, which together with the original classification loss function of the basic model constitute the total loss function. This guides the optimization of network parameters, ensuring that the feature representation learned by the model strictly conforms to the real biological movement mechanism of the musk deer.

[0077] This method does not rely on massive, specially labeled musk deer movement datasets. Instead, it transforms profound biomechanical prior knowledge into differentiable mathematical constraints, directly guiding the model to learn features that conform to the real movement patterns of musk deer. Thus, even with limited data, it can achieve high-precision identification of musk deer-specific behaviors (such as vertical take-off and fixed-point alertness), breaking through the data dependence bottleneck and improving identification accuracy.

[0078] By forcing the model to follow the skeletal kinematic constraints of the musk deer (such as limb phase, spinal flexibility, and hoof-ground contact), the model effectively avoids learning spurious correlation features that contradict biological mechanisms, and significantly improves the model's generalization ability and robustness under different breeding environments, individuals, and postures.

[0079] The prior knowledge of musk deer skeletal movement is solidified into the model in a structured and differentiable form (loss function / graph structure constraints), rather than through simple data augmentation or post-processing. This deep integration of domain knowledge makes this method highly domain-specific and irreplaceable.

[0080] Because the training process of the model is subject to explicit physical and biological constraints, its internal feature representation is easier to interpret, thus improving the model's interpretability. For example, the model implicitly learns to recognize the support and swing phases of the limbs, making the decision-making basis for behavior recognition more reliable and transparent, which helps in attribution analysis of abnormal behaviors in aquaculture management.

[0081] Furthermore, the specific steps for transforming the biomechanical constraints into a loss function term include:

[0082] Gait phase is predicted for the current frame and the next frame to obtain predicted phase values, and a phase prediction consistency loss is constructed based on the predicted phase values; the predicted phase values ​​of the current frame are input into the mapping model to obtain predicted joint angles, and a motion reconstruction loss is constructed based on the predicted joint angles and the actual joint angles; a phase period constraint loss is obtained based on the phase prediction consistency loss and the motion reconstruction loss.

[0083] The formula for calculating the phase period constraint loss is as follows:

[0084] ;

[0085] ;

[0086] ;

[0087] in, This indicates the phase prediction consistency loss. Indicates the loss of sports reconstruction. Indicates the number of joints involved in the constraint (including the shoulder, elbow, wrist, hip, knee, and ankle of the limbs). Indicates the first In-frame joints The actual predicted joint angle, Indicates the first In-frame joints The reconstructed joint angle, This represents the node index in the joints of the limbs. This represents the phase period constraint loss. This represents the total number of frames in the musk deer video sequence. and They represent the first Frame and the Predicted phase variables of the frame, Represents angular frequency. Indicates the time interval between two adjacent frames;

[0088] The penetration depth is obtained based on the relative distance, and the positional penetration loss is obtained based on the penetration depth; the velocity and acceleration of the musk deer's hooves in the vertical direction are obtained, and the velocity loss and acceleration loss are obtained based on the velocity and acceleration, respectively; the contact geometric constraint loss is obtained based on the positional penetration loss, the velocity loss, and the acceleration loss.

[0089] The formula for calculating the contact geometric constraint loss is as follows:

[0090] ;

[0091] ;

[0092] ;

[0093] ;

[0094] ;

[0095] ;

[0096] ;

[0097] in, Indicates location penetration loss. Indicates the number of key points on the hooves of the musk deer. Indicates the depth of penetration. Indicates speed loss, This represents a state mask based on a soft threshold. This indicates the vertical velocity of the musk deer's hooves. Indicates acceleration loss, This indicates the vertical acceleration of the musk deer's hooves. This represents the contact geometric constraint loss. This represents the weighting coefficient of each item;

[0098] A distance consistency loss is constructed based on the Euclidean distance and the preset distance range; a bending angle loss is constructed based on the bending angle and the maximum allowable bending angle; and a spinal flexibility constraint loss is obtained based on the distance consistency loss and the bending angle loss.

[0099] The formula for calculating the spinal flexibility constraint loss is as follows:

[0100] ;

[0101] ;

[0102] ;

[0103] in, This represents the distance consistency loss. Indicates the number of key points in the spine. Indicates the index of key points in the spine. This represents the Euclidean distance between two adjacent spinal key points. Indicates the maximum distance. Indicates the minimum distance. Indicates the loss of bending angle. Indicates the sagittal plane curvature angle. and These represent the maximum permissible bending angles in the sagittal plane (i.e., the primary plane of motion) and the coronal plane (i.e., the lateral bending plane), respectively. This indicates loss of spinal flexibility restraint. All of these represent weighting coefficients.

[0104] The constructed phase period constraint, contact geometry constraint, and spinal flexibility constraint are transformed into differentiable loss function terms, which together with the original classification loss function of the basic model constitute the total loss function. This guides the optimization of network parameters, ensuring that the feature representation learned by the model strictly conforms to the real biological movement mechanism of the musk deer.

[0105] Furthermore, the biomechanical constraints also include motion causal structure constraints and behavioral temporal logic constraints. The specific steps for constructing the motion causal structure constraints include:

[0106] The skeletal key points of the musk deer are divided into several layers, including a core layer, a conduction layer, and a terminal layer. A directed acyclic graph (DAG) is constructed to characterize the causal relationship of motion propagation. The edge direction of the DAG is set from the core layer to the conduction layer and from the conduction layer to the terminal layer. For any two adjacent layers, the motion state of the constrained target layer is predicted by the motion state of the source layer through a causal function, and the forward prediction error is much smaller than the backward prediction error.

[0107] The specific steps for transforming the aforementioned motion causal structure constraints into loss function terms include:

[0108] Obtain the motion state splicing vector of the target layer and the source layer, construct a forward predictor and a backward predictor, and obtain causal directionality loss and causal temporal loss based on the motion state splicing vector, the forward predictor and the backward predictor; obtain causal structure constraint loss based on the causal directionality loss and the causal temporal loss.

[0109] The formula for calculating the causal structure constraint loss is as follows:

[0110] ;

[0111] ;

[0112] ;

[0113] ;

[0114] in, Indicates causal directionality loss. Indicates the target layer key points at the 1st level The state vector of a frame. This represents the change in the state of the target layer. Indicates the forward predictor. Indicates the source layer keypoint at the 1st The state vector of a frame. This represents the amount of change in the source layer state. Indicates the reverse predictor. Represents the positive part function (i.e. ), Indicates the time delay of causal propagation. Represents the timing delay mapping function, Indicates causal time loss, Represents the causal structure constraint loss. All represent weighting coefficients;

[0115] The specific steps for constructing the behavioral timing logic constraints include:

[0116] The temporal logic rules governing typical behaviors of the musk deer are obtained. These temporal logic rules include at least behavior transition rules, movement precursor rules, and state persistence rules. The form of these temporal logic rules is set as a precondition followed by subsequent behavior.

[0117] Based on the behavior transition rules, a behavior transition matrix is ​​constructed to constrain the consistency between the probability distribution of behavior transitions between adjacent frames and the behavior transition matrix; based on the motion precursor rules, a template matching function is constructed to constrain the degree of matching between the motion features before a specific behavior occurs and a preset template; based on the state persistence rules, the motion parameters during the persistence of a specific state are constrained not to exceed a preset threshold.

[0118] The specific steps for transforming the aforementioned behavioral temporal logic constraints into loss function terms include:

[0119] The behavior transition loss is obtained based on the behavior transition matrix and the probability distribution; the motion precursor loss is obtained based on the template matching function and the motion features; the angular velocity is obtained based on the motion parameters, and the state persistence loss is obtained based on the angular velocity; the behavior temporal logic constraint loss is obtained based on the behavior transition loss, the motion precursor loss and the state persistence loss.

[0120] The formula for calculating the loss of the behavioral temporal logic constraints is as follows:

[0121] ;

[0122] ;

[0123] ;

[0124] ;

[0125] ;

[0126] in, Indicates behavioral conversion loss. Indicates the first Frame belongs to behavior category The probability, Indicates the first Frame belongs to behavior category The probability, Indicates from behavior To behavior Transfer allowance, Indicates loss of motor prodrome. This represents the set of frame indices in which the vertical jump occurred. This represents the template matching function. This represents the motion feature vector prior to the vertical jump. This represents the rate of change of the head elevation angle within the window. This represents the sequence of vertical accelerations of the limbs and hooves within the window. This is an indicator function that returns 1 if the condition within the curly braces is true, and 0 otherwise. This indicates a continuous loss of state. Indicates behavior behavioral categories, Indicates the first Frame header rotation angular velocity, This indicates the maximum permissible angular velocity of the head under alert conditions. This represents the loss due to behavioral temporal logic constraints. All of these represent weighting coefficients.

[0127] By constructing differentiable constraint terms based on the causal structure of motion and the temporal logic of behavior, the model not only constrains the geometric form of motion but also the mechanism of motion generation (causal direction) and semantic logic (behavioral transformation rules), enabling the model to truly understand the essence of the musk deer's movement. Because the constraints are on the intrinsic causal laws of motion rather than the surface geometric form, the model can maintain stable recognition performance when facing different individuals, different shooting angles, and different ambient lighting conditions. Both the causal structure constraints and the temporal logic constraints originate from the unique biological and behavioral characteristics of the musk deer.

[0128] One or more technical solutions provided by this invention have at least the following technical effects or advantages:

[0129] 1. This method, based on the classic high-resolution network (HRNet), introduces a multi-resolution feature parallel fusion mechanism and a self-attention enhancement strategy: The multi-resolution parallel backbone network adopts the HRNet multi-resolution parallel architecture, maintaining four branches from high to low resolution. Each branch interacts with information through cross-resolution connections and feature addition, forming a unified and multi-level feature representation, realizing multi-scale feature fusion; The linear mapping module maps multiple independent linear layers into query, key, and value matrices. After feature fusion, a scaling dot product self-attention mechanism is introduced to enhance the focusing ability on key pose regions, significantly improving the localization ability of easily occluded and small-scale key points in the musk deer's pose. It is suitable for real-time pose estimation tasks in complex breeding scenarios, making it more suitable for local feature learning of musk deer pose, accurately capturing key point information of various parts of the musk deer's body, and learning the local pose semantics of the musk deer.

[0130] 2. The topological feature learning model abstracts each frame's pose as a dynamic graph structure, where nodes represent keypoints and edges are defined based on anatomical connectivity and real-time spatial proximity. Edge weights are dynamically calculated using a Gaussian kernel function to construct an adjacency matrix that adjusts with pose changes. Multi-layer Chebyshev graph convolutions are used to process the single-frame graph structure, and a K-order approximation of the normalized Laplacian operator is performed using Chebyshev polynomials. This allows the model to capture multi-level semantic information from local joint constraints to overall pose configuration and explicitly model long-range dependencies such as limb support and trunk balance. A parallel temporal modeling path is designed. Continuous multi-frame graph feature sequences are cached and input into a temporal convolutional network. A dilated convolutional kernel with a variable receptive field is used to adaptively learn motion patterns at different time scales—from subtle head movements to sustained strides—thus extracting temporal features that characterize the evolution of movement. Attached Figure Description

[0131] The accompanying drawings, which are provided to further illustrate embodiments of the invention and constitute a part of this invention, are not intended to limit the scope of the invention.

[0132] Figure 1 This is a flowchart illustrating the pose recognition method for musk deer based on multimodal semantic learning according to the present invention.

[0133] Figure 2 This is a schematic diagram of the structure of a global feature learning model;

[0134] Figure 3 This is a schematic diagram of the structure of a local feature learning model;

[0135] Figure 4 This is a schematic diagram of the topological feature learning model;

[0136] Figure 5 This is a schematic diagram of the musk deer action recognition model. Detailed Implementation

[0137] To better understand the above-mentioned objectives, features, and advantages of the present invention, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be noted that, where there is no conflict, the embodiments of the present invention and the features thereof can be combined with each other.

[0138] Many specific details are set forth in the following description in order to provide a full understanding of the invention. However, the invention may also be practiced in other ways different from those described herein, and therefore the scope of protection of the invention is not limited to the specific embodiments disclosed below.

[0139] Example 1

[0140] refer to Figures 1-5 This embodiment provides a method for musk deer pose recognition based on multimodal semantic learning, the method comprising:

[0141] Obtain the cropped image of the musk deer and global feature maps at different resolutions from the image of the musk deer to be identified;

[0142] The global feature vector is obtained based on the global feature map;

[0143] A local feature learning model is constructed, comprising a multi-resolution parallel backbone network, a linear mapping module, and a regression head network connected sequentially. The multi-resolution parallel backbone network includes several resolution branch channels with different resolutions, and the linear mapping module includes several first linear layers. In this embodiment, the multi-resolution parallel backbone network maintains four resolution branches from high to low.

[0144] High-resolution branch (1 / 4 scale): Preserves rich spatial details, suitable for precise positioning;

[0145] Medium to high resolution branch (1 / 8 scale): balances detail and semantic information;

[0146] Mid-to-low resolution branch (1 / 16 scale): Extracts mid-level semantic features;

[0147] Low-resolution branch (1 / 32 scale): Captures global context and high-level semantics.

[0148] The cropped image of the musk deer is input into the local feature learning model. Based on the multi-resolution parallel backbone network, local feature maps of each resolution branch channel are obtained. Multi-scale feature fusion is performed on the local feature maps through cross-resolution connections to obtain fused features. The linear mapping module enhances the fused features to obtain enhanced features. Based on the regression head network, the enhanced features are analyzed to obtain several channel heatmaps. The channel heatmaps are decoded to obtain several musk deer skeleton key points of each frame of the cropped image of the musk deer, as well as the two-dimensional coordinates and confidence scores corresponding to each musk deer skeleton key point.

[0149] In this embodiment, each channel corresponds to a key anatomical point of the musk deer, which includes the ears, eyes, nose, neck, shoulders, elbows, wrists, hips, knees, hooves, etc.

[0150] A topological semantic feature vector is obtained based on the two-dimensional coordinates and the confidence level, and a multi-frame skeleton sequence is obtained based on all the two-dimensional coordinates of the cropped images of the musk deer in multiple frames;

[0151] The spatiotemporal joint features of the multi-frame skeleton sequence are extracted, and the global feature vector, the topological semantic feature vector and the spatiotemporal joint features are fused to obtain a multimodal feature fusion vector. Based on the multimodal feature fusion vector and the pre-trained musk deer action recognition model, the probability distribution of different musk deer actions is obtained, and the musk deer posture recognition result is obtained based on the probability distribution.

[0152] In this embodiment, the multi-frame skeleton sequence is a 16-frame skeleton sequence, representing a set of data of 16 time units (frames) continuously extracted from the musk deer keypoint coordinates (i.e., skeleton) optimized and output by the local feature learning model, in video time sequence. Each frame in the sequence contains the complete posture of the same musk deer at a specific moment, represented as a set of two-dimensional coordinates of 15 predefined anatomical keypoints (ear, eye, nose, shoulder, elbow, wrist, hip, knee, hoof, etc.). These 16 frames are arranged in the chronological order of video capture, with continuous time intervals between adjacent frames (e.g., in a video of 30 frames / second, 16 frames cover approximately 0.53 seconds), thus recording the continuous change process of the musk deer's posture over a period of time. The system temporarily caches these coordinates, and when 16 consecutive frames are accumulated, a complete skeleton sequence is formed, which serves as the input for the subsequent action recognition module. The reason for choosing 16 frames per second is that many basic movements of the musk deer (such as a complete sniffing, a jump, or the initial phase of a straddling movement) can exhibit discriminative motion patterns within a time window of about 0.5 seconds. 16 frames (at 30fps) is sufficient to capture the key dynamic cycles of these movements.

[0153] The specific steps for obtaining enhanced features include:

[0154] The first linear layer maps the fused features into a first query matrix, a first key matrix, and a value matrix;

[0155] A first attention weight matrix is ​​obtained based on the first query matrix and the first key matrix;

[0156] The value matrix is ​​weighted based on the first attention weight matrix to obtain a weighted value matrix. The weighted value matrix is ​​then residually connected to the fused feature to obtain the enhanced feature.

[0157] The formula for calculating the first attention weight matrix is ​​as follows:

[0158] (1)

[0159] The second calculation formula for obtaining the enhanced features is:

[0160] (2)

[0161] in, This represents the first attention weight matrix. This represents the Softmax function. This represents the first query matrix. Represents the first bond matrix. Indicates transpose. Indicates the scaling factor. Represents a value matrix, Indicates enhanced features, This indicates the fusion feature.

[0162] In this embodiment, to further enhance the local feature learning model's ability to locate key points in complex aquaculture environments, the linear mapping module specifically introduces a multi-head scaling dot product self-attention mechanism. Specifically, the number of attention heads is set to 8 (i.e., hx = 8 heads). The mapped first query matrix Q, first key matrix K, and value matrix V are uniformly divided into 8 parallel subspaces along the channel dimension. The 8 attention heads perform scaling dot product attention calculations in parallel and independently, and then their output features are concatenated and enhanced. The introduced multi-head self-attention mechanism enables the model to simultaneously capture the spatial dependencies of various parts of the musk deer's body from multiple semantic perspectives (e.g., focusing specifically on limb contours, focusing specifically on local fur color and texture), effectively filtering out interference from background occlusions.

[0163] The specific steps for obtaining the global feature vector based on the global feature map include:

[0164] A global feature learning model is constructed, which includes a fusion module, a self-attention sub-module, and an output module connected in sequence. The self-attention sub-module includes a multilayer perceptron, a second linear layer, and a softmax function layer connected in sequence.

[0165] The global feature map is input into the global feature learning model, such as a module receiving a multi-scale feature map output from the YOLO11 backbone network:

[0166] P3: Higher resolution features, preserving rich details (such as the outline and texture of the musk deer's limbs);

[0167] P4: Medium-resolution features, balancing details and semantic information;

[0168] P5: Lower resolution features that contain high-level semantic and contextual information (such as the relationship between the musk deer and the scene).

[0169] The fusion module concatenates the global feature maps of different resolutions to generate multi-scale fusion features, and performs convolution and normalization processing on the multi-scale fusion features in sequence to obtain the first feature.

[0170] The multilayer perceptron and the second linear layer perform a nonlinear transformation on the first feature to generate a second query matrix and a second key matrix. The Softmax function layer normalizes the second query matrix and the second key matrix to generate a second attention weight matrix. The second attention weight matrix is ​​then multiplied element-wise with the first feature to obtain an attention-weighted feature.

[0171] The output module adds the attention-weighted feature to the first feature by performing residual addition to obtain the global feature vector.

[0172] The local feature learning model and the global feature learning model in this invention have substantial differences and complementary relationships in terms of purpose, network topology and feature representation: 1) In terms of scope and purpose, the global feature learning model focuses on macro feature extraction, aiming to capture the overall outline distribution of the musk deer and its global contextual relationship with the complex breeding enclosure environment (such as light and background); while the local feature learning model focuses on micro-level fine-grained feature mining, which is specifically used to solve the problem of accurate positioning of key anatomical parts such as the limbs and hooves of the musk deer in complex occlusion conditions such as feeding troughs and perches. 2) In terms of network structure and input / output, the global feature learning model employs a lightweight channel concatenation and element-wise multiplication attention mechanism to directly process multi-scale feature maps extracted from the backbone network, outputting a one-dimensional global feature vector. The local feature learning model, on the other hand, receives cropped images (ROIs) of individual musk deer, uses a multi-branch parallel high-resolution network as its backbone, and introduces a scaling dot product self-attention mechanism based on query (Q), key (K), and value (V) matrix mapping. Through deep multi-resolution feature interaction, it ultimately outputs a channel heatmap with spatial dimensions and decodes it into refined two-dimensional coordinates. The two work together to form a complete feature representation system from macroscopic environmental perception to microscopic precise localization.

[0173] The specific steps for obtaining the topological semantic feature vector based on the two-dimensional coordinates and the confidence level include:

[0174] A topological feature learning model is constructed, which includes a Chebyshev graph convolutional module and a temporal convolutional network connected in sequence. The Chebyshev graph convolutional module includes several parallel Chebyshev graph convolutional layers.

[0175] Based on the confidence level and the preset filtering threshold, the key points of the musk deer skeleton are screened to obtain target points. Based on the target points and the two-dimensional coordinates of the target points, a single-frame graph structure is obtained, and the single-frame graph structure is input into the topological feature learning model.

[0176] The Chebyshev graph convolutional layer processes the single-frame graph structure, performs a K-order approximation of the normalized Laplacian operator using the Chebyshev polynomial, aggregates features within the local neighborhood, and captures multi-level spatial semantic information from local joint constraints to the overall pose configuration in a hierarchical manner, and obtains spatial topological features based on the multi-level spatial semantic information.

[0177] Based on the spatial topological features, a graph feature sequence is obtained. The graph feature sequence of R consecutive frames is input into the temporal convolutional network to obtain temporal features, where R represents an integer greater than or equal to 1.

[0178] The spatial topological features and the temporal features are fused to obtain the topological semantic feature vector;

[0179] The operational formula for the Chebyshev diagram convolutional layer is as follows:

[0180] (3)

[0181] in, Indicates the first The output features of the layer Indicates the first The input features of the layer Indicates the first The weight matrix of the layer, Indicates the network layer index. This represents the Sigmoid function. Indicates the local neighborhood range. Indicates the summation index. Represents the Chebyshev polynomial. This represents the normalized Laplace matrix.

[0182] The topological feature learning model constructs a dynamic graph structure for each frame's keypoint set based on spatial proximity and anatomical connectivity, with nodes representing keypoints and edge weights reflecting connection strength. Multi-layer Chebyshev graph convolutions are employed, reducing computational overhead through sparse matrix operations, to learn topological evolution features and capture long-range dependencies. For continuous frame feature sequences, a temporal convolutional network is used to extract dynamic patterns, employing a variable receptive field design to adapt to different action timescales.

[0183] Connection strength specifically refers to the weight of the edge connecting two keypoints (nodes) in a graph structure. It comprehensively reflects the inherent association between keypoints based on anatomical structure and their spatial proximity in the current frame. Specifically, its determination involves two levels: First, based on prior anatomical knowledge of the musk deer (such as skeletal connections), a basic connection relationship is assigned to keypoint pairs with direct physiological connections (such as shoulder and elbow); second, in each frame, based on the actual two-dimensional coordinates of the keypoints, their instantaneous spatial distance is calculated using metrics such as the Gaussian kernel function; the closer the distance, the higher the connection strength. Therefore, connection strength is not a fixed value, but a weight that dynamically changes with posture, allowing the graph structure to flexibly capture changes in the relative positions of the musk deer's limbs during movement.

[0184] Dynamic patterns refer to discriminative temporal evolution patterns extracted from the topological feature sequences of consecutive frames. They describe how the spatial relationships of different body parts (corresponding graph nodes) of the musk deer coordinately change over time, thus forming specific movements. For example, walking movements exhibit a periodic pattern of alternating forward limb movements, while alert movements may present a steady-state pattern of suddenly raising the head and holding it still. Temporal convolutional networks (TCNs), through their variable receptive field design (i.e., using convolutional kernels with different dilation coefficients), can adaptively capture dynamic patterns across multiple time scales, from subtle changes in short periods (such as ear rotation) to coordinated movements over long periods (such as complete climbing sequences), thereby achieving a precise understanding of musk deer movements at different speeds and durations.

[0185] The method for obtaining the spatiotemporal joint features includes: mapping each frame of the multi-frame skeleton sequence into a single-frame skeleton graph with key points of the musk deer skeleton as nodes and the anatomical connection relationships of each key point of the musk deer skeleton as edges; combining the single-frame skeleton graphs into a multi-frame spatiotemporal skeleton graph based on the time order; extracting the temporal and spatial features of the multi-frame spatiotemporal skeleton graph; and fusing the temporal and spatial features to obtain the spatiotemporal joint features.

[0186] The musk deer action recognition model comprises a spatiotemporal graph attention network and a pooling layer connected in sequence. The spatiotemporal graph attention network contains several parallel sub-modules, each with built-in temporal graph attention and spatial graph attention mechanisms. The pooling layer includes a global pool and an attention pool connected in sequence. The spatiotemporal graph attention network maps the multi-frame skeleton sequence to a multi-frame spatiotemporal skeleton map and extracts the spatiotemporal joint features of the multi-frame spatiotemporal skeleton map. The pooling layer maps the spatiotemporal joint features to a probability distribution of musk deer actions. The spatiotemporal graph attention network in this model is similar to the linear mapping module in a local feature learning model based on an improved HRNet. The network automatically learns which key points (nodes) in each frame are more discriminative in temporal changes. Specifically, by calculating the similarity matrix between node features, attention weights are generated to dynamically enhance the feature representation of key points with large movement amplitude and significant changes (such as the head and limbs).

[0187] Example 2

[0188] Based on Embodiment 1, in this embodiment, the method further includes:

[0189] An accuracy evaluation index is obtained based on the multi-frame skeleton sequence, and the accuracy evaluation index is used to evaluate the fusion accuracy of the multi-frame skeleton sequence.

[0190] The formula for calculating the accuracy evaluation index is as follows:

[0191] (4)

[0192] in, Indicates the accuracy evaluation index. This represents the index of key points on the musk deer skeleton. Indicates the first The Euclidean distance between the predicted and true values ​​of key points on a musk deer skeleton Indicates the first Normalization factor corresponding to the key point type of the forest musk deer skeleton Indicates the target scale. This represents the Kronecker indicator function, which takes the value 1 if the condition within the parentheses is met, and 0 otherwise. Indicates the first Visible markers of key points on the skeleton of a forest musk deer.

[0193] If the OKS of the model on the validation set is lower than the set threshold (e.g., 0.75), the training strategy needs to be adjusted, such as increasing the number of occluded samples, optimizing the loss function, or adjusting the network depth.

[0194] Example 3

[0195] Based on the above embodiments, in this embodiment, the cropped image of the musk deer to be identified and global feature maps at different resolutions are obtained based on the YOLO model. The YOLO model includes a backbone network for extracting global feature maps of the musk deer image to be identified, a neck network and a detection head for fusing features, and a detection module for extracting the cropped image of the musk deer.

[0196] The front portion of the backbone network has its parameters frozen during training, while the rear portion of the backbone network, the neck network, and the detection head are fine-tuned using the target object dataset.

[0197] If the existing YOLO model is used, its training adopts a two-stage fine-tuning strategy based on transfer learning: first, the model weights pre-trained on the general COCO dataset are loaded; then, on the self-built musk deer dataset, the front feature extraction layer of the backbone network is frozen to retain the general visual representation ability, and only the Neck feature pyramid layer and the detection head layer of the network are unfrozen and fine-tuned to enable the model to quickly adapt to the morphology and scene features of the musk deer target; during training, the SGD optimizer can be used, with the initial learning rate set to 0.001 and the batch size to 32, and data augmentation techniques such as mosaic enhancement and random affine transformation are used to improve the robustness of the model.

[0198] Example 4

[0199] Based on the above embodiments, in this embodiment, the specific steps for training the musk deer action recognition model based on the musk deer dataset to obtain the musk deer dataset include:

[0200] Extract the key points of the two-dimensional skeleton of the musk deer from the video sequence and construct the skeletal topology map of the musk deer;

[0201] A prior knowledge base of musk deer kinematics is constructed, which includes several biomechanical constraints.

[0202] The biomechanical constraints are transformed into a loss function term;

[0203] The musk deer action recognition model is trained based on the musk deer skeletal topology map and the loss function term.

[0204] The biomechanical constraints include phase period constraints, contact geometry constraints, and spinal flexibility constraints.

[0205] The specific steps for constructing the phase period constraint include: constructing a continuous phase variable based on a sine function, and constructing a mapping model between the trajectory features of the key points of the limbs of the musk deer and the phase variable;

[0206] As a typical ruminant ungulate, the musk deer's limb movements follow a strict gait periodic pattern. Within a complete gait cycle, the movement of one limb can be divided into a stance phase (the hoof contacts the ground and bears weight) and a swing phase (the hoof lifts off the ground and moves forward). The duration ratio of the stance phase to the swing phase, the trajectory of joint angle changes, and angular velocity all exhibit high periodicity and predictability. This pattern is a fundamental characteristic of ruminant kinematics, significantly different from that of humans or animals such as cats and dogs, and constitutes an important priori understanding of the musk deer's locomotion pattern.

[0207] Define a continuous phase variable. This is used to characterize the temporal position within a gait cycle. To facilitate neural network learning and differential calculation, a sine function is used to model the phase, giving the phase variable smoothness and periodicity.

[0208] The formula for calculating the phase variable is:

[0209] ;

[0210] in, Indicates the first The phase variable of the frame, Represents angular frequency. Indicates the initial phase. Indicates the image frame index; and A lightweight auxiliary network (such as one consisting of 2-3 fully connected layers, with skeletal keypoint features as input and output as output) can be used. and End-to-end learning is performed on the predicted values.

[0211] Taking the left foreleg as an example, the elbow joint is selected as a representative joint, and its joint angle... Phase within a gait cycle A definite functional relationship exists. This embodiment uses Fourier series approximation to construct the mapping function;

[0212] The calculation formula for the mapping model is:

[0213] ;

[0214] in, Indicates the first Predicted joint angles for frames , and All of these represent learnable parameters. This represents the order of the Fourier series. Indicates the order index of the Fourier series;

[0215] The movement trajectory of a musk deer's hooves during contact and departure from the ground is strictly constrained by physical contact mechanics. During the stance phase, the hooves remain stationary relative to the ground (zero velocity), and slippage or penetration is not permitted; during the oscillation phase, the hoof trajectory should conform to a smooth cycloidal motion. Posture predictions that violate these physical laws (such as hooves suspending in the air or sinking below the ground) are biologically unreasonable.

[0216] First, the ground in the aquaculture scenario is modeled. Assuming the ground is planar, the ground equation in the world coordinate system can be obtained through camera extrinsic parameter calibration. (The z-axis is perpendicular to the ground and pointing upwards). For the four key hoof points Define its distance from the ground. .

[0217] Contact state determination uses a differentiable soft threshold function. The differentiable contact state function allows the constraint terms to participate in gradient backpropagation.

[0218] The specific steps for constructing the contact geometric constraints include: constructing the ground equation for the musk deer breeding scenario; obtaining the relative distance between the hooves and the ground based on the key points of the musk deer's hooves and the ground equation; constructing a soft threshold function based on the relative distance; and determining the motion state of the musk deer's hooves relative to the ground based on the soft threshold function; when... hour, Indicates the contact state; when hour, The motion state is indicated by the swinging state. If the motion state is a contact state, the velocity and acceleration of the constrained musk deer hoof in the vertical direction are close to zero. If the motion state is a swinging state, the motion trajectory of the constrained musk deer hoof follows a preset swinging trajectory curve.

[0219] The formula for calculating the soft threshold function is as follows:

[0220] (5)

[0221] (6)

[0222] in, Represents the soft threshold function. This represents the Sigmoid function. Indicates the contact threshold. Indicates the first The relative distance between frames, Indicates the first The coordinates of the key points of the musk deer's hooves in the frame in the vertical direction. Indicates smoothness, Index of key points on the hooves of the musk deer;

[0223] The spine of the musk deer is composed of multiple vertebrae and possesses a degree of flexibility, but its curvature has physiological limits. Excessive bending or abnormal twisting (such as excessive lateral bending of the body or an unnatural S-shape of the spine) does not occur during normal movement. Simultaneously, the distance between adjacent key points of the spine remains relatively stable during movement, without significant stretching or compression. This constraint can be used to distinguish between normal walking and injured / painful postures.

[0224] The specific steps for constructing the spinal flexibility constraint include: constructing a sequence of key points along the spine of the musk deer, including key points at the neck, center of the shoulders, center of the trunk, center of the hips, and root of the tail. Obtaining the Euclidean distance and direction vector between any two adjacent key points; the Euclidean distance between adjacent spinal key points should remain within a certain range and should not change drastically. Obtaining the bending angle between any three adjacent key points; the bending direction of the spine mainly occurs in the anterior-posterior direction (sagittal plane), and the lateral bending (coronal plane) and torsion (transverse plane) angles should be limited to physiological ranges; the Euclidean distance constraint is within a preset distance range, and the bending angle is less than or equal to the maximum permissible bending angle.

[0225] The specific steps for transforming the biomechanical constraints into loss function terms include:

[0226] Gait phases of the current and next frames are predicted to obtain predicted phase values, and a phase prediction consistency loss is constructed based on these predicted phase values. The phase of the current frame is then predicted using an existing auxiliary network. Furthermore, self-supervised constraints are constructed using temporal continuity. Theoretically, for two consecutive frames, the phase difference should satisfy... Therefore, a loss is constructed.

[0227] The predicted phase value of the current frame is input into the mapping model to obtain the predicted joint angle. A motion reconstruction loss is constructed based on the predicted joint angle and the actual joint angle. The predicted phase... Input the mapping function to obtain the reconstructed joint angles. The joint angles predicted by the model By making comparisons, a loss can be constructed.

[0228] The phase period constraint loss is obtained based on the phase prediction consistency loss and the motion reconstruction loss. This loss function is differentiable, and its gradient will propagate back to the feature extraction layer of the backbone network and the phase prediction network, forcing the joint motion features learned by the backbone network to conform to the periodic gait pattern.

[0229] The formula for calculating the phase period constraint loss is as follows:

[0230] (7)

[0231] (8)

[0232] (9)

[0233] in, This indicates the phase prediction consistency loss. Indicates the loss of sports reconstruction. Indicates the number of joints involved in the constraint (including the shoulder, elbow, wrist, hip, knee, and ankle of the limbs). Indicates the first In-frame joints The actual predicted joint angle, Indicates the first In-frame joints The reconstructed joint angle, This represents the node index in the joints of the limbs. This represents the phase period constraint loss. This represents the total number of frames in the musk deer video sequence. and They represent the first Frame and the Predicted phase variables of the frame, Represents angular frequency. Indicates the time interval between two adjacent frames;

[0234] The penetration depth is obtained based on the relative distance, and the positional penetration loss is obtained based on the penetration depth to prevent the hoof from penetrating the ground; the velocity and acceleration of the musk deer's hoof in the vertical direction are obtained, and the velocity loss and acceleration loss are obtained based on the velocity and acceleration, respectively; the contact geometric constraint loss is obtained based on the positional penetration loss, the velocity loss, and the acceleration loss.

[0235] The formula for calculating the contact geometric constraint loss is as follows:

[0236] (10)

[0237] (11)

[0238] (12)

[0239] (13)

[0240] (14)

[0241] (15)

[0242] (16)

[0243] in, Indicates location penetration loss. Indicates the number of key points on the hooves of the musk deer. Indicates the depth of penetration. Indicates speed loss, This represents a soft-threshold-based state mask that applies constraints only to the hoof in the contact state. This indicates the vertical velocity of the musk deer's hooves. This represents acceleration loss; further constraints are placed on the acceleration under contact conditions to approach zero in order to suppress high-frequency jitter. This indicates the vertical acceleration of the musk deer's hooves. This represents the contact geometric constraint loss. This represents the weighting coefficient of each item;

[0244] A distance consistency loss is constructed based on the Euclidean distance and the preset distance range; a bending angle loss is constructed based on the bending angle and the maximum allowable bending angle; and a spinal flexibility constraint loss is obtained based on the distance consistency loss and the bending angle loss.

[0245] The formula for calculating the spinal flexibility constraint loss is as follows:

[0246] (17)

[0247] (18)

[0248] (19)

[0249] in, This represents the distance consistency loss. Indicates the number of key points in the spine. Indicates the index of key points in the spine. This represents the Euclidean distance between two adjacent spinal key points. Indicates the maximum distance. Indicates the minimum distance. Indicates the loss of bending angle. Indicates the sagittal plane curvature angle. and These represent the maximum permissible bending angles in the sagittal plane (primary plane of motion) and the coronal plane (lateral bending), respectively. This indicates loss of spinal flexibility restraint. All represent weighting coefficients.

[0250] Phase periodic constraints primarily constrain the temporal movement patterns of the limb joints, ensuring the model understands gait cycles. Contact geometry constraints primarily constrain the physical interaction between the hooves and the environment, ensuring the model understands the physical rationality of support and swing states. Spinal flexibility constraints primarily constrain the overall posture of the trunk, ensuring the model generates a spinal configuration that conforms to biomechanical limits. These three constraints work together to form a complete three-tiered constraint system: limb movement – ​​foot-ground interaction – trunk posture. This ensures that the model is consistently guided and constrained by the unique biological movement patterns of the musk deer throughout the entire feature extraction process, from skeletal keypoint input to behavior classification output, thereby achieving a deep integration of domain knowledge and data-driven models.

[0251] Example 5

[0252] Based on the above embodiments, in this embodiment, the biomechanical constraints also include motion causal structure constraints and behavioral temporal logic constraints;

[0253] As a quadrupedal mammal, the musk deer's locomotion follows a strict law of motion propagation: trunk movement drives the limbs, rather than the limbs driving the trunk. Specifically, during normal walking or running, the swaying and tilting of the trunk (spine) are first generated by the core muscle groups, and then transmitted to the limbs through the shoulder girdle and pelvic girdle, causing the limbs to swing. Conversely, if the predicted posture shows a reversed sequence of limb movement followed by trunk movement, or if there is a false causal relationship between the limb joints and the trunk that does not conform to skeletal topology, it constitutes an abnormal pattern that violates biomechanics.

[0254] The specific steps for constructing the motion causal structure constraints include:

[0255] The skeletal key points of the forest musk deer are divided into several levels, including the core layer: key points of the trunk (such as the center of the shoulder, the center of the trunk, and the center of the hip), the conduction layer: proximal joints (such as the shoulder joint and the hip joint), and the distal layer: distal joints (such as the elbow joint, the knee joint, the wrist joint, the ankle joint, and the hoof).

[0256] Construct a directed acyclic graph to characterize the causal relationships of motion propagation. The edge direction of the directed acyclic graph is set to point from the core layer to the conduction layer and from the conduction layer to the terminal layer. For example, the motion (position, velocity, acceleration) of the shoulder center affects the shoulder joint, which in turn affects the elbow and wrist joints.

[0257] For any two adjacent layers, the motion state of the constrained target layer can be predicted from the motion state of the source layer through causal functions (i.e., forward predictor and backward predictor), and the forward prediction error should be much smaller than the backward prediction error (i.e., the error that violates the causal direction).

[0258] The specific steps for transforming the aforementioned motion causal structure constraints into loss function terms include:

[0259] Obtain the motion state splicing vector of the target layer and the source layer, construct a forward predictor and a backward predictor, and obtain causal directionality loss and causal temporal loss based on the motion state splicing vector, the forward predictor (which can be implemented by two MLP layers) and the backward predictor (which can be implemented by two MLP layers); obtain causal structure constraint loss based on the causal directionality loss and the causal temporal loss.

[0260] The formula for calculating the causal structure constraint loss is as follows:

[0261] (20)

[0262] ;(twenty one)

[0263] ;(twenty two)

[0264] ;(twenty three)

[0265] in, This represents the causal directionality loss, which forces the forward prediction error to be as small as possible while maximizing the backward prediction error, thereby compelling the feature representation learned by the model to conform to the causal direction of the core driving end. Indicates the target layer key points at the 1st level The frame's state vector includes position, velocity, and acceleration, etc. This represents the change in the state of the target layer. Indicates the forward predictor. Indicates the source layer keypoint at the 1st The state vector of a frame. This represents the amount of change in the source layer state. Indicates the reverse predictor. Represents the positive part function (i.e. ), Indicates the time delay of causal propagation. Represents the timing delay mapping function, This represents the causal temporal loss. During motion propagation, causal effects are delayed (nerve conduction and muscle contraction time). Changes in motion at the source layer should cause changes in the target layer after several frames. This constraint forces the model to learn causal propagation that conforms to physiological temporal order. Represents the causal structure constraint loss. All of these represent weighting coefficients.

[0266] The typical behavior of the musk deer does not occur randomly, but follows specific temporal logic and semantic rules. For example:

[0267] Alert behavior typically precedes escape behavior or wary feeding; vertical leaping behavior (marking territory or being startled) often begins with a rapid upward tilt of the head and simultaneous push-off from the ground by all four limbs, followed by rapid running or stationary observation; in a stationary alert state, the musk deer will remain still, only turning its head to scan its field of vision, and the speed of head turning is limited. These temporal logical rules reflect the inherent semantic structure of musk deer behavior and are important priors for distinguishing different behavioral patterns.

[0268] The specific steps for constructing the behavioral timing logic constraints include:

[0269] The temporal logic rules governing typical behaviors of the musk deer are obtained. These temporal logic rules include at least behavior transition rules, movement precursor rules, and state persistence rules. The form of these temporal logic rules is set as a precondition followed by subsequent behavior.

[0270] Rule 1 states that if the current frame is in a state of alert, the probability of escape within the next 5 frames should be higher than the threshold, and eating should not occur immediately.

[0271] Rule 2: If the current frame is a vertical jump, then the motion characteristics of rapid head tilting and simultaneous limb push-off should be detected within the previous 2 frames.

[0272] Rule 3: During the period of fixed-point vigilance, the head rotation angular velocity should be less than the preset maximum value.

[0273] Based on the behavior transition rules, a differentiable behavior transition matrix is ​​constructed to constrain the consistency between the probability distribution of behavior transitions between adjacent frames (which can be sampled from the behavior probability distribution output by the model during training) and the behavior transition matrix; based on the motion precursor rules, a template matching function is constructed to constrain the degree of matching between the motion features before a specific behavior occurs and a preset template (e.g., the rate of change of head elevation angle, vertical acceleration of limbs); based on the state duration rules, the motion parameters during the duration of a specific state are constrained not to exceed a preset threshold, such as in a fixed-point alert state, the Euler angles (yaw angle, pitch angle) of key head points are extracted, and their values ​​do not exceed the preset threshold.

[0274] The specific steps for transforming the aforementioned behavioral temporal logic constraints into loss function terms include:

[0275] The behavior transition loss is obtained based on the behavior transition matrix and the probability distribution; the motion precursor loss is obtained based on the template matching function and the motion features; the angular velocity is obtained based on the motion parameters, and the state persistence loss is obtained based on the angular velocity; the behavior temporal logic constraint loss is obtained based on the behavior transition loss, the motion precursor loss and the state persistence loss.

[0276] The formula for calculating the behavioral temporal logic constraint loss is as follows:

[0277] ;(twenty four)

[0278] (25)

[0279] (26)

[0280] (27)

[0281] (28)

[0282] in, Indicates behavioral conversion loss, Indicates the first Frame belongs to behavior category The probability, Indicates the first Frame belongs to behavior category The probability, Indicates from behavior To behavior Transfer allowance, Indicates loss of motor prodrome. This represents the set of frame indices in which the vertical jump occurred. This represents the template matching function. This represents the motion feature vector prior to the vertical jump. This represents the rate of change of the head elevation angle within the window. This represents the sequence of vertical accelerations of the limbs and hooves within the window. This is an indicator function that returns 1 if the condition within the curly braces is true, and 0 otherwise. This indicates a continuous loss of state. Indicates behavior behavioral categories, Indicates the first Frame header rotation angular velocity, This indicates the maximum permissible angular velocity of the head under alert conditions. This represents the loss due to behavioral temporal logic constraints. All of these represent weighting coefficients.

[0283] Causal structure constraints act on the intermediate feature layer of the model (spatiotemporal features of skeletal key points), guiding the graph convolutional network to learn feature representations that conform to the causal relationship of motion propagation; temporal logic constraints act on the output layer of the model (behavior probability distribution) and intermediate motion features, ensuring the temporal coherence and semantic rationality of the behavior sequence.

[0284] Both constraints are implemented through differentiable mathematical forms, and their gradients can be backpropagated to the backbone network, enabling end-to-end guidance of domain knowledge for model training.

[0285] Although preferred embodiments of the invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including both the preferred embodiments and all changes and modifications falling within the scope of the invention.

[0286] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of the claims of this invention and their equivalents, this invention also intends to include these modifications and variations.

Claims

1. A method for musk deer pose recognition based on multimodal semantic learning, characterized in that, The method includes: Obtain the cropped image of the musk deer and global feature maps at different resolutions from the image of the musk deer to be identified; The global feature vector is obtained based on the global feature map; A local feature learning model is constructed, comprising a multi-resolution parallel backbone network, a linear mapping module, and a regression head network connected in sequence. The multi-resolution parallel backbone network includes several resolution branch channels with different resolutions, and the linear mapping module includes several first linear layers. The cropped image of the musk deer is input into the local feature learning model. Based on the multi-resolution parallel backbone network, local feature maps of each resolution branch channel are obtained. Multi-scale feature fusion is performed on the local feature maps through cross-resolution connections to obtain fused features. The linear mapping module enhances the fused features to obtain enhanced features. Based on the regression head network, the enhanced features are analyzed to obtain several channel heatmaps. The channel heatmaps are decoded to obtain several musk deer skeleton key points of each frame of the cropped image of the musk deer, as well as the two-dimensional coordinates and confidence scores corresponding to each musk deer skeleton key point. A topological semantic feature vector is obtained based on the two-dimensional coordinates and the confidence level, and a multi-frame skeleton sequence is obtained based on all the two-dimensional coordinates of the cropped images of the musk deer in multiple frames; The spatiotemporal joint features of the multi-frame skeleton sequence are extracted, and the global feature vector, the topological semantic feature vector and the spatiotemporal joint features are fused to obtain a multimodal feature fusion vector. Based on the multimodal feature fusion vector and the pre-trained musk deer action recognition model, the probability distribution of different musk deer actions is obtained, and the musk deer posture recognition result is obtained based on the probability distribution.

2. The method for musk deer pose recognition based on multimodal semantic learning according to claim 1, characterized in that, The specific steps for obtaining the enhanced features include: The first linear layer maps the fused features into a first query matrix, a first key matrix, and a value matrix; A first attention weight matrix is ​​obtained based on the first query matrix and the first key matrix; The value matrix is ​​weighted based on the first attention weight matrix to obtain a weighted value matrix. The weighted value matrix is ​​then residually connected to the fused feature to obtain the enhanced feature. The formula for calculating the first attention weight matrix is ​​as follows: ； The formula for calculating the enhanced features is as follows: ； in, This represents the first attention weight matrix. This represents the Softmax function. This represents the first query matrix. Represents the first bond matrix. Indicates transpose. Indicates the scaling factor. Represents a value matrix, Indicates enhanced features, This indicates the fusion feature.

3. The method for musk deer pose recognition based on multimodal semantic learning according to claim 1, characterized in that, The specific steps for obtaining the global feature vector based on the global feature map include: A global feature learning model is constructed, which includes a fusion module, a self-attention sub-module, and an output module connected in sequence. The self-attention sub-module includes a multilayer perceptron, a second linear layer, and a softmax function layer connected in sequence. The global feature map is input into the global feature learning model. The fusion module concatenates the global feature maps of different resolutions to generate multi-scale fusion features. The multi-scale fusion features are then subjected to convolution and normalization processes to obtain the first feature. The multilayer perceptron and the second linear layer perform a nonlinear transformation on the first feature to generate a second query matrix and a second key matrix. The Softmax function layer normalizes the second query matrix and the second key matrix to generate a second attention weight matrix. The second attention weight matrix is ​​then multiplied element-wise with the first feature to obtain an attention-weighted feature. The output module adds the attention-weighted feature to the first feature by performing residual addition to obtain the global feature vector.

4. The method for musk deer pose recognition based on multimodal semantic learning according to claim 1, characterized in that, The specific steps for obtaining the topological semantic feature vector based on the two-dimensional coordinates and the confidence level include: A topological feature learning model is constructed, which includes a Chebyshev graph convolutional module and a temporal convolutional network connected in sequence. The Chebyshev graph convolutional module includes several parallel Chebyshev graph convolutional layers. Based on the confidence level and the preset filtering threshold, the key points of the musk deer skeleton are screened to obtain target points. Based on the target points and the two-dimensional coordinates of the target points, a single-frame graph structure is obtained, and the single-frame graph structure is input into the topological feature learning model. The Chebyshev graph convolutional layer performs a K-order approximation of the normalized Laplacian operator using Chebyshev polynomials, aggregates features in the local neighborhood, and captures multi-level spatial semantic information from local joint constraints to the overall pose configuration in a hierarchical manner, and obtains spatial topological features based on the multi-level spatial semantic information. Based on the spatial topological features, a graph feature sequence is obtained. The graph feature sequence of R consecutive frames is input into the temporal convolutional network to obtain temporal features, where R represents an integer greater than or equal to 1. By fusing the spatial topological features and the temporal features, the topological semantic feature vector is obtained; The operational formula for the Chebyshev diagram convolutional layer is as follows: ； in, Indicates the first The output features of the layer Indicates the first The input features of the layer Indicates the first The weight matrix of the layer, Indicates the network layer index. This represents the Sigmoid function. Indicates the local neighborhood range. Indicates the summation index. Represents the Chebyshev polynomial. This represents the normalized Laplace matrix.

5. The method for musk deer pose recognition based on multimodal semantic learning according to claim 1, characterized in that, The method for obtaining the spatiotemporal joint features includes: mapping each frame of the multi-frame skeleton sequence into a single-frame skeleton graph with key points of the musk deer skeleton as nodes and the anatomical connection relationships of each key point of the musk deer skeleton as edges; combining the single-frame skeleton graphs into a multi-frame skeleton spatiotemporal graph based on the time order; extracting the temporal and spatial features of the multi-frame skeleton spatiotemporal graph; and fusing the temporal and spatial features to obtain the spatiotemporal joint features.

6. The method for musk deer pose recognition based on multimodal semantic learning according to claim 1, characterized in that, An accuracy evaluation index is obtained based on the multi-frame skeleton sequence, and the accuracy evaluation index is used to evaluate the fusion accuracy of the multi-frame skeleton sequence. The formula for calculating the accuracy evaluation index is as follows: ； in, Indicates the accuracy evaluation index. This represents the index of key points on the musk deer skeleton. Indicates the first The Euclidean distance between the predicted and true values ​​of key points on a musk deer skeleton Indicates the first Normalization factor corresponding to the key point type of the forest musk deer skeleton Indicates the target scale. This represents the Kronecker indicator function. Indicates the first Visible markers of key points on the skeleton of a forest musk deer.

7. The method for musk deer pose recognition based on multimodal semantic learning according to claim 3, characterized in that, The YOLO model is used to obtain cropped images of musk deer and global feature maps at different resolutions from images of musk deer to be identified. The YOLO model includes a backbone network for extracting global feature maps from images of musk deer to be identified, a neck network and detection head for fusing features, and a detection module for extracting cropped images of musk deer.

8. The method for musk deer pose recognition based on multimodal semantic learning according to claim 1, characterized in that, The specific steps for training the musk deer action recognition model based on musk deer video sequences include: Extract the key points of the two-dimensional skeleton of the musk deer from the video sequence and construct the skeletal topology map of the musk deer; A prior knowledge base of musk deer kinematics is constructed, which includes several biomechanical constraints. The biomechanical constraints are transformed into a loss function term; The musk deer action recognition model is trained based on the musk deer skeletal topology map and the loss function term; The biomechanical constraints include phase period constraints, contact geometry constraints, and spinal flexibility constraints. The specific steps for constructing the phase period constraint include: constructing phase variables based on sine functions, and constructing a mapping model between the trajectory features of key points on the limbs of the musk deer and the phase variables; The formula for calculating the phase variable is: ； in, Indicates the first The phase variable of the frame, Represents angular frequency. Indicates the initial phase. Indicates the image frame index; The calculation formula for the mapping model is as follows: ； in, Indicates the first Predicted joint angles for frames. , and All of these represent learnable parameters. This represents the order of the Fourier series. Indicates the order index of the Fourier series; The specific steps for constructing the contact geometric constraints include: constructing the ground equation for the ground in the musk deer breeding scenario; obtaining the relative distance between the hoof and the ground based on the key points of the musk deer's hoof and the ground equation; constructing a soft threshold function based on the relative distance; and determining the motion state of the musk deer's hoof and the ground based on the soft threshold function. If the motion state is a contact state, the velocity and acceleration of the musk deer's hoof in the vertical direction are constrained to approach zero. If the motion state is a swinging state, the motion trajectory of the musk deer's hoof is constrained to follow a preset swinging trajectory curve. The formula for calculating the soft threshold function is as follows: ； ； in, Represents the soft threshold function. This represents the Sigmoid function. Indicates the contact threshold. Indicates the first The relative distance between frames, Indicates the first The coordinates of the key points of the musk deer's hooves in the frame in the vertical direction. Indicates smoothness, Index of key points on the hooves of the musk deer; The specific steps for constructing the spinal flexibility constraint include: constructing a sequence of key points along the spine of the musk deer, obtaining the Euclidean distance and direction vector between any two adjacent key points, and obtaining the bending angle between any three adjacent key points; the Euclidean distance constraint is within a preset distance range, and the bending angle is less than or equal to the maximum allowable bending angle.

9. The method for musk deer pose recognition based on multimodal semantic learning according to claim 8, characterized in that, The specific steps for transforming the biomechanical constraints into loss function terms include: Gait phase is predicted for the current frame and the next frame to obtain predicted phase values, and a phase prediction consistency loss is constructed based on the predicted phase values; the predicted phase values ​​of the current frame are input into the mapping model to obtain predicted joint angles, and a motion reconstruction loss is constructed based on the predicted joint angles and the actual joint angles; a phase period constraint loss is obtained based on the phase prediction consistency loss and the motion reconstruction loss. The formula for calculating the phase period constraint loss is as follows: ； ； ； in, This indicates the phase prediction consistency loss. Indicates the loss of sports reconstruction. Indicates the number of joints involved in the constraint. Indicates the first In-frame joints The actual predicted joint angle, Indicates the first In-frame joints The reconstructed joint angle, This represents the node index in the joints of the limbs. This represents the phase period constraint loss. This represents the total number of frames in the musk deer video sequence. and They represent the first Frame and the Predicted phase variables of the frame, Represents angular frequency. Indicates the time interval between two adjacent frames; The penetration depth is obtained based on the relative distance, and the positional penetration loss is obtained based on the penetration depth; the velocity and acceleration of the musk deer's hooves in the vertical direction are obtained, and the velocity loss and acceleration loss are obtained based on the velocity and acceleration, respectively; the contact geometric constraint loss is obtained based on the positional penetration loss, the velocity loss, and the acceleration loss. The formula for calculating the contact geometric constraint loss is as follows: ； ； ； ； ； ； ； in, Indicates location penetration loss. Indicates the number of key points on the hooves of the musk deer. Indicates the depth of penetration. Indicates speed loss, This represents a state mask based on a soft threshold. This indicates the vertical velocity of the musk deer's hooves. Indicates acceleration loss, This indicates the vertical acceleration of the musk deer's hooves. This represents the contact geometric constraint loss. Represents the weighting coefficient of each item; A distance consistency loss is constructed based on the Euclidean distance and the preset distance range; a bending angle loss is constructed based on the bending angle and the maximum allowable bending angle; and a spinal flexibility constraint loss is obtained based on the distance consistency loss and the bending angle loss. The formula for calculating the spinal flexibility constraint loss is as follows: ； ； ； in, This represents the distance consistency loss. Indicates the number of key points in the spine. Indicates the index of key points in the spine. This represents the Euclidean distance between two adjacent spinal key points. Indicates the maximum distance. Indicates the minimum distance. Indicates the loss of bending angle. Indicates the sagittal plane curvature angle. and These represent the maximum permissible curvature angles in the sagittal and coronal planes, respectively. This indicates loss of spinal flexibility restraint. All of these represent weighting coefficients.

10. The method for musk deer pose recognition based on multimodal semantic learning according to claim 9, characterized in that, The biomechanical constraints also include motion causal structure constraints and behavioral temporal logic constraints. The specific steps for constructing the motion causal structure constraints include: The skeletal key points of the musk deer are divided into several layers, including a core layer, a conduction layer, and a terminal layer. A directed acyclic graph (DAG) is constructed to characterize the causal relationship of motion propagation. The edge direction of the DAG is set from the core layer to the conduction layer and from the conduction layer to the terminal layer. For any two adjacent layers, the motion state of the constrained target layer is predicted by the motion state of the source layer through a causal function, and the forward prediction error is much smaller than the backward prediction error. The specific steps for transforming the aforementioned motion causal structure constraints into loss function terms include: Obtain the motion state splicing vector of the target layer and the source layer, construct a forward predictor and a backward predictor, and obtain causal directionality loss and causal temporal loss based on the motion state splicing vector, the forward predictor and the backward predictor; obtain causal structure constraint loss based on the causal directionality loss and the causal temporal loss. The formula for calculating the causal structure constraint loss is as follows: ； ； ； ； in, Indicates causal directionality loss. Indicates the target layer key points at the 1st level The state vector of a frame. This represents the change in the state of the target layer. Indicates the forward predictor. Indicates the source layer keypoint at the 1st The state vector of a frame. This represents the amount of change in the source layer state. Indicates the reverse predictor. Denotes the positive part function, Indicates the time delay of causal propagation. Represents the timing delay mapping function, Indicates causal time loss, Represents the causal structure constraint loss. All represent weighting coefficients; The specific steps for constructing the behavioral timing logic constraints include: The temporal logic rules governing typical behaviors of the musk deer are obtained. These temporal logic rules include at least behavior transition rules, movement precursor rules, and state persistence rules. The form of these temporal logic rules is set as a precondition followed by subsequent behavior. Based on the behavior transition rules, a behavior transition matrix is ​​constructed to constrain the consistency between the probability distribution of behavior transitions between adjacent frames and the behavior transition matrix; based on the motion precursor rules, a template matching function is constructed to constrain the degree of matching between the motion features before a specific behavior occurs and a preset template; based on the state persistence rules, the motion parameters during the persistence of a specific state are constrained not to exceed a preset threshold. The specific steps for transforming the aforementioned behavioral temporal logic constraints into loss function terms include: The behavior transition loss is obtained based on the behavior transition matrix and the probability distribution; the motion precursor loss is obtained based on the template matching function and the motion features; the angular velocity is obtained based on the motion parameters, and the state persistence loss is obtained based on the angular velocity; the behavior temporal logic constraint loss is obtained based on the behavior transition loss, the motion precursor loss and the state persistence loss. The formula for calculating the behavioral temporal logic constraint loss is as follows: ； ； ； ； ； in, Indicates behavioral conversion loss, Indicates the first Frame belongs to behavior category The probability, Indicates the first Frame belongs to behavior category The probability, Indicates from behavior To behavior Transfer allowance, Indicates loss of motor prodrome. This represents the set of frame indices in which the vertical jump occurred. This represents the template matching function. This represents the motion feature vector prior to the vertical jump. This represents the rate of change of the head elevation angle within the window. This represents the sequence of vertical accelerations of the limbs and hooves within the window. Indicates an indicator function, This indicates a continuous loss of state. Indicates behavior behavioral categories, Indicates the first Frame header rotation angular velocity, This indicates the maximum permissible angular velocity of the head under alert conditions. This represents the loss due to behavioral temporal logic constraints. All of these represent weighting coefficients.

Images