Dairy cow daily behavior deep learning identification method based on skeleton features
By constructing the MAR-GCN model and combining multiple attention and multi-scale temporal graph convolution, the problems of cross-site correlation and fine-grained feature extraction in dairy cow behavior recognition were solved, achieving efficient dairy cow behavior monitoring and recognition.
Patent Information
- Application Number
- CN202511771736.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-11-28
- Publication Date
- 2026-02-27
AI Technical Summary
Existing technologies struggle to capture long-distance relationships across body parts in dairy cow behavior recognition, lack fine-grained feature extraction capabilities, cannot simultaneously handle multi-scale temporal modeling, and lack standardized datasets, resulting in low recognition accuracy and difficulty in deployment.
We construct the MAR-GCN model, which combines a spatial semantic feature aggregation graph convolution module and a multi-attention mechanism with a multi-scale temporal graph convolution module. By utilizing keypoints, skeletal features, and motion features, we achieve cross-part association, fine-grained feature extraction, and multi-scale temporal modeling, and build a high-quality dataset.
It improves the accuracy and efficiency of dairy cow behavior recognition, effectively distinguishes similar behaviors, and solves the problems of inaccurate dynamic behavior recognition and missing datasets in traditional methods, thus achieving efficient dairy cow behavior monitoring.
Smart Images
Figure CN121582968A_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of computer vision technology, and specifically relates to a deep learning method for recognizing the daily behavior of dairy cows based on skeleton features. Background Technology
[0002] The daily behaviors of dairy cows (such as standing, walking, lying down, feeding, drinking, and lameness) directly reflect their physiological health status. As modern animal husbandry transforms towards large-scale and intelligent operations, accurate monitoring of individual dairy cow behavior has become crucial for improving breeding efficiency, ensuring animal welfare, and providing early disease warnings. Farms primarily rely on manual inspections, which suffers from drawbacks such as high labor intensity, strong subjectivity, and difficulty in achieving continuous, 24 / 7 monitoring.
[0003] Existing recognition methods based on RGB images or videos suffer from the problems of ignoring dynamic features of actions and requiring substantial computational resources, resulting in recognition models that cannot meet the needs of actual production. Skeleton-based behavior recognition methods, which model by extracting the coordinate sequences of key joints of an organism, can provide more effective motion features. However, when applied to dairy cow behavior recognition, skeleton-based behavior recognition methods face the following technical bottlenecks:
[0004] 1. Traditional graph convolutional neural networks (GCNs) are often used, which rely solely on local adjacency matrices to aggregate features of adjacent nodes. While they can capture local key relationships, they struggle to establish long-distance dependencies across different parts. For example, the "lame" behavior of dairy cows involves not only leg movements but is also closely related to head swaying and back arching; traditional methods struggle to capture these globally coordinated features.
[0005] 2. When faced with behaviors with highly similar action patterns (such as "normal walking" and "slight limping", "head down to eat" and "head down to drink"), existing models lack fine-grained feature extraction capabilities and are prone to confusion.
[0006] 3. The time spans of different behaviors vary greatly (such as instantaneous posture adjustment and long-term lying down), and convolution operations on a single time scale are difficult to simultaneously take into account the feature modeling of short-term explosive actions and long-term periodic actions.
[0007] 4. The lack of standardized cow skeleton datasets that cover a variety of daily behaviors limits the training and validation of related algorithms.
[0008] Therefore, there is an urgent need for a deep learning-based method for recognizing the daily behavior of dairy cows based on skeleton features, which supports long-distance node association capture, fine-grained action feature extraction, and multi-scale temporal modeling to meet the practical needs of "accuracy, efficiency, and deployability" in modern smart farming scenarios. Summary of the Invention
[0009] The purpose of this invention is to provide a deep learning-based method for recognizing daily behaviors of dairy cows based on skeleton features, comprising the following steps:
[0010] Collect video samples of cow behavior, process the video samples frame by frame, detect and locate key joints of the cow's body in each frame, and construct a cow skeleton behavior dataset based on the key joints; the cow skeleton behavior dataset includes four modal data streams, namely: joint features, skeleton features, joint motion features, and skeleton motion features.
[0011] The MAR-GCN model is constructed, which includes nine stacked graph convolutional neural network blocks. Each graph convolutional neural network block includes a spatial semantic feature aggregation graph convolutional module and a cascaded hybrid multi-scale temporal graph convolutional module. A multi-attention module is set after every three graph convolutional neural network blocks. The MAR-GCN model finally obtains the final cow behavior category through global average pooling and fully connected layers.
[0012] Set the parameters of the MAR-GCN model, and train four independent MAR-GCN models for the four modal data streams respectively; adopt a decision-level fusion strategy, and perform a weighted summation of the Softmax probability scores output by the four independent MAR-GCN models, and select the category with the highest probability as the final cow behavior recognition result.
[0013] The process involves collecting video samples of cow behavior, processing each video sample frame by frame, detecting and locating key joints of the cow's body in each frame, and constructing a cow skeletal behavior dataset based on these key joints.
[0014] A pre-trained and fine-tuned YOLOv8-x pose estimation model was used to process the collected videos of cow behavior frame by frame to detect and locate the cow's body. Identify key nodes and obtain the two-dimensional coordinates of each key node. and confidence score;
[0015] The keypoint features are defined as the original coordinates. In the formula This represents the coordinates of the nth frame and the nth joint in the skeleton sequence in C-dimensional space.
[0016] The skeletal features are defined as the vector differences between adjacent joints. , used to characterize the length and direction of limbs;
[0017] The joint motion features are defined as the inter-frame difference of the coordinates. ;
[0018] The skeletal motion features are defined as the inter-frame variations of the skeletal vectors.
[0019] The spatial semantic feature aggregation graph convolution module includes four parallel branches. The first three branches process the three edge types of the adjacency matrix of the traditional graph structure—nodes themselves, centripetal connections, and centrifugal connections—to extract local features. The fourth branch utilizes the semantic hyperedge association matrix. To aggregate global semantic features;
[0020] The spatial semantic feature aggregation graph convolution module introduces an adaptive adjacency matrix in each branch. It generates the input features by projecting and calculating similarity through two learnable 1D convolutional layers, i.e. ;
[0021] The output features of the spatial semantic feature aggregation graph convolution module It is obtained by weighted aggregation of physical connections, semantic connections, and adaptive connections, achieving deep fusion of spatial features, and the formula is expressed as:
[0022] ;
[0023] In the formula, This represents a predefined static physical connection or semantic hyperedge. As input features, These are learnable weight parameters. It is a set of edges consisting of three types: the node itself, centripetal connections, and centrifugal connections. This is the semantic hyperedge correlation matrix. is the activation function, and k is a specific adjacency matrix or incidence matrix.
[0024] The method for obtaining the semantic hyperedge association matrix is as follows: based on the movement physiology characteristics of dairy cows, all 16 joints are divided into 6 subsets and correspond to 6 parts: "head", "trunk", "right forelimb", "left forelimb", "right hindlimb" and "left hindlimb". The joints of each part constitute a hyperedge; the parts with cooperative movement relationship are defined as the "meta-action" set.
[0025] The multi-attention module adopts a cascaded structure, recalibrating features sequentially in the three dimensions of time, space, and channel. The temporal attention branch calculates temporal weights through a multi-head self-attention mechanism to filter out key frames where the behavior occurs. The spatial attention branch, based on temporal weighting, focuses on the key joints in each frame that contribute the most to behavior discrimination. The channel attention branch uses one-dimensional convolution to capture the dependencies between channels.
[0026] The attention mechanism is as follows:
[0027] SCA(F)=F⊗σ(MHA(GAP(F),n));
[0028] In the formula, F is the input feature tensor, SCA is the channel attention module, GAP is global average pooling, and MHA is the multi-head self-attention mechanism. Let σ be the number of heads in the self-attention mechanism, σ be the activation function, and ⊗ be the element-wise multiplication.
[0029] The multi-attention module is represented by a cascaded computational structure as follows:
[0030] ;
[0031] In the formula, F is the input feature tensor. A cascaded structure is used for the multi-attention module. , , These represent implementations achieved by performing MHA on the channel dimension, spatial dimension, and time dimension, and performing GAP on the other two dimensions.
[0032] The hybrid multi-scale temporal graph convolution module performs temporal modeling; it sets up branches with different receptive fields in parallel along the temporal dimension, including: one branch uses parallel dilated convolution operations with smaller kernels and two different dilation rates to capture short, fast instantaneous action details; another branch uses parallel dilated convolution operations with larger kernels and two different dilation rates to expand the receptive field along the temporal dimension, thereby capturing long-term behavioral dependencies; and a max-pooling branch is used to preserve salient features.
[0033] By fusing the outputs of multi-scale branches, the hybrid multi-scale temporal graph convolution module can generate spatiotemporal feature representations containing rich dynamic information, ensuring that the model can both identify instantaneous pose adjustments and accurately judge persistent long-term behaviors.
[0034] The step of setting the MAR-GCN model parameters and training four independent MAR-GCN models for the four modal data streams includes: training the model based on the PyTorch deep learning framework; using a stochastic gradient descent optimizer with a momentum coefficient set to 0.9 and a weight decay coefficient set to 0.001.
[0035] The cross-entropy loss function was selected; the training strategy adopted a step-by-step learning rate decay, with the initial learning rate set to 0.1. During the training process, the learning rate was decayed to 0.1 times its original value every 20 rounds, and the total number of training rounds was set to 100 rounds; the batch size was set to 16.
[0036] Another object of the present invention is to provide a computer device, including a memory and a processor, wherein the memory stores a computer program, and when the processor runs the computer program stored in the memory, the processor executes the deep learning recognition method for daily cow behavior based on skeleton features according to the present invention.
[0037] Another object of the present invention is to provide a computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the processor performs the deep learning recognition method for daily cow behavior based on skeleton features according to the present invention.
[0038] The beneficial effects of this invention are as follows:
[0039] This invention applies a deep learning-based method for recognizing daily cow behaviors based on skeleton features, proposing a method for recognizing cow skeleton behaviors that integrates semantic action segmentation, multi-dimensional attention mechanisms, and multi-scale temporal modeling. This method addresses the technical problems of existing cow behavior recognition models, such as low accuracy in dynamic behavior recognition, low distinguishability between similar behaviors, and inability to be deployed practically. Specific beneficial effects are as follows:
[0040] 1. Solve the problem of difficult collaborative modeling of long-distance key points: By constructing semantic hyperedges, the limitations of local adjacency matrices are broken, and the global collaborative relationships of different parts of the cow's body (such as head, trunk, and limbs) under specific behaviors are captured.
[0041] 2. Solve the problem of difficulty in distinguishing differences in similar behavioral features: Utilize a multi-attention mechanism to focus on fine-grained features from three dimensions: time, space, and channel, to improve the recognition accuracy of easily confused behaviors such as limping and drinking.
[0042] 3. Solve the problem of inaccurate capture of complex temporal dynamics: By using hybrid multi-scale temporal graph convolution, comprehensive extraction of behavioral features at different time scales can be achieved.
[0043] 4. Address the issue of missing datasets: Construct a high-quality, standardized dairy cow skeleton dataset to provide fundamental support for recognizing daily dairy cow behaviors. Attached Figure Description
[0044] Figure 1 This is a flowchart illustrating a deep learning method for recognizing daily behaviors of dairy cows based on skeleton features, according to the present invention.
[0045] Figure 2 This is a schematic diagram of the division of the cow skeleton parts in an embodiment of the present invention, wherein (a) is the definition method of the joint points of the cow skeleton in the self-built dataset CSBD, and (b) is the division method of the joint points in the cow skeleton;
[0046] Figure 3This is a schematic diagram of the semantic hyperedge construction of cow behavior metaactions in an embodiment of the present invention, wherein (a) is a behavior with high coordination between the trunk and head, (b) is a cooperative dependency behavior between the head and forelimbs, (c) is a cooperative dependency behavior between the head and hindlimbs, (d) is a cooperative dependency behavior between the trunk and the right forelimb, (e) is a cooperative dependency behavior between the trunk and the right hindlimb, (f) is a cooperative dependency behavior between the trunk and the left forelimb, (g) is a cooperative dependency behavior between the trunk and the left hindlimb, and (h) is a cooperative dependency behavior between the four limbs.
[0047] Figure 4 This is a schematic diagram of the MAR-GCN model framework according to an embodiment of the present invention;
[0048] Figure 5 This is a schematic diagram of the spatial semantic feature aggregation graph convolution module according to an embodiment of the present invention;
[0049] Figure 6 This is a schematic diagram of the structure of the multi-attention module according to an embodiment of the present invention;
[0050] Figure 7 This is a schematic diagram of the structure of the hybrid multi-scale time-graph convolution module according to an embodiment of the present invention;
[0051] Figure 8 This is a schematic diagram illustrating the visualization effect of the MAR-GCN model in recognizing limp behavior according to an embodiment of the present invention. Detailed Implementation
[0052] This invention provides a deep learning method for recognizing daily behaviors of dairy cows based on skeleton features. The invention will be further described in detail below with reference to the accompanying drawings.
[0053] This invention is primarily aimed at practical applications such as intelligent ranch management, health monitoring, and disease early warning, aiming to improve the level of automation and precision management in animal husbandry. This invention proposes a method for recognizing the daily behavior of dairy cows based on skeletal features. The core of the method lies in constructing a fine-grained dairy cow skeletal behavior recognition model MAR-GCN (Multi-Attention Recognition Graph Convolutional Network) based on multiple attention, and designing a complete process from data input to feature extraction around this model.
[0054] like Figure 1 The embodiment of the present invention shown discloses a deep learning method for recognizing daily behaviors of dairy cows based on skeleton features, including the following steps:
[0055] Collect video samples of cow behavior, process the video samples frame by frame, detect and locate key joints of the cow's body in each frame, and construct a cow skeleton behavior dataset based on the key joints; the cow skeleton behavior dataset includes four modal data streams, namely: joint features, skeleton features, joint motion features, and skeleton motion features.
[0056] The MAR-GCN model is constructed, which includes nine stacked graph convolutional neural network blocks. Each graph convolutional neural network block includes a spatial semantic feature aggregation graph convolutional module and a cascaded hybrid multi-scale temporal graph convolutional module. A multi-attention module is set after every three graph convolutional neural network blocks. The MAR-GCN model finally obtains the final cow behavior category through global average pooling and fully connected layers.
[0057] Set the parameters of the MAR-GCN model, and train four independent MAR-GCN models for the four modal data streams respectively; adopt a decision-level fusion strategy, and perform a weighted summation of the Softmax probability scores output by the four independent MAR-GCN models, and select the category with the highest probability as the final cow behavior recognition result.
[0058] The aforementioned deep learning-based method for recognizing daily behaviors of dairy cows based on skeletal features involves a combination of graph convolutional networks, semantic action segmentation, and multiple attention mechanisms. This method is suitable for high-precision recognition and dynamic monitoring of typical behaviors of dairy cows, such as standing, walking, lying down, drinking, feeding, and lameness, in a pasture environment. The following sections provide detailed explanations of each step.
[0059] The division of the cow skeleton in this embodiment is as follows: Figure 2 As shown, (a) is the definition method of the joints of the cow skeleton in the self-built dataset CSBD, and (b) is the division method of the joints in the cow skeleton.
[0060] The process involves collecting video samples of cow behavior, processing each video sample frame by frame, detecting and locating key joints of the cow's body in each frame, and constructing a cow skeletal behavior dataset based on these key joints.
[0061] A pre-trained and fine-tuned YOLOv8-x pose estimation model was used to process the collected videos of cow behavior frame by frame to detect and locate the cow's body. Identify key nodes and obtain the two-dimensional coordinates of each key node. and confidence score;
[0062] The keypoint features are defined as the original coordinates. In the formula This represents the coordinates of the nth frame and the nth joint in the skeleton sequence in C-dimensional space.
[0063] The skeletal features are defined as the vector differences between adjacent joints. , used to characterize the length and direction of limbs;
[0064] The joint motion features are defined as the inter-frame difference of the coordinates. ;
[0065] The skeletal motion features are defined as the inter-frame variations of the skeletal vectors.
[0066] In this embodiment, YOLOv8-based keypoint extraction technology is used to construct the basic input data for the model. The pre-trained and fine-tuned YOLOv8-x pose estimation model is used to process the collected cow behavior videos frame by frame, detecting and locating the cow's body... Obtain the two-dimensional coordinates of key nodes. And confidence score.
[0067] To fully extract the implicit information in the skeleton sequence, this invention constructs four complementary data modalities: joint features, i.e., original coordinates. ,in This represents the coordinates of the nth frame and the nth joint in the skeleton sequence in C-dimensional space; the bone feature is defined as the vector difference between adjacent joints. Used to characterize the length and orientation of limbs; joint motion features, i.e., the inter-frame difference of coordinates. The data streams from these four modalities are fed into the network, providing rich foundational information for subsequent feature learning. This includes bone motion features, which are the inter-frame variations of bone vectors.
[0068] In this embodiment, the semantic hyperedge constructs the cow behavior meta-action as follows: Figure 3 As shown, this includes the construction of hyperedges between different body parts and the division of meta-actions. (a) represents behaviors with high coordination between the trunk and head; (b) represents cooperative behaviors between the head and forelimbs; (c) represents cooperative behaviors between the head and hindlimbs; (d) represents cooperative behaviors between the trunk and the right forelimb; (e) represents cooperative behaviors between the trunk and the right hindlimb; (f) represents cooperative behaviors between the trunk and the left forelimb; (g) represents cooperative behaviors between the trunk and the left hindlimb; and (h) represents cooperative behaviors between all four limbs. Figure 3In the diagram, H represents the head, B represents the trunk, RFL represents the right forelimb, RHL represents the right hindlimb, LFL represents the left forelimb, and LHL represents the left hindlimb. (a)-(g) represent a meta-action, specifically including: (a) representing behaviors with high coordination between the trunk and head, such as the cow's arched back and nodding in the "lame" behavior. Similarly, (b) and (c) represent behaviors focusing on the synergistic dependence between the head and forelimbs, such as "feeding" and "drinking". (d)-(g) represent actions requiring full-body coordination, such as "standing" and "lying down". Finally, (h) represents capturing the spatiotemporal characteristic changes of the limbs during "walking".
[0069] The spatial semantic feature aggregation graph convolution module includes four parallel branches. The first three branches process the three edge types of the adjacency matrix of the traditional graph structure—nodes themselves, centripetal connections, and centrifugal connections—to extract local features. The fourth branch utilizes the semantic hyperedge association matrix. To aggregate global semantic features;
[0070] The spatial semantic feature aggregation graph convolution module introduces an adaptive adjacency matrix in each branch. It generates the input features by projecting and calculating similarity through two learnable 1D convolutional layers, i.e. ;
[0071] The output features of the spatial semantic feature aggregation graph convolution module It is obtained by weighted aggregation of physical connections, semantic connections, and adaptive connections, achieving deep fusion of spatial features, and the formula is expressed as:
[0072] ;
[0073] In the formula, This represents a predefined static physical connection or semantic hyperedge. As input features, These are learnable weight parameters. It is a set of edges consisting of three types: the node itself, centripetal connections, and centrifugal connections. This is the semantic hyperedge correlation matrix. is the activation function, and k is a specific adjacency matrix or incidence matrix.
[0074] In this embodiment, a Spatial Semantic Feature Aggregation Graph Convolutional Module (SSA-GC) is designed to perform specific spatial feature extraction operations. The SSA-GC module contains four parallel branches. The first three branches process the three edge types of the adjacency matrix of the traditional graph structure—nodes themselves, centripetal connections, and centrifugal connections—to extract local features, respectively. The fourth branch utilizes the semantic hyperedge association matrix. This module aggregates global semantic features. Furthermore, to compensate for the shortcomings of the predefined graph structure, it introduces an adaptive adjacency matrix in each branch. It generates the input features by projecting and calculating similarity through two learnable 1D convolutional layers, i.e. Ultimately, the output characteristics of the SSA-GC module... It is obtained by weighted aggregation of physical connections, semantic connections, and adaptive connections, achieving deep fusion of spatial features, expressed by the formula as follows:
[0075]
[0076] in, This represents a predefined static physical connection or semantic hyperedge. As input features, These are learnable weight parameters. It is a set of three types of edges: the node itself, centripetal connections, and centrifugal connections.
[0077] The method for obtaining the semantic hyperedge association matrix is as follows: based on the movement physiology characteristics of dairy cows, all 16 joints are divided into 6 subsets and correspond to 6 parts: "head", "trunk", "right forelimb", "left forelimb", "right hindlimb" and "left hindlimb". The joints of each part constitute a hyperedge; the parts with cooperative movement relationship are defined as the "meta-action" set.
[0078] In this embodiment, to overcome the limitation of traditional graph convolution focusing only on physically connected neighbors, this invention proposes a semantic hyperedge action partitioning strategy. This strategy, based on the movement physiology of dairy cows, breaks the constraints of physical connections in the skeleton. First, all 16 joints are divided into 6 subsets, corresponding to 6 parts: "head," "trunk," "right forelimb," "left forelimb," etc., with each part's joints forming a hyperedge. Then, parts with cooperative movement relationships are defined as a set of "meta-actions." For example, the head and forelimbs are grouped into one set of semantic hyperedges to capture head-hoof coordination features during "drinking" or "grazing"; the limbs are grouped into another set to capture gait coordination features during "walking." A semantic hyperedge association matrix is constructed. The model can establish direct connections across parts at the logical level, thereby introducing high-level semantic prior knowledge in the spatial modeling stage and enhancing the model's ability to understand the global structure of complex behaviors.
[0079] The multi-attention module adopts a cascaded structure, recalibrating features sequentially in the three dimensions of time, space, and channel. The temporal attention branch calculates temporal weights through a multi-head self-attention mechanism to filter out key frames where the behavior occurs. The spatial attention branch, based on temporal weighting, focuses on the key joints in each frame that contribute the most to behavior discrimination. The channel attention branch uses one-dimensional convolution to capture the dependencies between channels.
[0080] The attention mechanism is as follows:
[0081] SCA(F)=F⊗σ(MHA(GAP(F),n));
[0082] In the formula, F is the input feature tensor, SCA is the channel attention module, GAP is global average pooling, and MHA is the multi-head self-attention mechanism. Let σ be the number of heads in the self-attention mechanism, σ be the activation function, and ⊗ be the element-wise multiplication.
[0083] The multi-attention module is represented by a cascaded computational structure as follows:
[0084] ;
[0085] In the formula, F is the input feature tensor. A cascaded structure is used for the multi-attention module. , , These represent implementations achieved by performing MHA on the channel dimension, spatial dimension, and time dimension, and performing GAP on the other two dimensions.
[0086] In this embodiment, to further enhance the model's ability to distinguish similar behaviors, a multi-attention module (TAC) is embedded in the network. This module adopts a cascaded structure, recalibrating features sequentially in three dimensions: time, space, and channel. First, the temporal attention branch calculates temporal weights using a multi-head self-attention mechanism to filter out key frames where the behavior occurs. Second, the spatial attention branch, based on temporal weighting, focuses on the key nodes in each frame that contribute the most to behavior discrimination. Finally, the channel attention branch uses one-dimensional convolution to capture the dependencies between channels. This cascaded computation method... It can filter redundant background noise layer by layer, enabling the model to accurately focus on fine-grained features such as gait differences such as "limping" or micro-head movements such as "drinking".
[0087] The hybrid multi-scale temporal graph convolution module performs temporal modeling; it sets up branches with different receptive fields in parallel along the temporal dimension, including: one branch uses parallel dilated convolution operations with smaller kernels and two different dilation rates to capture short, fast instantaneous action details; another branch uses parallel dilated convolution operations with larger kernels and two different dilation rates to expand the receptive field along the temporal dimension, thereby capturing long-term behavioral dependencies; and a max-pooling branch is used to preserve salient features.
[0088] By fusing the outputs of multi-scale branches, the hybrid multi-scale temporal graph convolution module can generate spatiotemporal feature representations containing rich dynamic information, ensuring that the model can both identify instantaneous pose adjustments and accurately judge persistent long-term behaviors.
[0089] In this embodiment, to address the issue of significant differences in the duration of cow behaviors, the present invention constructs a Hybrid Multi-Scale Temporal Graph Convolutional Module (HMT-GC) for temporal series modeling. The structure of the Hybrid Multi-Scale Temporal Graph Convolutional Module (HMT-GC) is as follows: Figure 7 As shown, the HMT-GC module achieves multi-scale extraction and modeling of short- and long-term temporal features of cow behavior by fusing convolutional branches with different kernel sizes and dilation rates, as well as a max-pooling branch. The HMT-GC module sets up branches with different receptive fields in parallel along the temporal dimension: one branch uses parallel dilated convolution operations with smaller kernels and two different dilation rates to capture short, rapid instantaneous action details; another branch uses parallel dilated convolution operations with larger kernels and two different dilation rates to expand the receptive field along the temporal dimension, thereby capturing long-term behavioral dependencies; it also includes a max-pooling branch to preserve salient features. By fusing the outputs of these multi-scale branches, the HMT-GC module can generate spatiotemporal feature representations containing rich dynamic information, ensuring that the model can recognize both instantaneous posture adjustments and accurately determine continuous long-term behaviors.
[0090] The step of setting the MAR-GCN model parameters and training four independent MAR-GCN models for the four modal data streams includes:
[0091] The model was trained using the PyTorch deep learning framework; a stochastic gradient descent optimizer was used, with the momentum coefficient set to 0.9 and the weight decay coefficient set to 0.001.
[0092] The cross-entropy loss function was selected; the training strategy adopted a step-by-step learning rate decay, with the initial learning rate set to 0.1. During the training process, the learning rate was decayed to 0.1 times its original value every 20 rounds, and the total number of training rounds was set to 100 rounds; the batch size was set to 16.
[0093] The MAR-GCN model framework in this embodiment is as follows: Figure 4 As shown, the model achieves accurate identification of fine-grained cow behavior by stacking a backbone network containing spatial and hybrid multi-scale temporal map convolutions and embedding a TAC (Tracking Acoustic Array). The MAR-GCN model comprises nine stacked GCN blocks, each containing an SSA-GC module and a cascaded HMT-GC module. A TAC module is placed after every three GCN blocks, for a total of two TAC modules. The model finally obtains the final cow behavior category through global average pooling and fully connected layers. A complete end-to-end cow behavior recognition technology framework is implemented, relying on a deep learning framework, through a process of data acquisition, multimodal feature construction, model network construction, and training optimization. The following specific embodiment verifies the technical effectiveness of the method disclosed in this invention.
[0094] First, in the dataset construction and preprocessing stage, this embodiment selects a real dairy farming environment as the data collection site. Using a handheld high-definition camera (such as a Canon EOS R6), video data covering six typical daily behaviors of dairy cows—standing, walking, lying down, drinking, feeding, and lameness—was collected under conditions of sufficient lighting and avoiding excessive occlusion. The video parameters were uniformly set to a resolution of 1920×1080 and a frame rate of 30fps. Subsequently, the original videos were cleaned and edited, blurry or targetless segments were removed, and the videos were cropped to a uniform length of 3 seconds, resulting in a total of 1100 valid data segments. In order to extract skeletal features from the videos, this embodiment uses a YOLOv8-x pose estimation model pre-trained with manually labeled data to perform frame-by-frame inference on each video sample, detecting and outputting the two-dimensional coordinates of 16 key joints of each dairy cow's body (head, neck, spine, left / right front / hind leg root, left / right front / hind knee, left / right front / hind hoof, and tailbone). The confidence scores were obtained, and multiple data streams were further constructed based on the original joint coordinates, including skeletal features, joint motion features, and skeletal motion features. These four modalities were encapsulated into tensors of a unified format and used as input to subsequent models.
[0095] Secondly, in the model construction phase, this embodiment builds the MAR-GCN model. The backbone network of this model consists of 9 stacked graph convolutional (GCN) units, with the number of feature channels increasing sequentially with the depth of the layers (64, 64, 64, 128, 128, 128, 256, 256, 256 respectively). Each GCN unit adopts a serial structure and contains one SSA-GC module and one HMT-GC module.
[0096] In this embodiment, the structure of the Spatial Semantic Feature Aggregation Graph Convolutional Module (SSA-GC) is as follows: Figure 5As shown, the physical skeleton connection is combined with cross-part semantic hyperedges through four parallel branches, and an adaptive graph convolution mechanism is introduced to effectively aggregate the overall process of local and global spatial features of the cow skeleton. In the SSA-GC module, a semantic hyperedge association matrix is predefined (dividing the body into 6 parts: head, trunk, right forelimb, left forelimb, right hindlimb, and left hindlimb, and 8 meta-actions), and a learnable adaptive adjacency matrix is introduced to model the collaborative relationship of long-distance joints across parts.
[0097] In the HMT-GC module, three parallel branches are designed: one branch uses two parallel dilated convolutions with kernels of 3 and dilation rates of 1 and 2 to capture short-term actions; another branch uses two parallel dilated convolutions with kernels of 5 and dilation rates of 1 and 2 to capture long-term dependencies; and the third branch uses a max pooling layer to preserve salient features. Finally, the three are fused to adapt to the time span of different behaviors.
[0098] In this embodiment, the structure of the multiple attention module (TAC) is as follows: Figure 6 As shown, a multi-dimensional enhancement and fine-grained modeling of the features of the cow skeleton sequence is achieved by sequentially cascading attention mechanisms in three dimensions: time, space, and channel. To enhance the extraction capability of fine-grained features, this embodiment inserts TAC modules after the GCN units in layers 3 and 6 of the backbone network. Within the TAC module, three attention sub-modules are sequentially connected in the order of "time-space-channel," filtering out keyframes, key joints, and key feature channels in the behavioral sequence layer by layer.
[0099] Finally, in the model training and parameter setting phase, this embodiment uses the PyTorch deep learning framework for model training. A stochastic gradient descent (SGD) optimizer is employed, with a momentum coefficient set to 0.9 and a weight decay coefficient set to 0.001. The cross-entropy loss function is used. A step-by-step learning rate decay strategy is adopted, with an initial learning rate set to 0.1. Every 20 epochs during training, the learning rate is decayed to 0.1 times its original value, for a total of 100 epochs. The batch size is set to 16. During training, four independent MAR-GCN models are trained for the four modalities of data streams constructed above: joints, skeletons, joint motion, and skeleton motion. In the testing or actual inference phase, a decision-level fusion strategy is used, weighted summing of the Softmax probability scores output by the four models, and selecting the category with the highest probability as the final cow behavior recognition result.
[0100] In this embodiment, the visualization effect of the MAR-GCN model in recognizing limp behavior is as follows: Figure 8 As shown, in Figure 8 The visualization results of the MAR-GCN model for recognizing lameness in dairy cows, shown in the image, demonstrate that the model can accurately capture subtle gait features and correctly identify lameness in dairy cows.
[0101] The present invention discloses a deep learning method for recognizing daily behaviors of dairy cows based on skeleton features. This method achieves recognition of dairy cow skeleton behaviors based on semantic action segmentation and multiple attention cascades. After rigorous comparative and ablation experiments, it has been verified that the method has significantly outperformed existing technologies in terms of recognition accuracy, fine-grained feature capture, and complex temporal modeling.
[0102] On a self-built CSBD dataset, the proposed MAR-GCN model was compared with mainstream skeleton-based behavior recognition models (including ST-GCN, AGCN, CTR-GCN, Shift-GCN, etc.). Experimental results show that the MAR-GCN model achieves the best performance across all metrics.
[0103] 1. Significantly superior average recognition accuracy: The average recognition accuracy (Top-1 Accuracy) of the model of this invention reaches 94.78%. Compared with the classic baseline model ST-GCN (87.27%), the accuracy is significantly improved by 7.51%; compared with the improved 2s-AGCN model (92.05%), it is improved by 2.73%, demonstrating the superiority of the overall architecture of this invention.
[0104] 2. Overcoming bottlenecks in behavior recognition: For the "limping" behavior, where action features are easily confused, the model of this invention improves its recognition accuracy from 20.00% to 70.00%; for the "drinking" behavior, the accuracy is improved from 70.00% to 100.00%, which greatly solves the problem of misjudgment caused by the inability of traditional models to distinguish subtle gait differences and head and neck micro-movements.
[0105] Ablation experiments using a step-by-step model decomposition verified the design mechanism of each core module and its specific contribution to performance improvement:
[0106] 1. SSA-GC constructs "semantic hyperedges" to logically connect non-adjacent parts with cooperative motion relationships, capturing long-distance cooperative features. After introducing SSA-GC, the model's recognition accuracy on the joint data stream significantly improved from 87.27% for ST-GCN to 92.27%, and the accuracy on the skeleton data stream improved to 92.73%. This demonstrates that semantic hyperedges effectively enhance the model's understanding of the overall pose structure.
[0107] 2. TAC employs a cascaded filtering mechanism of "time-space-channel" to distinguish similar behaviors. After adding the TAC module to the model incorporating SSA-GC, the recognition accuracy of the joint data stream reached 94.09%, and the accuracy of the joint motion data stream reached 94.55%. This indicates that the TAC module greatly enhances the model's sensitivity to subtle differences in movements.
[0108] 3. HMT-GC uses convolutional kernels of different sizes to construct multi-layered temporal receptive fields in parallel, taking into account behaviors with different durations. Experiments show that the average accuracy of the model is 92.96% when using only a single scale (k=5), while the average accuracy is improved to 94.78% after using hybrid multi-scale fusion, demonstrating its powerful modeling ability for complex temporal dynamics.
[0109] This invention also constructed a cow skeleton behavior dataset (CSBD), containing 1100 high-quality video samples covering six typical behaviors, and which underwent rigorous cleaning and standardization. Data quality verification experiments show that even using only a portion of the data stream (such as skeleton data), the model can achieve a high accuracy of 95.00% on this dataset, proving that the dataset has a reasonable sample distribution and high class discrimination, providing a reliable data benchmark for the research and evaluation of related algorithms.
[0110] Another embodiment of the present invention provides a computer device including a memory and a processor, wherein the memory stores a computer program, and when the processor runs the computer program stored in the memory, the processor executes the deep learning recognition method for daily cow behavior based on skeleton features according to the present invention.
[0111] Another embodiment of the present invention provides a computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the processor performs the deep learning recognition method for daily cow behavior based on skeleton features according to the present invention.
Claims
1. A deep learning method for recognizing dairy cow daily behavior based on skeleton features, characterized in that, The method comprises the following steps: Collecting dairy cow behavior video samples, processing the video samples frame by frame, detecting and positioning the key joints of the dairy cow body in each frame, and constructing a dairy cow skeleton behavior dataset based on the key joints; the dairy cow skeleton behavior dataset comprises four modal data streams, and the four modal data streams are respectively: joint feature, skeleton feature, joint motion feature and skeleton motion feature; A MAR-GCN model is built, the MAR-GCN model comprises nine stacked graph convolutional neural network blocks, the graph convolutional neural network blocks comprise a spatial semantic feature aggregation graph convolutional module and a serial mixed multi-scale time graph convolutional module, and a multiple attention module is arranged after every three graph convolutional neural network blocks, and the MAR-GCN model finally obtains the final dairy cow behavior category through global average pooling and a full connection layer; The parameters of the MAR-GCN model are set, four independent MAR-GCN models are trained for the four modal data streams respectively, a decision-level fusion strategy is adopted, the Softmax probability scores output by the four independent MAR-GCN models are weighted and summed, and the category with the maximum probability is selected as the final dairy cow behavior recognition result.
2. The deep learning method for recognizing dairy cow daily behavior based on skeleton features according to claim 1, characterized in that, The collecting of dairy cow behavior video samples, the processing of the video samples frame by frame, the detection and positioning of the key joints of the dairy cow body in each frame, and the construction of a dairy cow skeleton behavior dataset based on the key joints comprise: The pre-trained and fine-tuned YOLOv8-x pose estimation model is used for frame-by-frame processing of the collected dairy cow behavior video, detects and locates the key joint points of the dairy cow body, and obtains the two-dimensional coordinates and confidence scores of each key joint point . The joint feature is defined as original coordinates , where denotes the coordinate of the i-th joint in the t-th frame in the C-dimensional space. The bone features are defined as vector differences between adjacent joints to characterize the length and direction of the limbs; The joint motion feature is defined as the inter-frame difference of coordinates ; The skeleton motion feature is defined as the inter-frame change of the skeleton vector. 3.The deep learning method for recognizing dairy cow daily behavior based on skeleton features according to claim 1, characterized in that, The spatial semantic feature aggregation graph convolution module comprises four parallel branches, wherein the first three branches respectively process the node itself, the centripetal connection and the centrifugal connection of the traditional graph structure adjacency matrix to extract local features, and the other branch utilizes the semantic hyperedge association matrix to aggregate global semantic features The spatial semantic feature aggregation graph convolution module introduces an adaptive adjacency matrix in each branch , which is generated by projecting and similarity calculation of input features through two learnable 1D convolution layers, i.e. ; The output feature of the spatial semantic feature aggregation graph convolution module The spatial features are deeply fused by the physical connection, the semantic connection and the adaptive connection weighted aggregation, and a formula is represented as: ； wherein, denotes a predefined static physical connection or semantic hyperedge, is an input feature, is a learnable weight parameter, is a set of three edges of a node itself, centripetal connection and centrifugal connection, is a semantic hyperedge incidence matrix, is an activation function, and k is a specific adjacency matrix or incidence matrix.
4. The deep learning method for dairy cow routine behavior recognition based on skeleton features according to claim 3, characterized in that, The semantic super-edge association matrix is obtained by: according to the motion physiology characteristics of the dairy cow, dividing all the 16 joints into 6 sub-sets corresponding to "head", "trunk", "right forelimb", "left forelimb", "right hind limb" and "left hind limb", and the joints of each part form a super-edge; and defining the parts with a cooperative motion relationship as a "meta-action" set. 5.The deep learning method for recognizing dairy cow routine behavior based on skeleton features according to claim 1, characterized in that, The multiple attention module adopts a cascaded structure and re-labels the features in the time, space and channel dimensions in turn; the time attention branch calculates the time weight through a multi-head self-attention mechanism and selects the key frame in which the behavior occurs; the space attention branch focuses on the key joint with the largest contribution to behavior discrimination in each frame on the basis of time weighting; and the channel attention branch captures the dependency relationship between channels by using a one-dimensional convolution; The attention mechanism is: SCA(F)=F⊗σ(MHA(GAP(F),n)) ; In the formula, F is an input feature tensor, SCA is a channel attention module, GAP is a global average pooling, MHA is a multi-head self-attention mechanism, is the number of heads in the self-attention mechanism, σ is an activation function, and is an element-wise multiplication. The calculation mode of the multiple attention module adopting the cascaded structure is: ； In the formula, F is an input feature tensor, The multi-attention module adopts a cascade structure, 、 、 respectively represent that MHA is performed on the channel dimension, the spatial dimension and the time dimension, and GAP is performed on the other two dimensions. 6.The deep learning method for dairy cow routine behavior recognition based on skeleton features according to claim 1, characterized in that, The mixed multi-scale time graph convolutional module performs time series modeling; branches with different receptive fields are arranged in parallel in the time dimension, including: one branch adopts parallel operations with a small convolution kernel and two dilated convolution operations with different dilated rates, for capturing short and rapid instantaneous action details; another branch adopts parallel operations with a large convolution kernel and two dilated convolution operations with different dilated rates, for expanding the receptive field in the time dimension and thus capturing long-period behavior dependency relationships; and a max-pooling branch is arranged to retain significant features. By fusing the outputs of multi-scale branches, the mixed multi-scale temporal graph convolution module can generate a spatio-temporal feature representation containing rich dynamic information, ensuring that the model can not only identify instantaneous posture adjustments but also accurately judge persistent long-time behaviors.
7. The deep learning method for dairy cow routine behavior recognition based on skeleton features according to claim 1, characterized in that, The setting MAR-GCN model parameters, training four independent MAR-GCN models for the four kinds of modal data streams comprises: Model training is performed based on a PyTorch deep learning framework; a stochastic gradient descent optimizer is used, with a momentum coefficient set to 0.9 and a weight decay coefficient set to 0.001; A cross-entropy loss function is selected; a stepwise learning rate decay is used as the training strategy, with an initial learning rate set to 0.1, the learning rate decayed to 0.1 times the original value every 20 rounds during the training process, and the total training rounds set to 100 rounds; the batch size is set to 16.
8. A computer device, comprising: The computer program is executed by the processor, and the processor executes the deep learning identification method for daily behaviors of dairy cows based on skeleton features according to any one of claims 1-7.
9. A computer readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor, and the processor executes the deep learning identification method for daily behaviors of dairy cows based on skeleton features according to any one of claims 1-7.