A pig behavior intelligent analysis method and system based on multi-modal semantics

Abstract

The application provides a pig behavior intelligent analysis method and system based on multi-modal semantics, which combines pig posture features and visual features with a graph convolution network to identify fine-grained behaviors of pigs, and comprehensively analyzes high-level behavior semantics of pigs to realize intelligent identification of pig behaviors. The posture features are two-dimensional and three-dimensional coordinate data of pig skeletons and limb key points, and the visual features are appearance information of pigs in video frames. The method introduced in the application comprehensively utilizes advantages of posture semantics and appearance visual semantics, has high anti-interference performance on light, background changes and the like, can be used in large-fence pig group breeding environments and limit-bar breeding environments, and supports contactless pig behavior identification.

Description

Technical Field

[0001] This invention relates to the fields of smart farming and image semantic segmentation technology, and particularly to a method and system for intelligent analysis of pig behavior based on multimodal semantics. Background Technology

[0002] Existing technologies often employ information fusion-based methods for pig behavior recognition, utilizing convolutional neural networks to analyze acceleration-time curves of behaviors such as running and jumping, including the number of peaks and troughs. Alternatively, they may be based on video images, using the Naive Bayes nearest neighbor method to analyze the speed, acceleration, and trajectory characteristics of behaviors such as feeding and drinking. Or, they may be based on auditory features: using classification models to identify coughing, hunger cries, and squeals in pigs, as well as behaviors such as colds, hunger, and fighting.

[0003] The aforementioned pig behavior recognition based on information fusion establishes an acceleration-time curve using the pig's acceleration over a preset time period, determines the set of historical adjacent peak intervals and the set of adjacent peak-valley differences, and uses a convolutional neural network to identify the pig's running and jumping behaviors. However, its peak-valley frequency analysis of the acceleration-time curve makes it difficult to identify behaviors with small movement amplitudes, such as water consumption or feed intake.

[0004] The aforementioned pig behavior recognition based on video images utilizes the Naive Bayes nearest neighbor method to classify and identify pig behaviors, including feeding, drinking, and excretion. This method primarily relies on information such as speed, acceleration, positional changes, and movement trajectories during pig behavior events in the video data. However, relying solely on speed, acceleration, and positional changes has limitations in pig behavior recognition, lacking analysis of visual or gestural semantics; furthermore, the nearest neighbor method used is relatively simple and struggles to capture the higher-level semantics of pig behavior.

[0005] The aforementioned behavior recognition based on sound features involves collecting and storing pig sound audio, performing time warping and windowing filtering, and then using a classification model to identify the sounds of pigs in different states such as health, illness, gathering, and estrus. This method primarily relies on pig sound data. However, the sound information of pigs performing similar behaviors exhibits high consistency, making it difficult to identify fine-grained pig behaviors. For example, it is impossible to distinguish tail biting, ear biting, or fighting behaviors solely based on squeals.

[0006] In summary, existing technologies mainly suffer from two technical problems. On the one hand, existing pig behavior recognition methods cannot effectively identify fine-grained pig behaviors, and lack comprehensive evaluation of multiple co-occurring behaviors during the analysis process. On the other hand, in video-based pig behavior recognition methods, pig visual characteristics are greatly affected by factors such as changes in lighting and occlusion, such as multiple pigs clustering together or insufficient lighting in the corners of the enclosure. This results in limitations in pig behavior analysis, making it difficult to adapt to the complex breeding environments of real-world applications. Summary of the Invention

[0007] This invention aims to address the limitations of existing RGB video-based pig behavior recognition methods in fine-grained action recognition and improve the accuracy of pig behavior recognition.

[0008] This invention designs a pig behavior recognition method based on multimodal semantics. It primarily uses pig posture features, supplemented by visual features, and combines graph convolutional networks to identify fine-grained pig behaviors. Furthermore, it comprehensively analyzes the high-level behavioral semantics of pigs to achieve intelligent recognition of pig behaviors. The posture features used are two-dimensional and three-dimensional coordinate data of the pig's skeleton and limb key points, while the visual features are the appearance information of the pig in video frames. The method described in this invention comprehensively utilizes the advantages of both posture semantics and appearance visual semantics, exhibiting high resistance to interference from lighting and background changes. It can be applied to large-pen pig group rearing environments and gestation pen rearing environments, supporting contactless pig behavior recognition.

[0009] Specifically, this invention proposes an intelligent analysis method for pig behavior based on multimodal semantics, including:

[0010] Step 1: Obtain videos of pig behavior with labeled behavior categories as a pig behavior dataset; extract the 2D pose features of the pigs in the dataset by identifying the 2D coordinates of the pig pose key points in the video frames of the pig behavior dataset; input the 2D pose sequence into the Transformer model for 3D pose estimation to obtain the 3D pose features of the pigs in the dataset.

[0011] Step 2: Input the pig behavior dataset into a multi-branch convolutional neural network to extract visual features of the pig's limbs and head in the video frames; concatenate the visual features with the 3D pose features to obtain multimodal semantic features; input the multimodal semantic features into a pig behavior recognition network based on a graph convolutional network to obtain training classification results; and train the Transformer model, the multi-branch convolutional neural network, and the pig behavior recognition network based on the training classification results and the labeled behavior categories.

[0012] Step 3: Take the pig behavior video of the behavior category to be identified as the video to be classified, and use the trained Transformer model and the multi-branch convolutional neural network to extract the 3D pose features and visual features of the video to be classified, respectively. After concatenating the two, they are fed into the pig behavior recognition network to obtain the pig behavior category in the video to be classified.

[0013] The aforementioned intelligent analysis method for pig behavior based on multimodal semantics, wherein the 3D pose estimation includes:

[0014] In this Transformer model, parameters are set so that the input 2D pose feature sequence of a certain length outputs a single frame of 3D pose features. The attention layer, feedforward network, fully convolutional network, and max pooling network of this Transformer model are all part of the pose semantic learning process. The final convolutional network maps the feature matrix in the high-dimensional space to the low-dimensional space to form 3D pose features.

[0015] The aforementioned intelligent analysis method for pig behavior based on multimodal semantics employs a multi-head self-attention mechanism in its Transformer model. This mechanism consists of a parallel multi-head spatial self-attention mechanism and a multi-head channel self-attention mechanism. The two mechanisms perform feature transformation and abstract semantic learning respectively, and the output features of the two attention mechanisms are fused using an adaptive weight matrix to achieve local feature fusion. The adaptive weight matrix is ​​a parameter matrix learned during model training.

[0016] The aforementioned intelligent analysis method for pig behavior based on multimodal semantics includes a pig behavior recognition network comprising: a spatial max pooling layer, a convolutional network, and a temporal max pooling layer. The spatial max pooling layer extracts features from the spatial dimension of the multimodal semantic features and feeds them into the convolutional network. The convolutional network performs convolution calculations on the features to obtain spatial features. The temporal max pooling layer extracts features from the temporal dimension of the multimodal semantic features to obtain temporal features. Based on the spatial and temporal features, pig behavior categories are identified.

[0017] This invention also proposes an intelligent analysis system for pig behavior based on multimodal semantics, including:

[0018] The initial module is used to acquire videos of pig behavior with labeled behavior categories as a pig behavior dataset; by identifying the 2D coordinates of the pig posture key points in the video frames of the pig behavior dataset, the 2D posture features of the pigs in the pig behavior dataset are extracted; the 2D posture sequence is input into the Transformer model for 3D posture estimation to obtain the 3D posture features of the pigs in the pig behavior dataset.

[0019] The training module is used to input the pig behavior dataset into a multi-branch convolutional neural network to extract visual features of the pig's limbs and head in the video frames; to concatenate the visual features with the 3D pose features to obtain multimodal semantic features; to input the multimodal semantic features into a pig behavior recognition network based on a graph convolutional network to obtain training classification results; and to train the Transformer model, the multi-branch convolutional neural network, and the pig behavior recognition network based on the training classification results and the labeled behavior categories.

[0020] The recognition module is used to take the pig behavior video of the behavior category to be identified as the video to be classified. The trained Transformer model and the multi-branch convolutional neural network are used to extract the 3D pose features and visual features of the video to be classified, respectively. The two are then concatenated and fed into the pig behavior recognition network to obtain the pig behavior category in the video to be classified.

[0021] The aforementioned intelligent pig behavior analysis system based on multimodal semantics, wherein the 3D pose estimation includes:

[0022] In this Transformer model, parameters are set so that the input 2D pose feature sequence of a certain length outputs a single frame of 3D pose features. The attention layer, feedforward network, fully convolutional network, and max pooling network of this Transformer model are all part of the pose semantic learning process. The final convolutional network maps the feature matrix in the high-dimensional space to the low-dimensional space to form 3D pose features.

[0023] The aforementioned intelligent pig behavior analysis system based on multimodal semantics employs a multi-head self-attention mechanism in its Transformer model. This mechanism consists of a parallel multi-head spatial self-attention mechanism and a multi-head channel self-attention mechanism. The two mechanisms perform feature transformation and abstract semantic learning, respectively, and the output features of the two attention mechanisms are fused using an adaptive weight matrix to achieve local feature fusion. The adaptive weight matrix is ​​a parameter matrix learned during model training.

[0024] The aforementioned intelligent analysis system for pig behavior based on multimodal semantics includes a pig behavior recognition network comprising: a spatial max pooling layer, a convolutional network, and a temporal max pooling layer. The spatial max pooling layer extracts features from the spatial dimension of the multimodal semantic features and feeds them into the convolutional network. The convolutional network performs convolution calculations on the features to obtain spatial features. The temporal max pooling layer extracts features from the temporal dimension of the multimodal semantic features to obtain temporal features. Based on the spatial and temporal features, pig behavior categories are identified.

[0025] The present invention also proposes a storage medium for storing programs that execute any of the pig behavior intelligent analysis methods based on multimodal semantics.

[0026] This invention also proposes a client application for any intelligent pig behavior analysis system based on multimodal semantics.

[0027] As can be seen from the above solutions, the advantages of the present invention are:

[0028] This invention presents a low-cost, high-precision solution for intelligent pig behavior recognition in the context of smart farming. It eliminates the need for complex hardware installation and configuration. Based on video data from pigs raised in large enclosures or in gestation stalls, the system and method described in this invention automatically extract and analyze 3D posture features and RGB visual features. Combined with the pig's current growth and development stage, it intelligently analyzes the fine-grained behavior and high-level behavioral semantics of the pigs in the video. The learning of multi-granular posture semantics makes the method highly resistant to interference from factors such as changes in lighting and occlusion. Simultaneously, the learning of RGB visual semantics enhances the method's ability to recognize fine-grained behavior. This non-contact behavioral gendering method effectively avoids stress on the pigs and eliminates the need for excessive labor input from farm workers.

[0029] The method described in this invention relies on video data of pigs. In behavior recognition, it utilizes multi-granular topological semantics of pose data and visual semantics of RGB video frames, fully combining the advantages of both modalities and avoiding the limitations of single-modal data in pig behavior recognition. At the methodological level, a graph convolutional network model based on pose features or a multi-scale convolutional neural network model based on RGB visual features can be used independently to form a pig behavior recognition method based on single-modal semantics; joint training of the two networks yields a pig behavior recognition method based on multi-modal semantics. Attached Figure Description

[0030] Figure 1 This is a schematic diagram of pig posture estimation and posture sequence compression.

[0031] Figure 2 This is a schematic diagram of a lightweight pig behavior recognition method. Detailed Implementation

[0032] Different pig behaviors, such as lameness, mounting, and feed intake, can provide precise guidance for the scientific management of pig farming, which is conducive to improving the level of intelligent management and achieving cost reduction and efficiency improvement in large-scale pig farming. The multimodal semantics of pig skeletal posture and movement characteristics can reflect abnormal pig behaviors to a certain extent. This invention analyzes the behavior of pigs at different growth and breeding stages based on multi-category behavioral videos and posture semantics, forming a multimodal semantic-based intelligent analysis method and system for pig behavior. The main steps of this invention are as follows:

[0033] a) Collect videos of various behaviors of pigs, such as eating, drinking, mounting, and charging, to form a pig behavior dataset.

[0034] b) Design a lightweight pig pose estimation method based on Transformer for 3D pig pose estimation.

[0035] c) Design a lightweight multi-semantic adaptive learning method based on pose and visual features for intelligent analysis of pig behavior. Design a graph convolutional network model based on pose semantics for rapid recognition of pig behavior.

[0036] d) During the testing process, 3D pose estimation and behavior recognition are performed on the given pig videos to achieve pig behavior analysis. A multi-branch convolutional network model based on visual semantics is designed to enhance the semantic learning for recognizing fine-grained pig behaviors.

[0037] To make the above features and effects of the present invention clearer and easier to understand, specific embodiments are described below, and detailed descriptions are provided in conjunction with the accompanying drawings.

[0038] To achieve the above-mentioned objectives, this invention designs an intelligent analysis method for pig behavior based on multimodal semantics, comprising the following steps:

[0039] Video collection and annotation of multi-category pig behaviors. Multi-category behavioral videos of pigs raised in large enclosures were collected, and the video data were annotated according to fine-grained behavioral categories, coarse-grained behavioral categories, and pig growth and breeding stages to form a pig behavior recognition dataset.

[0040] Training of a 3D pose estimation model for pigs. A 2D-to-3D strategy is adopted for 3D pose estimation. First, the 2D coordinates of key points in the pig's pose in the video frame are estimated using a 2D pose estimation method to complete the 2D pose reconstruction of the pig. Then, based on the 2D pose and a pre-trained Transformer, 3D pose estimation is performed to complete the 3D pose reconstruction of the pig.

[0041] The pre-trained Transformer model can be StridedTransformer-Pose3D. Its training process can employ supervised or semi-supervised learning. Preferably, this invention uses an improved Transformer model for 3D pose estimation. The improvement process includes: adding a multi-head channel attention mechanism to the existing spatial attention mechanism in the Transformer model; the two attention mechanisms are parallel in the model. The training steps include: using the original model pre-trained on the human3D dataset; adjusting and improving the attention mechanism of the Transformer model through the above improvement process, and optimizing the parameters on the human3D model to obtain a 3D pose estimation model for pigs; and directly using this 3D pose estimation model to extract 3D pose features based on the 2D pose of pigs. Specifically, this 3D pose estimation model will be used as a feature extractor and jointly trained during action recognition.

[0042] Training a pig behavior recognition model. A lightweight pig behavior recognition network based on graph convolutional networks is designed and trained on 3D pose data to form a skeleton-based pig behavior recognition method. A multi-branch convolutional neural network based on HRNet is designed and trained on multi-class behavior video data to identify the limbs and head of pigs in video frames and extract corresponding RGB visual features.

[0043] Intelligent pig behavior recognition based on high-level behavioral semantics. This method concatenates local RGB visual features with pose features, and jointly trains a graphical convolutional network and a multi-branch convolutional neural network to form an intelligent pig behavior analysis method.

[0044] This invention provides a method for analyzing pig behavior based on multimodal semantics, the specific real-time process of which includes the following steps:

[0045] a) Fine-grained video collection and annotation of pig behavior.

[0046] Fine-grained multi-category behavioral video capture

[0047] RGB video data of various behaviors of pigs in large-scale penned group rearing and gestation pen were collected in actual pig farms. The main time points were the gilt period, 7 days of gestation, 30 days of gestation, 90 days of gestation, 120 days of gestation, and before weaning.

[0048] Table 1 Fine-grained behavior of pigs:

[0049]

[0050] Behavioral video data annotation:

[0051] Fine-grained behavior annotation is used for each frame of the image; for the video frame sequence, both fine-grained and coarse-grained behaviors are used for annotation. Generally, in a video sequence, the fine-grained behavior of pigs in consecutive frames does not change much, that is, the video segment has the same fine-grained behavior. Therefore, the entire video segment needs to be annotated with both fine-grained and coarse-grained behaviors.

[0052] The collected video data was manually labeled, including growth and reproduction stages, coarse-grained behavior categories at higher levels, and fine-grained behavior subdivisions. The ranges of coarse-grained and fine-grained behaviors are shown in Table 1.

[0053] b) 3D pose estimation of pigs

[0054] This invention focuses on pig posture features and employs a 2D-to-3D strategy for 3D posture estimation, i.e., 3D posture estimation based on 2D posture. The 3D posture estimation of pigs is as follows: Figure 1 As shown. For a fixed number of consecutive 2D poses, an improved VTE (Vanilla Transformer Encoder) network is used for consecutive 3D pose estimation. Specifically, as... Figure 1 As shown, Figure 1 The top image represents the 3D pose sequence, where the coordinates of key points in each frame are three-dimensional, representing the final acquired data. The bottom image shows the 2D pose sequence of the pig, with the coordinates of key points in each frame being two-dimensional; the length of the 3D pose sequence must be shorter than the length of the 2D pose sequence. Figure 1 The network model in the example performs two functions: (1) 3D pose estimation: input a 2D pose sequence and output a 3D pose sequence; (2) pose sequence compression: the length of the 3D pose sequence is less than the length of the 2D pose sequence. Example: input 4 frames of 2D pose and output only 1 frame of 3D pose.

[0055] Figure 1The reduction in sequence length is not fixed and can be chosen in various ways depending on the needs. The ratio of 2D poses to 3D poses can be 4:1, 8:1, or 16:1. The pose estimation process includes setting parameters in the network model so that an input 2D pose sequence of a certain length outputs a single frame of 3D pose. The attention layer, feedforward network, fully convolutional network, and max-pooling network in VTE are all part of the pose semantic learning process. The final convolutional network maps the feature matrix from high-dimensional space to low-dimensional space, forming 3D pose data. The guidance process includes the learning and evolution of 2D pose semantics in VTE. The final output process changes the form of the output dimension to form 3D pose features, that is, mapping the keypoints in the pose to three-dimensional space. It is important to note that 3D pose estimation here refers to fitting 2D spatial data to three-dimensional space.

[0056] The improvements include the fact that the multi-head self-attention mechanism used in the original VTE is mainly spatial attention, i.e., multi-head spatial self-attention, which only calculates the interaction between different poses in the pose sequence. The method introduced in this invention adds a multi-head channel self-attention mechanism to calculate the interaction between different skeletal keypoints in a single frame pose on different feature channels. Specifically, in the improved VTE method, the parallel multi-head spatial self-attention mechanism and multi-head channel self-attention mechanism perform feature transformation and abstract semantic learning respectively, and use an adaptive weight matrix to fuse the output features of the two attention mechanisms to complete local feature fusion; where the adaptive weight matrix is ​​the parameter matrix that needs to be learned during model training.

[0057] Following this, long-term semantic learning is performed using a feed-forward network (FFN) and fully convolutional networks (FCN) to obtain a 3D pose sequence of the same length as the 2D pose sequence. Temporal max pooling (TMP) is then used to perform max pooling on the temporal dimension of the 3D pose sequence, reducing the sequence length to 1, thus obtaining single-frame pose features. Finally, a conventional convolutional network is used to process the feature matrix, mapping it from a high-dimensional space to a low-dimensional space to obtain single-frame 3D pose data.

[0058] like Figure 2 As shown, pig behavior recognition includes:

[0059] Pose Topology Semantic Learning: 3D poses contain only the 3D coordinates of limb keypoints, and the number of limb keypoints is fixed, which can significantly reduce computational cost. This invention designs a multi-granularity topology semantic learning model based on graph convolutional networks to simultaneously learn fine-grained joint semantics of a single-frame 3D pose and coarse-grained associated pose semantics of multi-frame 3D pose sequences. To improve the accuracy of behavior recognition, a self-attention network is used to enhance the algorithm model's learning ability of key topology semantics, and matrix addition is used to fuse fine-grained joint semantics and coarse-grained pose semantics. Finally, spatial max-pooling layers, convolutional networks, and temporal max-pooling layers are used for temporal information learning. If the dimension of the feature matrix at this stage is (V, T), the spatial max-pooling layer is used to extract features from the spatial dimension (V) of the input features. The convolutional network is used to perform convolution calculations on the feature matrix after spatial max-pooling, completing the feature extraction for the current stage. The temporal max-pooling layer only extracts features from the temporal dimension (T) of the input features.

[0060] Discriminative RGB visual feature learning: Simultaneously meeting the dual requirements of speed and accuracy in behavior recognition, this invention improves upon object detection and segmentation techniques such as RefineDet and HRNet, designing a multi-branch convolutional neural network to identify the limbs and head of pigs from RGB images and extract corresponding RGB visual features. The extracted RGB visual features of the head and limbs are then fused, integrating local pose features with local visual features. These local pose features encompass both fine-grained joint semantics and coarse-grained pose semantics. This invention first performs feature fusion, then sequentially inputs the features into an SMP layer for spatial dimension (between pose key points in a single frame) feature extraction, and into a TMP layer for temporal dimension (between poses across multiple frames), enhancing the model's accuracy in recognizing fine-grained pig behaviors.

[0061] Based on the identified pig behaviors and their current growth and breeding stage, the system evaluates the multi-granularity semantics of pig behaviors to achieve intelligent analysis of pig behavior. Growth and breeding information serves as supplementary information and is not used in model training. It is combined with fine-grained pig behaviors for comprehensive analysis, identifying breeding-related behaviors based on coarse-grained behavior. For example, the appearance of mounting behavior after 7 days of breeding indicates mating failure and the onset of estrus.

[0062] like Figure 2 As shown, in data structures, adjacency matrices are used to represent the relationships between nodes in a graph, including whether edges exist between nodes and their weights. Adjacency matrices can be used to extract semantic information from graph structures using graph convolutional networks. In graph convolution, simply put, it involves multiplying the adjacency matrix with the input features to complete feature computation in a specific step.

[0063] Fine-grained adjacency matrix. A single-frame pose is viewed as a small-scale graph structure, with skeletal keypoints in the pose as nodes. The relationships between these keypoints can be represented by an adjacency matrix, such as the limb connection between the hand and elbow, or the abstract connection between the hand and foot. Specifically, this adjacency matrix is ​​limited to the current pose, hence the name fine-grained adjacency matrix.

[0064] Coarse-grained adjacency matrix. This method treats the pose sequence as a special graph structure, with each pose as a node. This allows for the calculation of the relationships between poses at different times. In this case, the adjacency matrix corresponds to the entire pose sequence and has a relatively large range, hence the name coarse-grained adjacency matrix.

[0065] Similar behaviors. This refers to similar behaviors in fine-grained behaviors, such as the similarity between walking and limping. The difference between these two fine-grained behaviors lies only in the limping area. It is difficult to identify them by simply relying on posture features. Therefore, it is necessary to enhance the model's recognition ability by introducing local visual features.

[0066] Interactive behavior. This refers to interactive behaviors within fine-grained behaviors, such as tail biting, ear biting, and pen biting. In these fine-grained actions, the pig's own posture is not significantly different (including local areas of the limbs). Therefore, it is necessary to capture the interaction relationships between the pig and other pigs and objects.

[0067] Similar behaviors and interactive behaviors are both fine-grained behaviors, which are the recognition functions that the method model needs to complete. In order to enhance the ability to recognize similar and interactive behaviors, a targeted method model is designed and trained.

[0068] The following are system embodiments corresponding to the above method embodiments. This embodiment can be implemented in conjunction with the above embodiments. The relevant technical details mentioned in the above embodiments are still valid in this embodiment, and will not be repeated here to reduce repetition. Accordingly, the relevant technical details mentioned in this embodiment can also be applied to the above embodiments.

[0069] This invention also proposes an intelligent analysis system for pig behavior based on multimodal semantics, including:

[0070] The initial module is used to acquire videos of pig behavior with labeled behavior categories as a pig behavior dataset; by identifying the 2D coordinates of the pig posture key points in the video frames of the pig behavior dataset, the 2D posture features of the pigs in the pig behavior dataset are extracted; the 2D posture sequence is input into the Transformer model for 3D posture estimation to obtain the 3D posture features of the pigs in the pig behavior dataset.

[0071] The training module is used to input the pig behavior dataset into a multi-branch convolutional neural network to extract visual features of the pig's limbs and head in the video frames; to concatenate the visual features with the 3D pose features to obtain multimodal semantic features; to input the multimodal semantic features into a pig behavior recognition network based on a graph convolutional network to obtain training classification results; and to train the Transformer model, the multi-branch convolutional neural network, and the pig behavior recognition network based on the training classification results and the labeled behavior categories.

[0072] The recognition module is used to take the pig behavior video of the behavior category to be identified as the video to be classified. The trained Transformer model and the multi-branch convolutional neural network are used to extract the 3D pose features and visual features of the video to be classified, respectively. The two are then concatenated and fed into the pig behavior recognition network to obtain the pig behavior category in the video to be classified.

[0073] The aforementioned intelligent pig behavior analysis system based on multimodal semantics, wherein the 3D pose estimation includes:

[0074] In this Transformer model, parameters are set so that the input 2D pose feature sequence of a certain length outputs a single frame of 3D pose features. The attention layer, feedforward network, fully convolutional network, and max pooling network of this Transformer model are all part of the pose semantic learning process. The final convolutional network maps the feature matrix in the high-dimensional space to the low-dimensional space to form 3D pose features.

[0075] The aforementioned intelligent pig behavior analysis system based on multimodal semantics employs a multi-head self-attention mechanism in its Transformer model. This mechanism consists of a parallel multi-head spatial self-attention mechanism and a multi-head channel self-attention mechanism. The two mechanisms perform feature transformation and abstract semantic learning, respectively, and the output features of the two attention mechanisms are fused using an adaptive weight matrix to achieve local feature fusion. The adaptive weight matrix is ​​a parameter matrix learned during model training.

[0076] The aforementioned intelligent analysis system for pig behavior based on multimodal semantics includes a pig behavior recognition network comprising: a spatial max pooling layer, a convolutional network, and a temporal max pooling layer. The spatial max pooling layer extracts features from the spatial dimension of the multimodal semantic features and feeds them into the convolutional network. The convolutional network performs convolution calculations on the features to obtain spatial features. The temporal max pooling layer extracts features from the temporal dimension of the multimodal semantic features to obtain temporal features. Based on the spatial and temporal features, pig behavior categories are identified.

[0077] The present invention also proposes a storage medium for storing programs that execute any of the pig behavior intelligent analysis methods based on multimodal semantics.

[0078] This invention also proposes a client application for any intelligent pig behavior analysis system based on multimodal semantics.

Claims

1. A method for intelligent analysis of pig behavior based on multimodal semantics, characterized in that, include: Step 1: Obtain videos of pig behaviors with labeled behavior categories as a pig behavior dataset; By identifying the 2D coordinates of key points of pig posture in video frames of the pig behavior dataset, the 2D posture features of the pigs in the dataset are extracted; the 2D posture sequence is then input into the Transformer model for 3D posture estimation to obtain the 3D posture features of the pigs in the dataset. Step 2: Input the pig behavior dataset into a multi-branch convolutional neural network to extract the visual features of the pig's limbs and head in the video frames; The visual features and the 3D pose features are spliced ​​together to obtain multimodal semantic features. The multimodal semantic features are then input into a pig behavior recognition network based on a graph convolutional network to obtain training classification results. Based on the training classification results and the labeled behavior categories, the Transformer model, the multi-branch convolutional neural network, and the pig behavior recognition network are trained. Step 3: Take the pig behavior video of the behavior category to be identified as the video to be classified. Use the trained Transformer model and the multi-branch convolutional neural network to extract the 3D pose features and visual features of the video to be classified, respectively. Then, concatenate the two and send them into the pig behavior recognition network to obtain the pig behavior category in the video to be classified. The Transformer model employs a multi-head self-attention mechanism consisting of a parallel multi-head spatial self-attention mechanism and a multi-head channel self-attention mechanism. The two mechanisms perform feature transformation and abstract semantic learning respectively, and the output features of the two attention mechanisms are fused using an adaptive weight matrix to achieve local feature fusion. The adaptive weight matrix is ​​the parameter matrix learned during model training; The pig behavior recognition network includes: a spatial max pooling layer, a convolutional network, and a temporal max pooling layer; The spatial max pooling layer is used to extract features from the spatial dimension of the multimodal semantic features and feed them into the convolutional network. The convolutional network performs convolution calculations on the features to obtain spatial features. The temporal max pooling layer extracts features from the temporal dimension of the multimodal semantic features to obtain temporal features. Based on the spatial features and temporal features, pig behavior categories are identified.

2. The intelligent analysis method for pig behavior based on multimodal semantics as described in claim 1, characterized in that, The 3D pose estimation includes: In this Transformer model, parameters are set so that the input 2D pose feature sequence of a certain length outputs a single frame of 3D pose features. The attention layer, feedforward network, fully convolutional network, and max pooling network of this Transformer model are all part of the pose semantic learning process. The final convolutional network maps the feature matrix in the high-dimensional space to the low-dimensional space to form 3D pose features.

3. A pig behavior intelligent analysis system based on multimodal semantics, characterized in that, include: The initial module is used to acquire videos of pig behavior with labeled behavior categories, as a pig behavior dataset; By identifying the 2D coordinates of key points of pig posture in video frames of the pig behavior dataset, the 2D posture features of the pigs in the dataset are extracted; the 2D posture sequence is then input into the Transformer model for 3D posture estimation to obtain the 3D posture features of the pigs in the dataset. The training module is used to input the pig behavior dataset into a multi-branch convolutional neural network to extract visual features of the pig's limbs and head from video frames; The visual features and the 3D pose features are spliced ​​together to obtain multimodal semantic features. The multimodal semantic features are then input into a pig behavior recognition network based on a graph convolutional network to obtain training classification results. Based on the training classification results and the labeled behavior categories, the Transformer model, the multi-branch convolutional neural network, and the pig behavior recognition network are trained. The recognition module is used to take the pig behavior video of the behavior category to be identified as the video to be classified. The trained Transformer model and the multi-branch convolutional neural network are used to extract the 3D pose features and visual features of the video to be classified, respectively. The two are then concatenated and fed into the pig behavior recognition network to obtain the pig behavior category in the video to be classified. The Transformer model employs a multi-head self-attention mechanism consisting of a parallel multi-head spatial self-attention mechanism and a multi-head channel self-attention mechanism. The two mechanisms perform feature transformation and abstract semantic learning respectively, and the output features of the two attention mechanisms are fused using an adaptive weight matrix to achieve local feature fusion. The adaptive weight matrix is ​​the parameter matrix learned during model training; The pig behavior recognition network includes: a spatial max pooling layer, a convolutional network, and a temporal max pooling layer; The spatial max pooling layer is used to extract features from the spatial dimension of the multimodal semantic features and feed them into the convolutional network. The convolutional network performs convolution calculations on the features to obtain spatial features. The temporal max pooling layer extracts features from the temporal dimension of the multimodal semantic features to obtain temporal features. Based on the spatial features and temporal features, pig behavior categories are identified.

4. The intelligent pig behavior analysis system based on multimodal semantics as described in claim 3, characterized in that, The 3D pose estimation includes: In this Transformer model, parameters are set so that the input 2D pose feature sequence of a certain length outputs a single frame of 3D pose features. The attention layer, feedforward network, fully convolutional network, and max pooling network of this Transformer model are all part of the pose semantic learning process. The final convolutional network maps the feature matrix in the high-dimensional space to the low-dimensional space to form 3D pose features.

5. A storage medium for storing the program of the intelligent analysis method for pig behavior based on multimodal semantics as described in claim 1 or 2.

6. A client application for the intelligent pig behavior analysis system based on multimodal semantics as described in claim 3 or 4.

Images